pespin has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ggsn/+/38500?usp=email ) Change subject: doc: Document MTU features in User Manual and example config files ...................................................................... doc: Document MTU features in User Manual and example config files Related: OS#6298 Related: SYS#7122 Change-Id: Ib6e974b38107fe48072380f768e1881f0fc95e80 --- M doc/examples/osmo-ggsn-kernel-gtp.cfg M doc/examples/osmo-ggsn.cfg M doc/manuals/chapters/configuration.adoc A doc/manuals/chapters/mtu.adoc M doc/manuals/chapters/running.adoc 5 files changed, 200 insertions(+), 12 deletions(-) Approvals: osmith: Looks good to me, but someone else must approve pespin: Looks good to me, approved Jenkins Builder: Verified laforge: Looks good to me, but someone else must approve diff --git a/doc/examples/osmo-ggsn-kernel-gtp.cfg b/doc/examples/osmo-ggsn-kernel-gtp.cfg index bed52b9..69ab51b 100644 --- a/doc/examples/osmo-ggsn-kernel-gtp.cfg +++ b/doc/examples/osmo-ggsn-kernel-gtp.cfg @@ -44,6 +44,7 @@ gtpu-mode kernel-gtp tun-device tun4 type-support v4 + mtu default apply ip prefix dynamic 172.16.222.0/24 ip dns 0 8.8.8.8 ip dns 1 8.8.4.4 diff --git a/doc/examples/osmo-ggsn.cfg b/doc/examples/osmo-ggsn.cfg index 53cebc1..52b1034 100644 --- a/doc/examples/osmo-ggsn.cfg +++ b/doc/examples/osmo-ggsn.cfg @@ -44,6 +44,7 @@ gtpu-mode tun tun-device tun4 type-support v4 + mtu default apply ip prefix dynamic 172.16.222.0/24 ip dns 0 8.8.8.8 ip dns 1 8.8.4.4 @@ -53,6 +54,7 @@ gtpu-mode tun tun-device tun6 type-support v6 + mtu default apply ipv6 prefix dynamic 2001:780:44:2000:0:0:0:0/56 ipv6 dns 0 2001:4860:4860::8888 ipv6 dns 1 2001:4860:4860::8844 @@ -62,6 +64,7 @@ gtpu-mode tun tun-device tun46 type-support v4v6 + mtu default apply ip prefix dynamic 172.16.46.0/24 ip dns 0 8.8.8.8 ip dns 1 8.8.4.4 diff --git a/doc/manuals/chapters/configuration.adoc b/doc/manuals/chapters/configuration.adoc index f0889d3..fc67cb2 100644 --- a/doc/manuals/chapters/configuration.adoc +++ b/doc/manuals/chapters/configuration.adoc @@ -1,3 +1,4 @@ +[[osmoggsn_configuring]] == Configuring OsmoGGSN All configuration of OsmoGGSN is performed using the VTY. For more @@ -22,6 +23,7 @@ gtpu-mode tun tun-device tun4 type-support v4 + mtu default apply ip prefix dynamic 176.16.222.0/24 ip dns 0 192.168.100.1 ip dns 1 8.8.8.8 @@ -162,18 +164,20 @@ ---- OsmoGGSN(config-ggsn-apn)# gtpu-mode tun <1> OsmoGGSN(config-ggsn-apn)# type-support v4 <2> -OsmoGGSN(config-ggsn-apn)# ip prefix dynamic 176.16.222.0/24 <3> -OsmoGGSN(config-ggsn-apn)# ip dns 0 192.168.100.1 <4> -OsmoGGSN(config-ggsn-apn)# ip dns 1 8.8.8.8 <5> -OsmoGGSN(config-ggsn-apn)# ip ifconfig 176.16.222.0/24 <6> +OsmoGGSN(config-ggsn-apn)# mtu 1420 apply <3> +OsmoGGSN(config-ggsn-apn)# ip prefix dynamic 176.16.222.0/24 <4> +OsmoGGSN(config-ggsn-apn)# ip dns 0 192.168.100.1 <5> +OsmoGGSN(config-ggsn-apn)# ip dns 1 8.8.8.8 <6> +OsmoGGSN(config-ggsn-apn)# ip ifconfig 176.16.222.0/24 <7> ---- <1> Use the userspace GTP-U handling using a TUN device <2> Support (only) IPv4 Addresses -<3> Specify the pool of dynamic IPv4 addresses to be allocated to PDP +<3> Specify MTU to announce to MS. Apply the MTU on the tunnel interface. +<4> Specify the pool of dynamic IPv4 addresses to be allocated to PDP contexts -<4> Specify the primary DNS server to be provided using IPCP/PCO -<5> Specify the secondary DNS server to be provided using IPCP/PCO -<6> Request OsmoGGSN to configure the `tun4` device network/netmask +<5> Specify the primary DNS server to be provided using IPCP/PCO +<6> Specify the secondary DNS server to be provided using IPCP/PCO +<7> Request OsmoGGSN to configure the `tun4` device network/netmask NOTE:: If you use the optional `ip ifconfig` command to set the network device address/mask, OsmoGGSN must run with root or `CAP_NET_ADMIN` @@ -271,6 +275,7 @@ ---- # ip addr add 192.168.7.1/24 dev apn0 +# ip link set mtu 1420 dev apn0 # ip link set apn0 up ---- @@ -298,13 +303,18 @@ [Match] Name=apn0 <1> +[Link] +MTUBytes=1420 <2> + [Network] -Address=192.168.7.1/24 <2> -IPMasquerade=yes <3> +Address=192.168.7.1/24 <3> +IPMasquerade=yes <4> ---- <1> The network device name, which must match the one in the apn0.netdev unit file above -<2> The local IP address configured on the device -<3> Requesting systemd to configure IP masquerading for this interface. Depending on your needs, +<2> Requesting systemd to set the MTU for this interface. The MTU of the tun + interface should be lower than regular, since it must accommodate the extra IP/UDP/GTPv1U headers. +<3> The local IP address configured on the device +<4> Requesting systemd to configure IP masquerading for this interface. Depending on your needs, You may not want this if you have proper end-to-end routing set up, and want to have transparent inbound IP access to your GPRS-attached devices. @@ -326,6 +336,7 @@ gtpu-mode tun tun-device apn0 type-support v4 + mtu 1420 ip prefix dynamic 192.168.7.0/24 ip dns 0 192.168.100.1 ip dns 1 8.8.8.8 diff --git a/doc/manuals/chapters/mtu.adoc b/doc/manuals/chapters/mtu.adoc new file mode 100644 index 0000000..4c2064c --- /dev/null +++ b/doc/manuals/chapters/mtu.adoc @@ -0,0 +1,171 @@ +=== MTU considerations + +When running OsmoGGSN, the user may want to take network Maximum Transmission +Unit (MTU) into consideration, and configure it based on network specific setup. + +Applying and announcing a proper MTU provides, for the MS employing it, reduced +transmission overhead (ie. due to IP fragmentation) and avoids potential +problems due to misconfigured nodes in the path (e.g. ICMP packet filtering). + +In OsmoGGSN, the MTU can be configured per APN through the VTY, see +<<osmoggsn_configuring>>. If told so by the config, osmo-ggsn will apply the MTU +on the APN network interface. + +==== MTU announced to MS + +The configured MTU is also announced to the MS through: + +* IPv4 APN: GTPv1C Create PDP Context Response, PCO IE "IPv4 Link MTU", 3GPP TS + 24.008 Table 10.5.154. +* IPv6 APN: ICMPv6 Routing Advertisement during IPv6 SLAAC procedure, RFC 4861. + +NOTE: It is up to the MS to request and use the link MTU size provided by the +network. Hence, providing an MTU size does not guarantee that there will be no +packets larger than the provided value. + +==== GTP-U tunnel overhead + +OsmoGGSN is encapsulating traffic over GTP-U, it means the packets being received, +encapsulated and transmitted over the tunnel get their size increased by the sum of +IP/UDP/GTPv1U headers being prepended: + +* IP: IPv4 headers can take up to 60 bytes (due to IPv4 options). IPv6 headers + can take up to 40 bytes (assuming no extension headers for IPv6 in general, + since they are uncommon). Hence, the 60 bytes of IPv4 are picked since that's + greater than the IPv4. +* UDP: The UDP header takes 8 bytes. +* GTPv1U: The GTPv1U header takes 12 bytes, assuming here no extensions headers + are used (OsmoGGSN doesn't use them). + +Hence, these headers add an overhead of up to `80`` bytes, as per the below formula: + +---- +GTPv1U_OVERHEAD = 60 + 8 + 12 = 80 bytes +---- + +==== Figuring out optimal MTU value + +There is no golden MTU value, since it really depends on the local (and remote) +networks where traffic is routed. The idea is finding out a value that: + +* Is as big as possible, to avoid need to split big chunks of data into lots of + small packets, hence affecting performance due to processing overhead: extra + headers being trnasmitted, plus processing of extra packets. +* Is small enough so that it can be transported over the lower layers of the + links involving the communication, avoiding IP fragmentation, which again hurts + performance. + +OsmoGGSN, by default, assumes that traffic is transported over an Ethernet +network, which has a standarized maximum MTU of 1500 bytes. Hence, by default it +announces an MTU of of `1420` bytes as per the following formula: + +---- +TUNNEL_MTU = ETH_MTU - GTPv1U_OVERHEAD = 1500 - 80 = 1420 bytes +---- + +Under certain networks, the base MTU may already be smaller than Ethernet's MTU +(1500 bytes), due to, for instance, existence of some sort of extra tunneling +protocol in the path, such as a VPN, ipsec, extra VLAN levels, etc. Under this +scenario, the user must take care of figuring out the new base MTU value to use +for the calculations presented above. This can be accomplished by packet +inspection (eg. `wireshark`) or with tools such as `ping`, running it with a +certain packet size and the IPv4 DF bit set, and see up to which packet size the +networks is able to forward the message. + +.Example: Test if packets of 1420 bytes can reach peer host 176.16.222.4 +---- +$ ping -M probe 176.16.222.4 -s 1420 +---- + +=== Increasing outer MTU + +Specifications at IEEE 802.3 establish that standard Ethernet has a maximum MTU +of `1500` bytes. +However, many Ethernet controllers can nowadays overcome this limit and allow +the use of so called _jumbo frames_. The _jumbo frames_ maximum MTU varies +depending on the implementation, with `9000` bytes being a commonly used limit. + +Note that using MTUs over the standarized `1500` bytes by means of _jumbo frames_ +can create interoperability problems with networks not supporting such frames +(eg. forcing of IP packet fragmentation), plus the fact that larger frames +consume more Ethernet link transmission time, causing greater delays and +increasing latency. + +Nevertheless, if the operator: + +* is in control of the whole GTP-U path between OsmoGGSN and the MS, and +* has Ethernet NICs supporting MTUs bigger than 1500 or uses any other link + layer supporting as well bigger MTUs. + +Then, it may be wise for the operator to configure such links with an increased +outer MTU so that they can end up transporting GTP-U inner payload of 1500 bytes +without fragmentation ocurring. + +Hence, following the examples presented on the above sections, one could +configure *all the links* which are part of the GTP-U path to use an outer MTU +of `1580` bytes, as per the following formula: + +---- +TUNNEL_MTU = ETH_MTU + GTPv1U_OVERHEAD = 1500 + 80 = 1580 bytes +---- + +.Example: Setting an MTU of `1580` to network interface `eth0` under Linux +---- +ip link set mtu 1580 dev eth0 +---- + +==== TCP MSS Clamping + +Usually endpoints use Path MTU Discovery (PMTUD) to determine the maximum MTU to +reach the peer. However, this technique may sometimes not be optimal for all +users of OsmoGGSN: + +* MS may not support requesting and/or configuring the MTU OsmoGGSN announced. +* MS may not support PMTUD on its network stack, or may not have it enabled or + may be buggy. +* Network may be misconfigured or some middlebox may be buggy (eg. not + forwarding ICMP `Packet Too Big` packets). + +Furthermore, PMTUD takes time to figure out the maximum MTU to use, since it +relies on sending data and checking if it got lost, and adapting to the fact, +reducing efficiency (throughput) of connections or even stalling them completely +when big packets are generated. + +Hence, it may become useful for the operator of OsmoGGSN to, on top of MTU +configuration, also configure its network to tune TCP Maximum Segment Size (MSS) +option of TCP connections being established over the GTPv1U tunnel. This will +make sure at least TCP connections can use the full capacity of the path MTU +without passing its imit. + +The MSS TCP option is an optional parameter in the TCP header sent during TCP +initial handshake (`SYN,SYN/ACK`) that specifies the maximum amount of bytes of +TCP payload a TCP chunk may transport. The MSS value doesn't count the +underlaying IP/TCP headers. + +Hence, following up on MTU size calculations from previous section, with a +sample GTPv1U MTU of 1420 bytes and IP header of 60 bytes, plus taking into +account that TCP header can span up to 56 bytes, we'd get to an MSS value of: + +---- +MSS = TUNNEL_MTU - IP_HDR - TCP_HDR = 1420 - 60 - 56 = 1304 +---- + +In linux, the MSS of TCP connections can be clamped using iptables: + +---- +iptables -t nat -A PREROUTING -p tcp --tcp-flags SYN,RST SYN -i apn0 -j TCPMSS --set-mss 1304 +iptables -t nat -I POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o apn0 -j TCPMSS --set-mss 1304 +ip6tables -t nat -A PREROUTING -p tcp --tcp-flags SYN,RST SYN -i apn0 -j TCPMSS --set-mss 1304 +ip6tables -t nat -I POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o apn0 -j TCPMSS --set-mss 1304 +---- + +==== Further Reading + +Check the following specs regarding MTU in 3GPP mobile networks: + +* 3GPP TS 29.061 section 11.2.1.5 +* 3GPP TS 290.060 section 13.2 IP Fragmentation +* 3GPP TS 25.414 section 6.1.3.3 +* 3GPP TS 23.060 section 9.3, Annex C +* 3GPP TS 24.008 (PCO IPv4 MTU) +* RFC 4861 (IPv6 Router Advertisement) diff --git a/doc/manuals/chapters/running.adoc b/doc/manuals/chapters/running.adoc index 04c5b06..386b6b5 100644 --- a/doc/manuals/chapters/running.adoc +++ b/doc/manuals/chapters/running.adoc @@ -52,6 +52,8 @@ your distribution's documentation -- e.g. look for @net.ipv4.ip_forward=1@ in @/etc/sysctl.d/@, and https://wiki.debian.org/iptables for masquerading. +include::{srcdir}/chapters/mtu.adoc[] + === Multiple instances Running multiple instances of `osmo-ggsn` is possible if all GGSN instances -- To view, visit https://gerrit.osmocom.org/c/osmo-ggsn/+/38500?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: osmo-ggsn Gerrit-Branch: master Gerrit-Change-Id: Ib6e974b38107fe48072380f768e1881f0fc95e80 Gerrit-Change-Number: 38500 Gerrit-PatchSet: 12 Gerrit-Owner: pespin &lt;pespin(a)sysmocom.de&gt; Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge &lt;laforge(a)osmocom.org&gt; Gerrit-Reviewer: lynxis lazus &lt;lynxis(a)fe80.eu&gt; Gerrit-Reviewer: osmith &lt;osmith(a)sysmocom.de&gt; Gerrit-Reviewer: pespin &lt;pespin(a)sysmocom.de&gt;