Hi,
I need to increase the throughput through the GTP tunnels to around 9Gb/s.
I have two servers and they are connected using two GTP tunnels over a Mellanox 25Gb/s link. Via the tunnels I am seeing a total of 4.2 Gb/s to 4.5 Gb/s using iPerf.
If data is sent without the tunnels I see a throughput of around 24.5Gb/s.
I have tried various tweaks to improve the throughput over the GTP tunnels and have only made marginal gains. I have tried iPerf 2.0.9 and iPerf3 and see the same results. Wireshark shows the iPerf MTU is reflected in the pcap logs.
I have tried changing the txqueuelen on the tunnels, changing osmo-ggsn niceness, network buffer sizes etc.
The restriction seems to be osmo-ggsn as that is taking up to 92% CPU on one of the cores when the throughput tests are running.
I'm hoping I have made a mistake in the configuration of OsmoGGSN and
sgsnemu.
--
The servers are running Ubuntu 16.04 desktop. 16GB RAM - 12.6GB free when running throughput tests. Intel E52620v3. Mellanox MCX516A-CCAT network adapter. 25Gb direct attach copper connecting the two servers OsmoGGSN version 1.1.0.75-3e44-dirty.
For one of the servers: -
Mellanox interface details
ens3f0 Link encap:Ethernet HWaddr ec:0d:9a:a0:1d:7c
inet addr:172.16.8.1 Bcast:172.16.255.255 Mask:255.255.0.0
inet6 addr: fe80::8f38:33e3:bd46:dd65/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:142373677 errors:0 dropped:0 overruns:0 frame:0
TX packets:157500406 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:10000
RX bytes:793825609487 (793.8 GB) TX bytes:879898325287 (879.8 GB)
Tunnel interface details
S1Utun Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.1.1.2 P-t-P:10.1.1.2 Mask:255.255.255.0
inet6 addr: fe80::b3fc:3d43:b751:bcbf/64 Scope:Link
UP POINTOPOINT RUNNING MTU:9000 Metric:1
RX packets:4640412 errors:0 dropped:0 overruns:0 frame:0
TX packets:2272797 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:37729672063 (37.7 GB) TX bytes:118866016 (118.8 MB)
X2Utun Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.1.2.2 P-t-P:10.1.2.2 Mask:255.255.255.0
inet6 addr: fe80::6ed8:8a79:b3e9:a51d/64 Scope:Link
UP POINTOPOINT RUNNING MTU:9000 Metric:1
RX packets:1824011 errors:0 dropped:0 overruns:0 frame:0
TX packets:3856238 errors:0 dropped:4229 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:95648485 (95.6 MB) TX bytes:31367266755 (31.3 GB)
client side iperf
server1
iperf3 -c 10.1.2.1 -i1 -t 30 -M8100 -w 2048K -P8
server2
iperf3 -c 10.1.2.1 -i1 -t 30 -M8100 -w 2048K -P8
server1 and 2 iperf server
iperf3 -s -i1
osmo-ggsn
-c /home/osmo-ggsn.cfg -g 172.16.1.1
contents
of osmo-ggsn.cfg
!
! OpenGGSN (0.94.1-adac) configuration saved from vty
!!
!
log stderr
logging filter all 1
logging color 1
logging print category 0
logging timestamp 0
logging level ip info
logging level tun info
logging level ggsn info
logging level sgsn notice
logging level icmp6 notice
logging level lglobal notice
logging level llapd notice
logging level linp notice
logging level lmux notice
logging level lmi notice
logging level lmib notice
logging level lsms notice
logging level lctrl notice
logging level lgtp info
logging level lstats notice
logging level lgsup notice
logging level loap notice
logging level lss7 notice
logging level lsccp notice
logging level lsua notice
logging level lm3ua notice
logging level lmgcp notice
!
stats interval 5
!
line vty
no login
!
ggsn X2Uggsn
gtp state-dir /tmp
gtp bind-ip 172.16.8.1
gtp gtp0-port 3386
gtp gtp1c-port 2123
gtp gtp1u-port 2153
gtp gtp1uremot-port 2153
apn X2U
gtpu-mode tun
tun-device X2Utun
type-support v4
ip prefix dynamic 10.1.2.0/24
ip dns 0 192.168.100.1
ip dns 1 8.8.8.8
ip ifconfig 10.1.2.2/24
no shutdown
default-apn X2U
no shutdown ggsn
ggsn S1Uggsn
gtp state-dir /tmp
gtp bind-ip 172.16.8.1
gtp gtp0-port 3387
gtp gtp1c-port 2124
gtp gtp1u-port 2152
gtp gtp1uremot-port 2152
apn S1U
gtpu-mode tun
tun-device S1Utun
type-support v4
ip prefix dynamic 10.1.1.0/24
ip dns 0 192.168.100.1
ip dns 1 8.8.8.8
ip ifconfig 10.1.1.2/24
no shutdown
default-apn S1U
no shutdown ggsn
sgsnemu
-l 127.0.0.2 -r 172.16.8.1 --contexts 1 -a S1U -m 4412345678 -q
0x00000000000b921f --defaultroute --pingcount 0 --gtpteid=8 --gtpversion=1
--gsnuip 172.16.1.1 --enduserip 10.1.1.1 --gtp0port 3387 --gtp1cport 2124
--nsapi 1
sgsnemu
-l 127.0.0.3 -r 172.16.8.1 --contexts 1 -a X2U -m 4412345678 -q
0x00000000000b921f --defaultroute --pingcount 0 --gtpteid=3 --gtpversion=1
--gsnuip 172.16.1.1 --enduserip 10.1.2.1 --gtp0port 3386 --gtp1cport 2123
--nsapi 2
Hopefully I have provided enough information. Any ideas on how to increase tunnel throughput?
Thanks
Dave