Hi,
I need to increase the throughput through the GTP tunnels to around 9Gb/s.
I have two servers and they are connected using two GTP tunnels over a Mellanox 25Gb/s link. Via the tunnels I am seeing a total of 4.2 Gb/s to 4.5 Gb/s using iPerf.
If data is sent without the tunnels I see a throughput of around 24.5Gb/s.
I have tried various tweaks to improve the throughput over the GTP tunnels and have only made marginal gains. I have tried iPerf 2.0.9 and iPerf3 and see the same results. Wireshark shows the iPerf MTU is reflected in the pcap logs.
I have tried changing the txqueuelen on the tunnels, changing osmo-ggsn niceness, network buffer sizes etc.
The restriction seems to be osmo-ggsn as that is taking up to 92% CPU on one of the cores when the throughput tests are running.
I'm hoping I have made a mistake in the configuration of OsmoGGSN and sgsnemu.
--
The servers are running Ubuntu 16.04 desktop. 16GB RAM - 12.6GB free when running throughput tests. Intel E52620v3. Mellanox MCX516A-CCAT network adapter. 25Gb direct attach copper connecting the two servers OsmoGGSN version 1.1.0.75-3e44-dirty.
For one of the servers: - Mellanox interface details ens3f0 Link encap:Ethernet HWaddr ec:0d:9a:a0:1d:7c inet addr:172.16.8.1 Bcast:172.16.255.255 Mask:255.255.0.0 inet6 addr: fe80::8f38:33e3:bd46:dd65/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:142373677 errors:0 dropped:0 overruns:0 frame:0 TX packets:157500406 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:10000 RX bytes:793825609487 (793.8 GB) TX bytes:879898325287 (879.8 GB)
Tunnel interface details S1Utun Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:10.1.1.2 P-t-P:10.1.1.2 Mask:255.255.255.0 inet6 addr: fe80::b3fc:3d43:b751:bcbf/64 Scope:Link UP POINTOPOINT RUNNING MTU:9000 Metric:1 RX packets:4640412 errors:0 dropped:0 overruns:0 frame:0 TX packets:2272797 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:37729672063 (37.7 GB) TX bytes:118866016 (118.8 MB)
X2Utun Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:10.1.2.2 P-t-P:10.1.2.2 Mask:255.255.255.0 inet6 addr: fe80::6ed8:8a79:b3e9:a51d/64 Scope:Link UP POINTOPOINT RUNNING MTU:9000 Metric:1 RX packets:1824011 errors:0 dropped:0 overruns:0 frame:0 TX packets:3856238 errors:0 dropped:4229 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:95648485 (95.6 MB) TX bytes:31367266755 (31.3 GB)
client side iperf server1 iperf3 -c 10.1.2.1 -i1 -t 30 -M8100 -w 2048K -P8 server2 iperf3 -c 10.1.2.1 -i1 -t 30 -M8100 -w 2048K -P8 server1 and 2 iperf server iperf3 -s -i1
osmo-ggsn -c /home/osmo-ggsn.cfg -g 172.16.1.1
contents of osmo-ggsn.cfg
!
! OpenGGSN (0.94.1-adac) configuration saved from vty
!!
!
log stderr
logging filter all 1
logging color 1
logging print category 0
logging timestamp 0
logging level ip info
logging level tun info
logging level ggsn info
logging level sgsn notice
logging level icmp6 notice
logging level lglobal notice
logging level llapd notice
logging level linp notice
logging level lmux notice
logging level lmi notice
logging level lmib notice
logging level lsms notice
logging level lctrl notice
logging level lgtp info
logging level lstats notice
logging level lgsup notice
logging level loap notice
logging level lss7 notice
logging level lsccp notice
logging level lsua notice
logging level lm3ua notice
logging level lmgcp notice
!
stats interval 5
!
line vty
no login
!
ggsn X2Uggsn
gtp state-dir /tmp
gtp bind-ip 172.16.8.1
gtp gtp0-port 3386
gtp gtp1c-port 2123
gtp gtp1u-port 2153
gtp gtp1uremot-port 2153
apn X2U
gtpu-mode tun
tun-device X2Utun
type-support v4
ip prefix dynamic 10.1.2.0/24
ip dns 0 192.168.100.1
ip dns 1 8.8.8.8
ip ifconfig 10.1.2.2/24
no shutdown
default-apn X2U
no shutdown ggsn
ggsn S1Uggsn
gtp state-dir /tmp
gtp bind-ip 172.16.8.1
gtp gtp0-port 3387
gtp gtp1c-port 2124
gtp gtp1u-port 2152
gtp gtp1uremot-port 2152
apn S1U
gtpu-mode tun
tun-device S1Utun
type-support v4
ip prefix dynamic 10.1.1.0/24
ip dns 0 192.168.100.1
ip dns 1 8.8.8.8
ip ifconfig 10.1.1.2/24
no shutdown
default-apn S1U
no shutdown ggsn
sgsnemu -l 127.0.0.2 -r 172.16.8.1 --contexts 1 -a S1U -m 4412345678 -q 0x00000000000b921f --defaultroute --pingcount 0 --gtpteid=8 --gtpversion=1 --gsnuip 172.16.1.1 --enduserip 10.1.1.1 --gtp0port 3387 --gtp1cport 2124 --nsapi 1
sgsnemu -l 127.0.0.3 -r 172.16.8.1 --contexts 1 -a X2U -m 4412345678 -q 0x00000000000b921f --defaultroute --pingcount 0 --gtpteid=3 --gtpversion=1 --gsnuip 172.16.1.1 --enduserip 10.1.2.1 --gtp0port 3386 --gtp1cport 2123 --nsapi 2
Hopefully I have provided enough information. Any ideas on how to increase tunnel throughput?
Thanks
Dave
Hi Dave,
I never personally used it, but as far as I know you should be able to get more performance by using "gtpu-mode kernel-gtp" instead of "gtpu-mode tun" in your config, to avoid doing all packet processing in userspace. Please someone else confirm this. I think you need a capable kernel and libgtpnl to be able to use it.
Also, it would be nice if you could provide a linux "perf" output on osmo-ggsn while running your test setup so we can spot possible optimizations to improve osmo-ggsn performance.
Hi Pau,
Thanks for the help so far. I have tried using the kernel-gtp mode and only managed to cause crashes in the omso-ggsn program. I have updated to the latest git repository for the osmo-ggsn.
I followed the instructions here https://osmocom.org/projects/openggsn/wiki/Kernel_GTP to load the kernel gtp module and the 'lsmod | grep gtp' shows the same as the results in the instructions.
After that the instructions mention using openggsn. I'm not sure if there is further configuration required for the kernel gtp module - do I need to create a tunnel? Or should this be done by osmo-ggsn as it did for the gtpu-mode tun setting?
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". <0002> ggsn.c:203 APN(S1U): Starting <0002> ggsn.c:281 APN(S1U): Opening Kernel GTP device gtp0 <0002> gtp-kernel.c:75 Initialized GTP kernel mode (genl ID is 30)
Program received signal SIGSEGV, Segmentation fault. gtp_kernel_init (gsn=0x0, devname=0x6a6fe0 "gtp0", prefix=prefix@entry=0x6a76e0, ipup=0x0) at gtp-kernel.c:94 94 if (gtp_dev_create(-1, devname, gsn->fd0, gsn->fd1u) < 0) {
I have tried, without success, to find a sample config file with the kernel-gtp mode in for osmo-ggsn. Is there a sample config file for osmo-ggsn with the kernel-gtp mode selected?
This is my current config file - I have gone through various attempts with different tun_device names.
! ! OpenGGSN (0.94.1-adac) configuration saved from vty !! ! ! kernel-gtp log stderr logging filter all 1 logging color 1 logging print category 0 logging timestamp 0 logging level ip info logging level tun info logging level ggsn info logging level sgsn notice logging level icmp6 notice logging level lglobal notice logging level llapd notice logging level linp notice logging level lmux notice logging level lmi notice logging level lmib notice logging level lsms notice logging level lctrl notice logging level lgtp info logging level lstats notice logging level lgsup notice logging level loap notice logging level lss7 notice logging level lsccp notice logging level lsua notice logging level lm3ua notice logging level lmgcp notice ! stats interval 5 ! line vty no login ! ggsn tun1 gtp state-dir /tmp gtp bind-ip 172.16.8.1 apn S1U gtpu-mode kernel-gtp tun-device gtp0 type-support v4 ip prefix dynamic 10.1.1.0/24 ip dns 0 192.168.100.1 ip dns 1 8.8.8.8 ip ifconfig 10.1.1.2/24 no shutdown default-apn S1U no shutdown ggsn
Thanks
Dave
On Thu, Mar 29, 2018 at 3:14 PM, Pau Espin Pedrol pespin@sysmocom.de wrote:
Hi Dave,
I never personally used it, but as far as I know you should be able to get more performance by using "gtpu-mode kernel-gtp" instead of "gtpu-mode tun" in your config, to avoid doing all packet processing in userspace. Please someone else confirm this. I think you need a capable kernel and libgtpnl to be able to use it.
Also, it would be nice if you could provide a linux "perf" output on osmo-ggsn while running your test setup so we can spot possible optimizations to improve osmo-ggsn performance.
--
- Pau Espin Pedrol pespin@sysmocom.de http://www.sysmocom.de/
=======================================================================
- sysmocom - systems for mobile communications GmbH
- Alt-Moabit 93
- 10559 Berlin, Germany
- Sitz / Registered office: Berlin, HRB 134158 B
- Geschaeftsfuehrer / Managing Director: Harald Welte
Hi,
On 04/04/18 16:59, Dave Ellis wrote:
After that the instructions mention using openggsn. I'm not sure if there is further configuration required for the kernel gtp module - do I need to create a tunnel? Or should this be done by osmo-ggsn as it did for the gtpu-mode tun setting?
I'm not sure here since I didn't find the opportunity to use the gtp kernel module myself yet. Looking at code in osmo-ggsn gtp-kernel.c, gtp_kernel_init() calls gtp_dev_create() which seems to create a GTP tunnel device, so osmo-ggsn seems to be creating the device.
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". <0002> ggsn.c:203 APN(S1U): Starting <0002> ggsn.c:281 APN(S1U): Opening Kernel GTP device gtp0 <0002> gtp-kernel.c:75 Initialized GTP kernel mode (genl ID is 30)
Program received signal SIGSEGV, Segmentation fault. gtp_kernel_init (gsn=0x0, devname=0x6a6fe0 "gtp0", prefix=prefix@entry=0x6a76e0, ipup=0x0) at gtp-kernel.c:94 94if (gtp_dev_create(-1, devname, gsn->fd0, gsn->fd1u) < 0) {
This looks like a bug in the code (gsn=NULL). Please open a ticket in redmine in OsmoGGSN project to remember to have a look at it. Also compiling everything with CFLAGS="-g -O0", then running under gdb and providing a full backtrace may help pinpoint the issue quicker.
Feel free to open a new task also about the bad performance when running without gtp kernel support and providing some performance figures, and if possible a linux perf output of the process.
osmocom-net-gprs@lists.osmocom.org
-
Dave Ellis -
Dave Ellis -
Pau Espin Pedrol