29 Mar
2018
29 Mar
'18
3:44 p.m.
Hi,
I need to increase the throughput through the GTP tunnels to around 9Gb/s.
I have two servers and they are connected using two GTP tunnels over a
Mellanox 25Gb/s link. Via the tunnels I am seeing a total of 4.2 Gb/s to
4.5 Gb/s using iPerf.
If data is sent without the tunnels I see a throughput of around 24.5Gb/s.
I have tried various tweaks to improve the throughput over the GTP tunnels
and have only made marginal gains. I have tried iPerf 2.0.9 and iPerf3 and
see the same results. Wireshark shows the iPerf MTU is reflected in the
pcap logs.
I have tried changing the txqueuelen on the tunnels, changing osmo-ggsn
niceness, network buffer sizes etc.
The restriction seems to be osmo-ggsn as that is taking up to 92% CPU on
one of the cores when the throughput tests are running.
I'm hoping I have made a mistake in the configuration of OsmoGGSN and
sgsnemu.
--
The servers are running Ubuntu 16.04 desktop. 16GB RAM - 12.6GB free when
running throughput tests. Intel E52620v3. Mellanox MCX516A-CCAT network
adapter. 25Gb direct attach copper connecting the two servers OsmoGGSN
version 1.1.0.75-3e44-dirty.
For one of the servers: -
Mellanox interface details
ens3f0 Link encap:Ethernet HWaddr ec:0d:9a:a0:1d:7c
inet addr:172.16.8.1 Bcast:172.16.255.255 Mask:255.255.0.0
inet6 addr: fe80::8f38:33e3:bd46:dd65/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:142373677 errors:0 dropped:0 overruns:0 frame:0
TX packets:157500406 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:10000
RX bytes:793825609487 (793.8 GB) TX bytes:879898325287 (879.8 GB)
Tunnel interface details
S1Utun Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.1.1.2 P-t-P:10.1.1.2 Mask:255.255.255.0
inet6 addr: fe80::b3fc:3d43:b751:bcbf/64 Scope:Link
UP POINTOPOINT RUNNING MTU:9000 Metric:1
RX packets:4640412 errors:0 dropped:0 overruns:0 frame:0
TX packets:2272797 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:37729672063 (37.7 GB) TX bytes:118866016 (118.8 MB)
X2Utun Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.1.2.2 P-t-P:10.1.2.2 Mask:255.255.255.0
inet6 addr: fe80::6ed8:8a79:b3e9:a51d/64 Scope:Link
UP POINTOPOINT RUNNING MTU:9000 Metric:1
RX packets:1824011 errors:0 dropped:0 overruns:0 frame:0
TX packets:3856238 errors:0 dropped:4229 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:95648485 (95.6 MB) TX bytes:31367266755 (31.3 GB)
client side iperf
server1
iperf3 -c 10.1.2.1 -i1 -t 30 -M8100 -w 2048K -P8
server2
iperf3 -c 10.1.2.1 -i1 -t 30 -M8100 -w 2048K -P8
server1 and 2 iperf server
iperf3 -s -i1
osmo-ggsn -c /home/osmo-ggsn.cfg -g 172.16.1.1
contents of osmo-ggsn.cfg
!
! OpenGGSN (0.94.1-adac) configuration saved from vty
!!
!
log stderr
logging filter all 1
logging color 1
logging print category 0
logging timestamp 0
logging level ip info
logging level tun info
logging level ggsn info
logging level sgsn notice
logging level icmp6 notice
logging level lglobal notice
logging level llapd notice
logging level linp notice
logging level lmux notice
logging level lmi notice
logging level lmib notice
logging level lsms notice
logging level lctrl notice
logging level lgtp info
logging level lstats notice
logging level lgsup notice
logging level loap notice
logging level lss7 notice
logging level lsccp notice
logging level lsua notice
logging level lm3ua notice
logging level lmgcp notice
!
stats interval 5
!
line vty
no login
!
ggsn X2Uggsn
gtp state-dir /tmp
gtp bind-ip 172.16.8.1
gtp gtp0-port 3386
gtp gtp1c-port 2123
gtp gtp1u-port 2153
gtp gtp1uremot-port 2153
apn X2U
gtpu-mode tun
tun-device X2Utun
type-support v4
ip prefix dynamic 10.1.2.0/24
ip dns 0 192.168.100.1
ip dns 1 8.8.8.8
ip ifconfig 10.1.2.2/24
no shutdown
default-apn X2U
no shutdown ggsn
ggsn S1Uggsn
gtp state-dir /tmp
gtp bind-ip 172.16.8.1
gtp gtp0-port 3387
gtp gtp1c-port 2124
gtp gtp1u-port 2152
gtp gtp1uremot-port 2152
apn S1U
gtpu-mode tun
tun-device S1Utun
type-support v4
ip prefix dynamic 10.1.1.0/24
ip dns 0 192.168.100.1
ip dns 1 8.8.8.8
ip ifconfig 10.1.1.2/24
no shutdown
default-apn S1U
no shutdown ggsn
sgsnemu -l 127.0.0.2 -r 172.16.8.1 --contexts 1 -a S1U -m 4412345678 -q
0x00000000000b921f --defaultroute --pingcount 0 --gtpteid=8 --gtpversion=1
--gsnuip 172.16.1.1 --enduserip 10.1.1.1 --gtp0port 3387 --gtp1cport 2124
--nsapi 1
sgsnemu -l 127.0.0.3 -r 172.16.8.1 --contexts 1 -a X2U -m 4412345678 -q
0x00000000000b921f --defaultroute --pingcount 0 --gtpteid=3 --gtpversion=1
--gsnuip 172.16.1.1 --enduserip 10.1.2.1 --gtp0port 3386 --gtp1cport 2123
--nsapi 2
Hopefully I have provided enough information. Any ideas on how to increase
tunnel throughput?
Thanks
Dave
29 Mar
29 Mar
4:14 p.m.
Hi Dave,
I never personally used it, but as far as I know you should be able to
get more performance by using "gtpu-mode kernel-gtp" instead of
"gtpu-mode tun" in your config, to avoid doing all packet processing in
userspace. Please someone else confirm this. I think you need a capable
kernel and libgtpnl to be able to use it.
Also, it would be nice if you could provide a linux "perf" output on
osmo-ggsn while running your test setup so we can spot possible
optimizations to improve osmo-ggsn performance.
--
- Pau Espin Pedrol <pespin(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
4 Apr
4 Apr
4:59 p.m.
Hi Pau,
Thanks for the help so far. I have tried using the kernel-gtp mode and
only managed to cause crashes in the omso-ggsn program. I have updated to
the latest git repository for the osmo-ggsn.
I followed the instructions here
https://osmocom.org/projects/openggsn/wiki/Kernel_GTP to load the kernel
gtp module and the 'lsmod | grep gtp' shows the same as the results in the
instructions.
After that the instructions mention using openggsn. I'm not sure if there
is further configuration required for the kernel gtp module - do I need to
create a tunnel? Or should this be done by osmo-ggsn as it did for the
gtpu-mode tun setting?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
<0002> ggsn.c:203 APN(S1U): Starting
<0002> ggsn.c:281 APN(S1U): Opening Kernel GTP device gtp0
<0002> gtp-kernel.c:75 Initialized GTP kernel mode (genl ID is 30)
Program received signal SIGSEGV, Segmentation fault.
gtp_kernel_init (gsn=0x0, devname=0x6a6fe0 "gtp0",
prefix=prefix@entry=0x6a76e0, ipup=0x0) at gtp-kernel.c:94
94 if (gtp_dev_create(-1, devname, gsn->fd0, gsn->fd1u) < 0) {
I have tried, without success, to find a sample config file with the
kernel-gtp mode in for osmo-ggsn. Is there a sample config file for
osmo-ggsn with the kernel-gtp mode selected?
This is my current config file - I have gone through various attempts with
different tun_device names.
!
! OpenGGSN (0.94.1-adac) configuration saved from vty
!!
!
! kernel-gtp
log stderr
logging filter all 1
logging color 1
logging print category 0
logging timestamp 0
logging level ip info
logging level tun info
logging level ggsn info
logging level sgsn notice
logging level icmp6 notice
logging level lglobal notice
logging level llapd notice
logging level linp notice
logging level lmux notice
logging level lmi notice
logging level lmib notice
logging level lsms notice
logging level lctrl notice
logging level lgtp info
logging level lstats notice
logging level lgsup notice
logging level loap notice
logging level lss7 notice
logging level lsccp notice
logging level lsua notice
logging level lm3ua notice
logging level lmgcp notice
!
stats interval 5
!
line vty
no login
!
ggsn tun1
gtp state-dir /tmp
gtp bind-ip 172.16.8.1
apn S1U
gtpu-mode kernel-gtp
tun-device gtp0
type-support v4
ip prefix dynamic 10.1.1.0/24
ip dns 0 192.168.100.1
ip dns 1 8.8.8.8
ip ifconfig 10.1.1.2/24
no shutdown
default-apn S1U
no shutdown ggsn
Thanks
Dave
On Thu, Mar 29, 2018 at 3:14 PM, Pau Espin Pedrol <pespin(a)sysmocom.de>
wrote:
Hi Dave,
I never personally used it, but as far as I know you should be able to get
more performance by using "gtpu-mode kernel-gtp" instead of "gtpu-mode
tun"
in your config, to avoid doing all packet processing in userspace. Please
someone else confirm this. I think you need a capable kernel and libgtpnl
to be able to use it.
Also, it would be nice if you could provide a linux "perf" output on
osmo-ggsn while running your test setup so we can spot possible
optimizations to improve osmo-ggsn performance.
--
- Pau Espin Pedrol <pespin(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
5 Apr
5 Apr
11:58 a.m.
Hi,
On 04/04/18 16:59, Dave Ellis wrote:
After that the instructions mention using openggsn.
I'm not sure if
there is further configuration required for the kernel gtp module - do I
need to create a tunnel? Or should this be done by osmo-ggsn as it did
for the gtpu-mode tun setting?
I'm not sure here since I didn't find the opportunity to use the gtp
kernel module myself yet. Looking at code in osmo-ggsn gtp-kernel.c,
gtp_kernel_init() calls gtp_dev_create() which seems to create a GTP
tunnel device, so osmo-ggsn seems to be creating the device.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
<0002> ggsn.c:203 APN(S1U): Starting
<0002> ggsn.c:281 APN(S1U): Opening Kernel GTP device gtp0
<0002> gtp-kernel.c:75 Initialized GTP kernel mode (genl ID is 30)
Program received signal SIGSEGV, Segmentation fault.
gtp_kernel_init (gsn=0x0, devname=0x6a6fe0 "gtp0",
prefix=prefix@entry=0x6a76e0, ipup=0x0) at gtp-kernel.c:94
94if (gtp_dev_create(-1, devname, gsn->fd0, gsn->fd1u) < 0) {
This looks like a bug in the code (gsn=NULL). Please open a ticket in
redmine in OsmoGGSN project to remember to have a look at it. Also
compiling everything with CFLAGS="-g -O0", then running under gdb and
providing a full backtrace may help pinpoint the issue quicker.
Feel free to open a new task also about the bad performance when running
without gtp kernel support and providing some performance figures, and
if possible a linux perf output of the process.
--
- Pau Espin Pedrol <pespin(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
2220
days inactive
2227
days old
osmocom-net-gprs@lists.osmocom.org
3 comments
3 participants
participants (3)
-
Dave Ellis -
Dave Ellis -
Pau Espin Pedrol