Data throughput issues. Increasing throughput.

Dave Ellis ssuellid at gmail.com
Thu Mar 29 13:44:23 UTC 2018


Hi,

I need to increase the throughput through the GTP tunnels to around 9Gb/s.

I have two servers and they are connected using two GTP tunnels over a
Mellanox 25Gb/s link.  Via the tunnels I am seeing a total of 4.2 Gb/s to
4.5 Gb/s using iPerf.

If data is sent without the tunnels I see a throughput of around 24.5Gb/s.

I have tried various tweaks to improve the throughput over the GTP tunnels
and have only made marginal gains. I have tried iPerf 2.0.9 and iPerf3 and
see the same results. Wireshark shows the iPerf MTU is reflected in the
pcap logs.

I have tried changing the txqueuelen on the tunnels, changing osmo-ggsn
niceness, network buffer sizes etc.

The restriction seems to be osmo-ggsn as that is taking up to 92% CPU on
one of the cores when the throughput tests are running.

I'm hoping I have made a mistake in the configuration of OsmoGGSN and
sgsnemu.

--

The servers are running Ubuntu 16.04 desktop. 16GB RAM - 12.6GB free when
running throughput tests. Intel E52620v3. Mellanox MCX516A-CCAT network
adapter. 25Gb direct attach copper connecting the two servers  OsmoGGSN
version 1.1.0.75-3e44-dirty.

For one of the servers: -
Mellanox interface details
ens3f0    Link encap:Ethernet  HWaddr ec:0d:9a:a0:1d:7c
          inet addr:172.16.8.1  Bcast:172.16.255.255  Mask:255.255.0.0
          inet6 addr: fe80::8f38:33e3:bd46:dd65/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:142373677 errors:0 dropped:0 overruns:0 frame:0
          TX packets:157500406 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10000
          RX bytes:793825609487 (793.8 GB)  TX bytes:879898325287 (879.8 GB)

Tunnel interface details
S1Utun    Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.1.1.2  P-t-P:10.1.1.2  Mask:255.255.255.0
          inet6 addr: fe80::b3fc:3d43:b751:bcbf/64 Scope:Link
          UP POINTOPOINT RUNNING  MTU:9000  Metric:1
          RX packets:4640412 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2272797 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:37729672063 (37.7 GB)  TX bytes:118866016 (118.8 MB)

X2Utun    Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.1.2.2  P-t-P:10.1.2.2  Mask:255.255.255.0
          inet6 addr: fe80::6ed8:8a79:b3e9:a51d/64 Scope:Link
          UP POINTOPOINT RUNNING  MTU:9000  Metric:1
          RX packets:1824011 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3856238 errors:0 dropped:4229 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:95648485 (95.6 MB)  TX bytes:31367266755 (31.3 GB)


client side iperf
server1
      iperf3 -c 10.1.2.1  -i1 -t 30 -M8100 -w 2048K  -P8
server2
      iperf3 -c 10.1.2.1  -i1 -t 30 -M8100 -w 2048K  -P8
server1 and 2 iperf server
    iperf3 -s -i1


osmo-ggsn -c /home/osmo-ggsn.cfg -g 172.16.1.1



contents of osmo-ggsn.cfg



!

! OpenGGSN (0.94.1-adac) configuration saved from vty

!!

!

log stderr

  logging filter all 1

  logging color 1

  logging print category 0

  logging timestamp 0

  logging level ip info

  logging level tun info

  logging level ggsn info

  logging level sgsn notice

  logging level icmp6 notice

  logging level lglobal notice

  logging level llapd notice

  logging level linp notice

  logging level lmux notice

  logging level lmi notice

  logging level lmib notice

  logging level lsms notice

  logging level lctrl notice

  logging level lgtp info

  logging level lstats notice

  logging level lgsup notice

  logging level loap notice

  logging level lss7 notice

  logging level lsccp notice

  logging level lsua notice

  logging level lm3ua notice

  logging level lmgcp notice

!

stats interval 5

!

line vty

no login

!

ggsn X2Uggsn

gtp state-dir /tmp

gtp bind-ip 172.16.8.1

gtp gtp0-port 3386

gtp gtp1c-port 2123

gtp gtp1u-port 2153

gtp gtp1uremot-port 2153

apn X2U

  gtpu-mode tun

  tun-device X2Utun

  type-support v4

  ip prefix dynamic 10.1.2.0/24

  ip dns 0 192.168.100.1

  ip dns 1 8.8.8.8

  ip ifconfig 10.1.2.2/24

  no shutdown

default-apn X2U

no shutdown ggsn

ggsn S1Uggsn

gtp state-dir /tmp

gtp bind-ip 172.16.8.1

gtp gtp0-port 3387

gtp gtp1c-port 2124

gtp gtp1u-port 2152

gtp gtp1uremot-port 2152

apn S1U

  gtpu-mode tun

  tun-device S1Utun

  type-support v4

  ip prefix dynamic 10.1.1.0/24

  ip dns 0 192.168.100.1

  ip dns 1 8.8.8.8

  ip ifconfig 10.1.1.2/24

  no shutdown

default-apn S1U

no shutdown ggsn





sgsnemu -l 127.0.0.2 -r 172.16.8.1 --contexts 1 -a S1U -m 4412345678 -q
0x00000000000b921f --defaultroute --pingcount 0 --gtpteid=8 --gtpversion=1
--gsnuip 172.16.1.1 --enduserip 10.1.1.1 --gtp0port 3387 --gtp1cport 2124
--nsapi 1





sgsnemu -l 127.0.0.3 -r 172.16.8.1 --contexts 1 -a X2U -m 4412345678 -q
0x00000000000b921f --defaultroute --pingcount 0 --gtpteid=3 --gtpversion=1
--gsnuip 172.16.1.1 --enduserip 10.1.2.1 --gtp0port 3386 --gtp1cport 2123
--nsapi 2

Hopefully I have provided enough information. Any ideas on how to increase
tunnel throughput?

Thanks

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/osmocom-net-gprs/attachments/20180329/c173f92a/attachment-0001.html>


More information about the osmocom-net-gprs mailing list