<div dir="ltr"><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Firstly I would like to say great thanks Firat for the reply, it certainly put me on a different investigation path. And apologies for not replying sooner. I wanted to make sure it was the correct path before I replied back to the group with the findings and associated solution.</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">If the GTP-U connection is connecting to the PG-W with a single IP at (src/dst) each side, and UDP flow has isn't been enabled on the network card of the host using gtp.ko in the kernel all the associated network traffic will be received on a single queue on the network, which is then serviced by a single ksoftirq thread. at somepoint the system will be receiving more traffic than there is available thread space to service the request, your ksoftirq will burn at 100%. That means all your traffic will be bound to a single network queue, bound to a single irq thread, and limit your overall throughput, no matter how big your network pipe is.</span></span></span><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">This is because the network card hashed the packet via SRC_IP:SRC_PORT:DEST_IP:DEST_PORT:PROTO to a single queue.</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"># take note on the discussions about udp-flow-hash udp4 using ethtool</span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><a href="https://home.regit.org/tag/performance/" target="_blank" rel="noreferrer">https://home.regit.org/tag/performance/</a></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><a href="https://www.joyent.com/blog/virtualizing-nics" target="_blank" rel="noreferrer">https://www.joyent.com/blog/virtualizing-nics</a><br></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><a href="https://www.serializing.me/2015/04/25/rxtx-buffers-rss-others-on-boot/" target="_blank" rel="noreferrer">https://www.serializing.me/2015/04/25/rxtx-buffers-rss-others-on-boot/</a></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">You can check if your card supports adjustable parameters by using "ethtool -k DEV | egrep -v fixed". As firat eludes to (below) udp flow hashing should be supported.</span></span></span></p></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">If you enabled UDP flow hash then it will spread the hash over multiple queues. The default number of queues on the network card can vary, depending on your hardware firmware driver, and any additional associated kernel parameters.</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Would recommend having the latest firmware driver for your network card, and latest kernel driver for the network card if possible.</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Alas the network cards used by my hardware didn't support flow hash, it had intel flow director, which wasn't granular enough and worked with TCP, so to work around this limitation having multiple SRC_IPs in different name spaces with the same GTP UDP PORT numbers resolved the problem. Of course if you are sending GTP-U to a single destination from multiple sources (say 6 IP's), via 6 different kernel name spaces, you spread the load over 6 queues, which is better than nothing on a limited feature network card. Time to upgrade the 10G network card....<br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">This took the system from 100% ksoftirq on a single cpu running at throughput 1G, to around 7 to 8GIG throughput at 90% ksoftirq over multiple cpu;s... There is still massive room for improvement.<br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">For performance some things to investigate/consider... Which I had different levels of success... Here are my ramblings.....<br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">on the linux host... Assuming your traffic is now spread across multiple queues (above) - or at least spread as best as can be...</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Kernel sysctl tweaking is always of benefit, if your using out of the box kernel config... Example udp buffers, queue sizes, paging and virtual memory settings... There is a application called "tuned", which allows you to adjust profiles for the kernel sysctl... My performance profile which suited the testing best was "throuhput-performance"<br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">if your looking for straight performance, disable audit processing like "auditd".</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Question use of SELINUX, enforcing/permissive or disabled. can bring results on performance, if you doing testing or load testing... ofcourse its a security consideration.. <br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">If you don't need to use ipfilters/firewall in my case can increase the throughput by a 3rd by disabling (cleaning the filter tables and unloading the modules). Black listing the modules so they dont get loaded at kernel time. Note you can stop modules getting loaded with kernel.modules_disabled=1, but be careful if your also messing with initramfs rebuilds, because you don't get any modules once your set that parameter, i learnt the hard way :)<br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Investigate smp_affinity and affinity_hint, along with irqbalane using the --hintpolicy=exact. understand which irq's service the network cards, and how many queues you have.. /proc/interrupts will guide you (grep 'CPU|rxtx' /proc/interrupts)... understand the smp_affinity numbers.. "for ((irq=START_IRQ; irq<=END_IRQ; irq++)); do cat /proc/irq/$irq/smp_affinity; done | sort –u", as you can adjust which queue goes to which ksoftirq to manually balance the queues if you so desire.</span></span></span><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"> brilliant document on irq debugging....<a href="https://events.static.linuxfound.org/sites/events/files/slides/LinuxConJapan2016_makita_160714.pdf" target="_blank" rel="noreferrer"> https://events.static.linuxfound.org/sites/events/files/slides/LinuxConJapan2016_makita_160714.pdf</a></span></span></span>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">you can monitor what calls are been executed on cpu's by using... I found this most useful to understand that ipfilter was eating a significant amount of CPU cycles, and also what other calls are eating up cycles inside ksoftirq.<a href="https://github.com/brendangregg/FlameGraph" target="_blank" rel="noreferrer"> https://github.com/brendangregg/FlameGraph</a></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Investigate additional memory management using numactl (numa daemon). remember if you are using virtualisation you might want to pin guests to specific sockets, along with numa pinning on the vmhost.. Also look at reserved memory allocation in the vmhost for the guest... This will make your guest perform better.</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">enable sysstat (sar) as it will aid your investigation if you havent already (sar -u ALL -P ALL 1). This will show which softirqs are eating most cpu and to which cpu they are bound, this also translates directly to the network queue that the traffic is coming in on.. Ie, network card queue 6, talks to cpu/6 talking to irq/6 and so on... Using flamegraph will help you understand what syscalls and chewing the CPU..<br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">If your using virtualisation then the number of default queues that vxnet (vmware in this example) presents to the guest might be less than the number of network card queues the vmhost sees (so watch out for that). You can adjust the number of queues to the guest by params in the vmware network driver... investigate VDMQ / netqueue, to increase the number of available hardware queues from the vmhost to the guest. depending which quest driver your using vxnet3, or others some drivers dont support NAPI (see further down).<br></span></span></span></div><div><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> VMDQ: array of int</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default=8)</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> RSS: array of int</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> Number of Receive-Side Scaling Descriptor Queues, default 1=number of cpus</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> MQ: array of int</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> Disable or enable Multiple Queues, default 1</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"> Node: array of int</span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"> set the starting node to allocate memory on, default -1</span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> IntMode: array of int</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> InterruptType: array of int</b></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b> Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated)</b></span></span></span></p></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Make sure your virtual switch (vmware) if used has <span class="gmail-stack-view-label"><span style="font-size:9pt">Pass-through
(Direct-path I/O) enabled. NIC teaming policy should be validated
depending on your requirement, example Policy "route based on IP hash"
can be of benefit.</span></span></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Check the network card is MSI-X, and the linux driver supports NAPI (most should these days, but you never know), also check your vmhost driver supports napi, if not get a NAPI supported kvm driver, or vmware driver (vib update).</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Upgrade your kernel, to a later release 4.x.. even consider using a later distro of linux... I tried fedora 29. I also compiled latest osmocom from source, with compile options for "optimisation -O3 and other such".<br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">"bmon -b" was a good tool understand throughput loads, along with loading through qdisc/fq_dodel mq's.... Understand qdisc via ip link or ifconfig (<a href="http://tldp.org/HOWTO/Traffic-Control-HOWTO/components.html">http://tldp.org/HOWTO/Traffic-Control-HOWTO/components.html</a>), adjusting the queues has some traction, but if unsure leave as default.</span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">TSO/UFo?GSO/LRO/GRO - understand your network card with respects to these, this can improve performance if you haven't already enabled (or adversely disabled options, since sometimes it doesn't actually help). You can get the your card options using ethool<br></span></span></span></div><div>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">TCP Segmentation Offload (TSO)</span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"> Uses the TCP protocol to send large packets. Uses the NIC to handle segmentation, and then adds the TCP, IP and data link layer protocol headers to each segment. </span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">UDP Fragmentation Offload (UFO)</span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"> Uses the UDP protocol to send large packets. Uses the NIC to handle IP fragmentation into MTU sized packets for large UDP datagrams. </span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Generic Segmentation Offload (GSO)</span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"> Uses the TCP or UDP protocol to send large packets. If the NIC cannot handle segmentation/fragmentation, GSO performs the same operations, bypassing the NIC hardware. This is achieved by delaying segmentation until as late as possible, for example, when the packet is processed by the device driver. </span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Large Receive Offload (LRO)</span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"> Uses the TCP protocol. All incoming packets are re-segmented as they are received, reducing the number of segments the system has to process. They can be merged either in the driver or using the NIC. A problem with LRO is that it tends to resegment all incoming packets, often ignoring differences in headers and other information which can cause errors. It is generally not possible to use LRO when IP forwarding is enabled. LRO in combination with IP forwarding can lead to checksum errors. Forwarding is enabled if /proc/sys/net/ipv4/ip_forward is set to 1. </span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Generic Receive Offload (GRO)</span></span></span></pre>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"> Uses either the TCP or UDP protocols. GRO is more rigorous than LRO when resegmenting packets. For example it checks the MAC headers of each packet, which must match, only a limited number of TCP or IP headers can be different, and the TCP timestamps must match. Resegmenting can be handled by either the NIC or the GSO code.</span></span></span></pre>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Traffic steering was on by default with the version of linux i was using, but worth checking if your using older versions.</span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><a href="https://www.kernel.org/doc/Documentation/networking/scaling.txt" target="_blank" rel="noreferrer">https://www.kernel.org/doc/Documentation/networking/scaling.txt</a></span></span></span></p>
<pre><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">(from the txt link) note: Some advanced NICs allow steering packets to queues based on programmable filters. For example, webserver bound TCP port 80 packets can be directed to their own receive queue. Such “n-tuple†filters can be configured from ethtool (--config-ntuple).</span></span></span></pre><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"></span></span></span></p></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Interestingly investigate your network card, for its hashing algorithms, how it distributes the traffic over its ring buffers, you can on some cards adjust the RSS hash function. Alas the card i was using stuck to "toeplitz" for hits hashing, which others were disabled and unavailable / xor and crc32. The indirection table can be adjusted based on the tuplets "ethtool -X" but didn't really assist too much on this.</span></span></span></div><div><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt">ethtool -x <dev></span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt">RX flow hash indirection table for ens192 with 8 RX ring(s):</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt"> 0: 0 1 2 3 4 5 6 7</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt"> 8: 0 1 2 3 4 5 6 7</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt"> 16: 0 1 2 3 4 5 6 7</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt"> 24: 0 1 2 3 4 5 6 7</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt">RSS hash key:</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt">Operation not supported</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt">RSS hash function:</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt"> toeplitz: on</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt"> xor: off</span></span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-size:8pt"> crc32: off</span></span></span></span></p></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Check the default size of the rx/tx ring buffers, they maybe suboptimal.<br></span></span></span></div><div><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">ethtool -g ens192
</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Ring parameters for ens192:</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Pre-set maximums:</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">RX: 4096</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">RX Mini: 0</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">RX Jumbo: 4096</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">TX: 4096</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Current hardware settings:</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">RX: 1024</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">RX Mini: 0</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">RX Jumbo: 256</span></span></span></p><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">
</span></span></span><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">TX: 512</span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><br></span></span></span></p><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt">If your using port channels, make sure you have the correct hashing policy enabled at the switch end...</span></span></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt"><br></span></span></span></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">I
haven't investigated this option yet but some switches also do scaling,
to assist (certainly with virtualisation)... Maybe one day i will get
around to this...<br></span></span></span></div><div><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Additionally CISCO describe that you should have VM-FEX optimisation</span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><a href="https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/vm_fex_best_practices_deployment_guide.html" target="_blank" rel="noreferrer">https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/vm_fex_best_practices_deployment_guide.html</a> <br></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">note:</span></span></span></p>
<p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b><span style="font-size:12pt">table 4.</span></b><b><span style="font-size:7pt">
</span></b><span style="font-size:12pt">Scaling
of Dynamic vNIC with VMDirectPath, Virtual Machines Running on Linux
Guest with VMXNET3 Emulated Driver and Multi-Queue Enabled</span></span></span></span></p>
<p class="gmail-ptablecaptioncmt"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><b>Table 5.</b><b><span style="font-size:7pt"> </span>
</b>Scaling of Dynamic vNIC with VMDirectPath, Virtual Machines Running
on Linux Guest with VMXNET3 Emulated Driver and Multi-Queue Disabled</span></span></span></p></div><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt"></span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt"><br></span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt"><br></span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt">Another thing to consider/investigate - openvswitch/bridging... If your using eth pairs to send your traffic down name spaces... you can have some varied results with performance by trying openvswitch/brctl</span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt"><br></span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt"><br></span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt"><br></span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt">I really enjoyed the investigation path, again thanks to Firat for the pointer, otherwise it would have taken longer to get the answer...</span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span class="gmail-stack-view-label"><span style="font-size:9pt"><br></span></span></span></span></span></p><p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif;color:rgb(31,73,125)"><span class="gmail-stack-view-label"><span style="font-size:9pt;font-family:"Helvetica",sans-serif;color:rgb(51,51,51)"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)">Tony</span></span></span><br></span></span></span></p><p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif;color:rgb(31,73,125)"><span class="gmail-stack-view-label"><span style="font-size:9pt;font-family:"Helvetica",sans-serif;color:rgb(51,51,51)"><br></span></span></span></p></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 21, 2019 at 6:50 AM fırat sönmez <<a href="mailto:firatssonmez@gmail.com">firatssonmez@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>It has been over 2 years that I have worked with gtp and I kind of had the same problem that time, we had a 10gbit cable and tried to see how much udp flow we could get. I think we used iperf to test it and when we list all the processes, the ksoftirq was using all the resource. Then I found this page: <a href="https://blog.cloudflare.com/how-to-receive-a-million-packets/" target="_blank">https://blog.cloudflare.com/how-to-receive-a-million-packets/</a>. I do not remember the exact solution, but I guess when you configure your out ethernet interface with the command below, it must work then. To my understanding all the packets are processed in the same core in your situation, because the port number is always the same. So, for example, if you add another network with gtp-u tunnel on another port (different than 3386) then again your packets will be processed on the other core, too. But with the below command, the interface will be configured in a way that it wont check the port to process on which core it should be processed, but it will use the hash from the packet to distribute over the cores.</div><div><span style="background-color:initial;font-family:Consolas,Monaco,"Andale Mono","Ubuntu Mono",monospace;font-size:13px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit;word-spacing:normal;color:black">ethtool -n (your_out_eth_interface) rx-flow-hash udp4 </span></div><div><span style="background-color:initial;font-family:Consolas,Monaco,"Andale Mono","Ubuntu Mono",monospace;font-size:13px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit;word-spacing:normal;color:black"><br></span></div><div><span style="background-color:initial;font-family:Consolas,Monaco,"Andale Mono","Ubuntu Mono",monospace;font-size:13px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit;word-spacing:normal;color:black">Hope it will work you.</span></div><div><span style="background-color:initial;font-family:Consolas,Monaco,"Andale Mono","Ubuntu Mono",monospace;font-size:13px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit;word-spacing:normal;color:black"><br></span></div><div><span style="background-color:initial;font-family:Consolas,Monaco,"Andale Mono","Ubuntu Mono",monospace;font-size:13px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit;word-spacing:normal;color:black">Fırat</span></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Tony Clark <<a href="mailto:chiefy.padua@gmail.com" target="_blank">chiefy.padua@gmail.com</a>>, 19 Haz 2019 Çar, 15:07 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear All,<br><br>I've been using the GTP-U kernel module to communicate with a P-GW.<br><br>Running Fedora 29, kernel 4.18.16-300.fc29.x86_64.<br><br>At high traffic levels through the GTP-U tunnel I see the performance degrade as 100% CPU is consumed by a single ksoftirqd process.<br><br>It is running on a multi-cpu machine and as far as I can tell the load is evenly spread across the cpus (ie either manually via smp_affinity, or even irqbalance, checking /proc/interrupts so forth.).<br><br>Has anyone else experienced this?<br><br>Is there any particular area you could recommend I investigate to find the root cause of this bottleneck, as i'm starting to scratch my head where to look next...<br><br>Thanks in advance<br>Tony<br> <br>---- FYI<br><br>modinfo gtp<br>filename: /lib/modules/4.18.16-300.fc29.x86_64/kernel/drivers/net/gtp.ko.xz<br>alias: net-pf-16-proto-16-family-gtp<br>alias: rtnl-link-gtp<br>description: Interface driver for GTP encapsulated traffic<br>author: Harald Welte <<a href="mailto:hwelte@sysmocom.de" target="_blank">hwelte@sysmocom.de</a>><br>license: GPL<br>depends: udp_tunnel<br>retpoline: Y<br>intree: Y<br>name: gtp<br>vermagic: 4.18.16-300.fc29.x86_64 SMP mod_unload <br><br>modinfo udp_tunnel<br>filename: /lib/modules/4.18.16-300.fc29.x86_64/kernel/net/ipv4/udp_tunnel.ko.xz<br>license: GPL<br>depends: <br>retpoline: Y<br>intree: Y<br>name: udp_tunnel<br>vermagic: 4.18.16-300.fc29.x86_64 SMP mod_unload <br></div>
</blockquote></div>
</blockquote></div></div>