pespin has uploaded this change for review.
testsuites: upf: Support setting net iface irqs in c240 host
The Cisco C240 machines consist of 2 CPU packages, each with 20 cores
(hyperthreading disabled). Those 2 CPUs packages are hence placed in 2
different NUMA zones.
By default, mlx5 driver creates one rx-queue per core, in this case it
creates 40 rx-queues with 40 irqs (one for each rx-queue/core).
This means it doesn't take into account the fact that the network card
is plugged into a given PCIe bus belonging to one of the 2 available
NUMA zones.
As a result, when a packet is received and put into the rx-queue+irq
belonging to the other NUMA zone, a penalty in memory-bandwitch is
incurred, which ends up in a performance throughput penalty.
With 100 flows at ~70Gbps being sent to the host, by default load spread
among all 40 cores can reach processing at 55Gbps 4.62 MPPS.
By decreasing the rx-queues to 20 and pinning the irqs to the cores in
the same NUMA node, using only those 20 CPUs we get a small gain
reaching ~62 Gbps 5.22 MPPS.
Change-Id: I26fce50c04b043b61ba418d7090b2573e7807b08
---
M testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh
1 file changed, 30 insertions(+), 3 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/upf-benchmark refs/changes/46/39846/1
diff --git a/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh b/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh
index 825e59f..947c465 100755
--- a/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh
+++ b/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh
@@ -1,8 +1,33 @@
#!/bin/bash -xe
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
# Interface towards UPF:
-IFACE_RAN="enp2s0f1np1"
-IFACE_CN="enp2s0f0np0"
+case $(hostname) in
+gtplab1)
+ IFACE_RAN="enp2s0f1np1"
+ IFACE_CN="enp2s0f0np0"
+ MAC_CN_PEER="ec:0d:9a:8a:27:53"
+ ;;
+c240-2)
+ IFACE_RAN="ens2f0np0"
+ IFACE_CN="ens2f1np1"
+ MAC_CN_PEER="88:e9:a4:3b:f0:49"
+ # Adapt Rx/Tx Queues and IRQs to Numa Node of device:
+ # cat /sys/class/net/$IFACE_RAN/device/numa_node
+ # 0
+ # cat /sys/class/net/$IFACE_RAN/device/local_cpulist
+ # 0-19
+ sudo ethtool -L $IFACE_RAN combined 20
+ sudo ethtool -L $IFACE_CN combined 20
+ sudo "$SCRIPT_DIR/../../../../scripts/set_irq_affinity_cpulist.sh" 0-19 $IFACE_RAN
+ sudo "$SCRIPT_DIR/../../../../scripts/set_irq_affinity_cpulist.sh" 0-19 $IFACE_CN
+ ;;
+*)
+ echo "UNEXPECTED HOSTNAME: $(hostname)"
+ exit 1
+ ;;
+esac
# Disable ethernet flow control:
sudo ethtool -A $IFACE_RAN autoneg off rx off tx off
@@ -26,7 +51,9 @@
# Set up UPF address:
sudo ip addr add 172.16.32.1/24 dev $IFACE_RAN || true;
+sudo ip link set up dev $IFACE_RAN || true;
sudo ip addr add 172.16.31.2/24 dev $IFACE_CN || true;
+sudo ip link set up dev $IFACE_CN || true;
# TRex doesn't answer ARPs, so we need to set up the CN-side peer:
-sudo ip neigh add 172.16.31.200 lladdr ec:0d:9a:8a:27:53 nud permanent dev $IFACE_CN || true;
+sudo ip neigh add 172.16.31.200 lladdr "$MAC_CN_PEER" nud permanent dev $IFACE_CN || true;
To view, visit change 39846. To unsubscribe, or for help writing mail filters, visit settings.