laforge has submitted this change. ( https://gerrit.osmocom.org/c/upf-benchmark/+/39846?usp=email ) Change subject: testsuites: upf: Support setting net iface irqs in c240 host ...................................................................... testsuites: upf: Support setting net iface irqs in c240 host The Cisco C240 machines consist of 2 CPU packages, each with 20 cores (hyperthreading disabled). Those 2 CPUs packages are hence placed in 2 different NUMA zones. By default, mlx5 driver creates one rx-queue per core, in this case it creates 40 rx-queues with 40 irqs (one for each rx-queue/core). This means it doesn't take into account the fact that the network card is plugged into a given PCIe bus belonging to one of the 2 available NUMA zones. As a result, when a packet is received and put into the rx-queue+irq belonging to the other NUMA zone, a penalty in memory-bandwitch is incurred, which ends up in a performance throughput penalty. With 100 flows at ~70Gbps being sent to the host, by default load spread among all 40 cores can reach processing at 55Gbps 4.62 MPPS. By decreasing the rx-queues to 20 and pinning the irqs to the cores in the same NUMA node, using only those 20 CPUs we get a small gain reaching ~62 Gbps 5.22 MPPS. Change-Id: I26fce50c04b043b61ba418d7090b2573e7807b08 --- M testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh 1 file changed, 30 insertions(+), 3 deletions(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved diff --git a/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh b/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh index 825e59f..947c465 100755 --- a/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh +++ b/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh @@ -1,8 +1,33 @@ #!/bin/bash -xe +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) + # Interface towards UPF: -IFACE_RAN="enp2s0f1np1" -IFACE_CN="enp2s0f0np0" +case $(hostname) in +gtplab1) + IFACE_RAN="enp2s0f1np1" + IFACE_CN="enp2s0f0np0" + MAC_CN_PEER="ec:0d:9a:8a:27:53" + ;; +c240-2) + IFACE_RAN="ens2f0np0" + IFACE_CN="ens2f1np1" + MAC_CN_PEER="88:e9:a4:3b:f0:49" + # Adapt Rx/Tx Queues and IRQs to Numa Node of device: + # cat /sys/class/net/$IFACE_RAN/device/numa_node + # 0 + # cat /sys/class/net/$IFACE_RAN/device/local_cpulist + # 0-19 + sudo ethtool -L $IFACE_RAN combined 20 + sudo ethtool -L $IFACE_CN combined 20 + sudo "$SCRIPT_DIR/../../../../scripts/set_irq_affinity_cpulist.sh" 0-19 $IFACE_RAN + sudo "$SCRIPT_DIR/../../../../scripts/set_irq_affinity_cpulist.sh" 0-19 $IFACE_CN + ;; +*) + echo "UNEXPECTED HOSTNAME: $(hostname)" + exit 1 + ;; +esac # Disable ethernet flow control: sudo ethtool -A $IFACE_RAN autoneg off rx off tx off @@ -26,7 +51,9 @@ # Set up UPF address: sudo ip addr add 172.16.32.1/24 dev $IFACE_RAN || true; +sudo ip link set up dev $IFACE_RAN || true; sudo ip addr add 172.16.31.2/24 dev $IFACE_CN || true; +sudo ip link set up dev $IFACE_CN || true; # TRex doesn't answer ARPs, so we need to set up the CN-side peer: -sudo ip neigh add 172.16.31.200 lladdr ec:0d:9a:8a:27:53 nud permanent dev $IFACE_CN || true; +sudo ip neigh add 172.16.31.200 lladdr "$MAC_CN_PEER" nud permanent dev $IFACE_CN || true; -- To view, visit https://gerrit.osmocom.org/c/upf-benchmark/+/39846?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: upf-benchmark Gerrit-Branch: master Gerrit-Change-Id: I26fce50c04b043b61ba418d7090b2573e7807b08 Gerrit-Change-Number: 39846 Gerrit-PatchSet: 1 Gerrit-Owner: pespin <pespin(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org>