laforge has submitted this change. ( https://gerrit.osmocom.org/c/upf-benchmark/+/39846?usp=email )
Change subject: testsuites: upf: Support setting net iface irqs in c240 host
......................................................................
testsuites: upf: Support setting net iface irqs in c240 host
The Cisco C240 machines consist of 2 CPU packages, each with 20 cores
(hyperthreading disabled). Those 2 CPUs packages are hence placed in 2
different NUMA zones.
By default, mlx5 driver creates one rx-queue per core, in this case it
creates 40 rx-queues with 40 irqs (one for each rx-queue/core).
This means it doesn't take into account the fact that the network card
is plugged into a given PCIe bus belonging to one of the 2 available
NUMA zones.
As a result, when a packet is received and put into the rx-queue+irq
belonging to the other NUMA zone, a penalty in memory-bandwitch is
incurred, which ends up in a performance throughput penalty.
With 100 flows at ~70Gbps being sent to the host, by default load spread
among all 40 cores can reach processing at 55Gbps 4.62 MPPS.
By decreasing the rx-queues to 20 and pinning the irqs to the cores in
the same NUMA node, using only those 20 CPUs we get a small gain
reaching ~62 Gbps 5.22 MPPS.
Change-Id: I26fce50c04b043b61ba418d7090b2573e7807b08
---
M testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh
1 file changed, 30 insertions(+), 3 deletions(-)
Approvals:
Jenkins Builder: Verified
laforge: Looks good to me, approved
diff --git a/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh b/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh
index 825e59f..947c465 100755
--- a/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh
+++ b/testsuites/gtplab-sysmo2025/tunmap/upf/configure.sh
@@ -1,8 +1,33 @@
#!/bin/bash -xe
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
# Interface towards UPF:
-IFACE_RAN="enp2s0f1np1"
-IFACE_CN="enp2s0f0np0"
+case $(hostname) in
+gtplab1)
+ IFACE_RAN="enp2s0f1np1"
+ IFACE_CN="enp2s0f0np0"
+ MAC_CN_PEER="ec:0d:9a:8a:27:53"
+ ;;
+c240-2)
+ IFACE_RAN="ens2f0np0"
+ IFACE_CN="ens2f1np1"
+ MAC_CN_PEER="88:e9:a4:3b:f0:49"
+ # Adapt Rx/Tx Queues and IRQs to Numa Node of device:
+ # cat /sys/class/net/$IFACE_RAN/device/numa_node
+ # 0
+ # cat /sys/class/net/$IFACE_RAN/device/local_cpulist
+ # 0-19
+ sudo ethtool -L $IFACE_RAN combined 20
+ sudo ethtool -L $IFACE_CN combined 20
+ sudo "$SCRIPT_DIR/../../../../scripts/set_irq_affinity_cpulist.sh" 0-19 $IFACE_RAN
+ sudo "$SCRIPT_DIR/../../../../scripts/set_irq_affinity_cpulist.sh" 0-19 $IFACE_CN
+ ;;
+*)
+ echo "UNEXPECTED HOSTNAME: $(hostname)"
+ exit 1
+ ;;
+esac
# Disable ethernet flow control:
sudo ethtool -A $IFACE_RAN autoneg off rx off tx off
@@ -26,7 +51,9 @@
# Set up UPF address:
sudo ip addr add 172.16.32.1/24 dev $IFACE_RAN || true;
+sudo ip link set up dev $IFACE_RAN || true;
sudo ip addr add 172.16.31.2/24 dev $IFACE_CN || true;
+sudo ip link set up dev $IFACE_CN || true;
# TRex doesn't answer ARPs, so we need to set up the CN-side peer:
-sudo ip neigh add 172.16.31.200 lladdr ec:0d:9a:8a:27:53 nud permanent dev $IFACE_CN || true;
+sudo ip neigh add 172.16.31.200 lladdr "$MAC_CN_PEER" nud permanent dev $IFACE_CN || true;
--
To view, visit https://gerrit.osmocom.org/c/upf-benchmark/+/39846?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: merged
Gerrit-Project: upf-benchmark
Gerrit-Branch: master
Gerrit-Change-Id: I26fce50c04b043b61ba418d7090b2573e7807b08
Gerrit-Change-Number: 39846
Gerrit-PatchSet: 1
Gerrit-Owner: pespin <pespin(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Attention is currently required from: pespin.
laforge has posted comments on this change by pespin. ( https://gerrit.osmocom.org/c/upf-benchmark/+/39846?usp=email )
Change subject: testsuites: upf: Support setting net iface irqs in c240 host
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit https://gerrit.osmocom.org/c/upf-benchmark/+/39846?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: upf-benchmark
Gerrit-Branch: master
Gerrit-Change-Id: I26fce50c04b043b61ba418d7090b2573e7807b08
Gerrit-Change-Number: 39846
Gerrit-PatchSet: 1
Gerrit-Owner: pespin <pespin(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Gerrit-Attention: pespin <pespin(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 27 Mar 2025 19:35:01 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Attention is currently required from: pespin.
laforge has posted comments on this change by pespin. ( https://gerrit.osmocom.org/c/upf-benchmark/+/39845?usp=email )
Change subject: Add scripts/ dir with helpers to operate on net iface irqs
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit https://gerrit.osmocom.org/c/upf-benchmark/+/39845?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: upf-benchmark
Gerrit-Branch: master
Gerrit-Change-Id: I646f57cb3940dbbd4e6606e013651116c38c9534
Gerrit-Change-Number: 39845
Gerrit-PatchSet: 1
Gerrit-Owner: pespin <pespin(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Gerrit-Attention: pespin <pespin(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 27 Mar 2025 19:34:39 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Attention is currently required from: fixeria, laforge.
pespin has posted comments on this change by fixeria. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email )
Change subject: s1gw_metrics: add histograms for gen_sctp:send/3 delay
......................................................................
Patch Set 4:
(1 comment)
Patchset:
PS3:
> > Can't gen_sctp:send be used in some sort of non-blocking way? […]
I'd go for something like:
* Use non_block_send, if send() fails, append the msg to a queue, and if queue was empty put some sort of timer that triggers each 100ms.
* When timer triggers, try re-sending all of the msgs in queue until it fails, then prepend the msg to the queue and rearm timer
* If path to call send() is called from some user and we are in "waiting-to-be-unblocked" mode (queue is not empty) try to first submit all msgs into the queue.
--
To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: erlang/osmo-s1gw
Gerrit-Branch: master
Gerrit-Change-Id: I7633a6e6182bda3fca4e7562e2b780b2b9f9708f
Gerrit-Change-Number: 39666
Gerrit-PatchSet: 4
Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Gerrit-CC: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: laforge <laforge(a)osmocom.org>
Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 27 Mar 2025 19:33:15 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: fixeria <vyanitskiy(a)sysmocom.de>
Comment-In-Reply-To: pespin <pespin(a)sysmocom.de>
Attention is currently required from: laforge, pespin.
fixeria has posted comments on this change by fixeria. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email )
Change subject: s1gw_metrics: add histograms for gen_sctp:send/3 delay
......................................................................
Patch Set 4:
(1 comment)
Patchset:
PS3:
> Can't gen_sctp:send be used in some sort of non-blocking way?
There is an option to make send non-blocking by sending `non_block_send`, but then we still need a logic that would (periodically?) try to send again. With all these layers of abstraction (gen_sctp), we cannot know when the socket becomes ready for sending :/
> Having N processes sleeping waiting to write to a socket sounds like a bit of a nightmare to me tbh, with eg. messages inside the proxied SCTP conn being reordered, etc.
This approach definitely needs careful thinking and planning. I am also a bit afraid of the complexity this would bring, especially given that the S1AP spec. explicitly requires the message ordering to be preserved.
--
To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: erlang/osmo-s1gw
Gerrit-Branch: master
Gerrit-Change-Id: I7633a6e6182bda3fca4e7562e2b780b2b9f9708f
Gerrit-Change-Number: 39666
Gerrit-PatchSet: 4
Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Gerrit-CC: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: laforge <laforge(a)osmocom.org>
Gerrit-Attention: pespin <pespin(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 27 Mar 2025 19:23:25 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: pespin <pespin(a)sysmocom.de>
Comment-In-Reply-To: fixeria <vyanitskiy(a)sysmocom.de>