March 2025 - gerrit-log - lists.osmocom.org

[M] Change in ...osmo-s1gw[master]: s1gw_metrics: add histograms for gen_sctp:send/3 delay

by fixeria

Attention is currently required from: fixeria, laforge. Hello Jenkins Builder, laforge, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email to look at the new patch set (#4). Change subject: s1gw_metrics: add histograms for gen_sctp:send/3 delay ...................................................................... s1gw_metrics: add histograms for gen_sctp:send/3 delay https://www.erlang.org/doc/apps/kernel/gen_sctp.html#send/3 clearly states that gen_sctp:send/3 is blocking, unless non_block_send is set. With the current architecture of the S1GW, gen_sctp:send/3 may block the S1GW-MME link (sctp_client) or the eNB-S1GW link (sctp_server) completely, preventing sending and receiving of other S1AP PDUs. Ideally, we should rework the S1GW to spawn a separate process for each received S1AP PDU, so that blocking gen_sctp:send/3 would at least not block receiving and processing of incoming PDUs. But this would require quite some effort. For now, let's add a histogram reflecting min/max/avg/mean/median delay caused by gen_sctp:send/3, so that we can detect this when the problem starts biting us. Change-Id: I7633a6e6182bda3fca4e7562e2b780b2b9f9708f Related: SYS#7288 --- M config/sys.config M include/s1gw_metrics.hrl M src/s1gw_metrics.erl M src/sctp_common.erl M src/sctp_proxy.erl M src/sctp_server.erl 6 files changed, 54 insertions(+), 7 deletions(-) git pull ssh://gerrit.osmocom.org:29418/erlang/osmo-s1gw refs/changes/66/39666/4 -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newpatchset Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I7633a6e6182bda3fca4e7562e2b780b2b9f9708f Gerrit-Change-Number: 39666 Gerrit-PatchSet: 4 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-CC: pespin <pespin(a)sysmocom.de> Gerrit-Attention: laforge <laforge(a)osmocom.org> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de>

3 months, 1 week

1
0
0 0

[M] Change in ...osmo-s1gw[master]: s1gw_metrics: add histograms for gen_sctp:send delay

by pespin

Attention is currently required from: fixeria, laforge. pespin has posted comments on this change by fixeria. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email ) Change subject: s1gw_metrics: add histograms for gen_sctp:send delay ...................................................................... Patch Set 3: (1 comment) Patchset: PS3: > > I'm not sure it makes sense to have this kind of metric in regular use of the software […] Can't gen_sctp:send be used in some sort of non-blocking way? Having N processes sleeping waiting to write to a socket sounds like a bit of a nightmare to me tbh, with eg. messages inside the proxied SCTP conn being reordered, etc. -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I7633a6e6182bda3fca4e7562e2b780b2b9f9708f Gerrit-Change-Number: 39666 Gerrit-PatchSet: 3 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-CC: pespin <pespin(a)sysmocom.de> Gerrit-Attention: laforge <laforge(a)osmocom.org> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Comment-Date: Thu, 27 Mar 2025 19:17:06 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: fixeria <vyanitskiy(a)sysmocom.de> Comment-In-Reply-To: pespin <pespin(a)sysmocom.de>

3 months, 1 week

1
0
0 0

[M] Change in ...osmo-s1gw[master]: s1gw_metrics: add histograms for gen_sctp:send delay

by fixeria

Attention is currently required from: laforge, pespin. fixeria has posted comments on this change by fixeria. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email ) Change subject: s1gw_metrics: add histograms for gen_sctp:send delay ...................................................................... Patch Set 3: (1 comment) Patchset: PS3: > I'm not sure it makes sense to have this kind of metric in regular use of the software I am not sure either, but I don't think it would hurt a lot. > only for really specific adhoc debugging. Well, we know that `gen_sctp:send` is blocking. And with the current architecture, it's possible that it blocks the whole process, be it `sctp_server` or `sctp_client` module. If this happens, we would at least **see** that it's because of `gen_sctp:send` taking too long. Harald proposed a different architecture, in which a separate process is spawned for each S1AP PDU. This approach is more robust, because `gen_sctp:send` would only be blocking sending of a single PDU, letting the S1GW to process other PDUs in the meantime. Once this is implemented, we can even get rid of this histogram. > In any case, this commit is missing rationale ... I will upload an updated version soon. -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I7633a6e6182bda3fca4e7562e2b780b2b9f9708f Gerrit-Change-Number: 39666 Gerrit-PatchSet: 3 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-CC: pespin <pespin(a)sysmocom.de> Gerrit-Attention: laforge <laforge(a)osmocom.org> Gerrit-Attention: pespin <pespin(a)sysmocom.de> Gerrit-Comment-Date: Thu, 27 Mar 2025 18:52:45 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: pespin <pespin(a)sysmocom.de>

3 months, 1 week

1
0
0 0

[XS] Change in ...osmo-s1gw[master]: s1gw_metrics: make exometer_report_statsd less verbose

by fixeria

fixeria has submitted this change. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39865?usp=email ) Change subject: s1gw_metrics: make exometer_report_statsd less verbose ...................................................................... s1gw_metrics: make exometer_report_statsd less verbose When running osmo-s1gw with {logger_level, debug}, this module clogs the logging output with dozens of "Report metric ..." lines. This makes debugging osmo-s1gw harder, so let's force it to use info. Change-Id: I317680252d052f6435d870d27ca51514ca5f0c09 --- M src/s1gw_metrics.erl 1 file changed, 1 insertion(+), 0 deletions(-) Approvals: Jenkins Builder: Verified pespin: Looks good to me, approved diff --git a/src/s1gw_metrics.erl b/src/s1gw_metrics.erl index dceb477..6eb917f 100644 --- a/src/s1gw_metrics.erl +++ b/src/s1gw_metrics.erl @@ -120,6 +120,7 @@ init() -> ?LOG_INFO("Initiating metrics"), + logger:set_module_level(exometer_report_statsd, info), [] = register_all(counter, ?S1GW_COUNTERS), [] = register_all(gauge, ?S1GW_GAUGES). -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39865?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I317680252d052f6435d870d27ca51514ca5f0c09 Gerrit-Change-Number: 39865 Gerrit-PatchSet: 1 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>

3 months, 1 week

1
0
0 0

[XS] Change in ...osmo-s1gw[master]: sctp_{client,server}: disable Nagle by setting sctp_nodelay

by fixeria

fixeria has submitted this change. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39864?usp=email ) Change subject: sctp_{client,server}: disable Nagle by setting sctp_nodelay ...................................................................... sctp_{client,server}: disable Nagle by setting sctp_nodelay The Nagle's algorithm [1] works by combining a number of small outgoing messages and sending them all at once. It's enabled by default when opening a socket. While it helps to reduce the network congestion by reducing the number of outgoing packets, it comes at the cost of increased delay. Disable it by setting sctp_nodelay to true. [1] https://en.wikipedia.org/wiki/Nagle%27s_algorithm Change-Id: I6058a593a617d67d479eea0673d899a5da2d49bf Related: SYS#7288 --- M src/sctp_client.erl M src/sctp_server.erl 2 files changed, 2 insertions(+), 0 deletions(-) Approvals: Jenkins Builder: Verified pespin: Looks good to me, approved diff --git a/src/sctp_client.erl b/src/sctp_client.erl index 5875f61..de3fefb 100644 --- a/src/sctp_client.erl +++ b/src/sctp_client.erl @@ -101,6 +101,7 @@ connect({LocAddr, RemAddr}, Port) -> {ok, Sock} = gen_sctp:open([{ip, LocAddr}, {type, seqpacket}, + {sctp_nodelay, true}, {active, true}]), gen_sctp:connect_init(Sock, RemAddr, Port, []), {ok, Sock}. diff --git a/src/sctp_server.erl b/src/sctp_server.erl index 27752d8..e9066ac 100644 --- a/src/sctp_server.erl +++ b/src/sctp_server.erl @@ -111,6 +111,7 @@ {port, BindPort}, {type, seqpacket}, {reuseaddr, true}, + {sctp_nodelay, true}, {active, true}]), ?LOG_INFO("SCTP server listening on ~w:~w", [BindAddr, BindPort]), ok = gen_sctp:listen(Sock, true), -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39864?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I6058a593a617d67d479eea0673d899a5da2d49bf Gerrit-Change-Number: 39864 Gerrit-PatchSet: 1 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>

3 months, 1 week

1
0
0 0

[M] Change in ...osmo-s1gw[master]: s1gw_metrics: add histograms for gen_sctp:send delay

by pespin

Attention is currently required from: fixeria, laforge. pespin has posted comments on this change by fixeria. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email ) Change subject: s1gw_metrics: add histograms for gen_sctp:send delay ...................................................................... Patch Set 3: (1 comment) Patchset: PS3: I'm not sure it makes sense to have this kind of metric in regular use of the software, only for really specific adhoc debugging. In any case, this commit is missing rationale on why is it interesting to have this kind of metric and what it exactly tries to measure. -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I7633a6e6182bda3fca4e7562e2b780b2b9f9708f Gerrit-Change-Number: 39666 Gerrit-PatchSet: 3 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-CC: pespin <pespin(a)sysmocom.de> Gerrit-Attention: laforge <laforge(a)osmocom.org> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Comment-Date: Thu, 27 Mar 2025 18:24:24 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No

3 months, 1 week

1
0
0 0

[XS] Change in ...osmo-s1gw[master]: s1gw_metrics: make exometer_report_statsd less verbose

by pespin

Attention is currently required from: fixeria. pespin has posted comments on this change by fixeria. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39865?usp=email ) Change subject: s1gw_metrics: make exometer_report_statsd less verbose ...................................................................... Patch Set 1: Code-Review+2 -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39865?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I317680252d052f6435d870d27ca51514ca5f0c09 Gerrit-Change-Number: 39865 Gerrit-PatchSet: 1 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Comment-Date: Thu, 27 Mar 2025 18:21:30 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes

3 months, 1 week

1
0
0 0

[XS] Change in ...osmo-s1gw[master]: sctp_{client,server}: disable Nagle by setting sctp_nodelay

by pespin

Attention is currently required from: fixeria. pespin has posted comments on this change by fixeria. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39864?usp=email ) Change subject: sctp_{client,server}: disable Nagle by setting sctp_nodelay ...................................................................... Patch Set 1: Code-Review+2 -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39864?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I6058a593a617d67d479eea0673d899a5da2d49bf Gerrit-Change-Number: 39864 Gerrit-PatchSet: 1 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Comment-Date: Thu, 27 Mar 2025 18:20:45 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes

3 months, 1 week

1
0
0 0

[M] Change in ...osmo-s1gw[master]: s1gw_metrics: add histograms for gen_sctp:send delay

by fixeria

fixeria has posted comments on this change by fixeria. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email ) Change subject: s1gw_metrics: add histograms for gen_sctp:send delay ...................................................................... Set Ready For Review -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39666?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I7633a6e6182bda3fca4e7562e2b780b2b9f9708f Gerrit-Change-Number: 39666 Gerrit-PatchSet: 3 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Comment-Date: Thu, 27 Mar 2025 17:32:32 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: No

3 months, 1 week

1
0
0 0

[XS] Change in ...osmo-s1gw[master]: s1gw_metrics: make exometer_report_statsd less verbose

by fixeria

fixeria has uploaded this change for review. ( https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39865?usp=email ) Change subject: s1gw_metrics: make exometer_report_statsd less verbose ...................................................................... s1gw_metrics: make exometer_report_statsd less verbose When running osmo-s1gw with {logger_level, debug}, this module clogs the logging output with dozens of "Report metric ..." lines. This makes debugging osmo-s1gw harder, so let's force it to use info. Change-Id: I317680252d052f6435d870d27ca51514ca5f0c09 --- M src/s1gw_metrics.erl 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/erlang/osmo-s1gw refs/changes/65/39865/1 diff --git a/src/s1gw_metrics.erl b/src/s1gw_metrics.erl index dceb477..6eb917f 100644 --- a/src/s1gw_metrics.erl +++ b/src/s1gw_metrics.erl @@ -120,6 +120,7 @@ init() -> ?LOG_INFO("Initiating metrics"), + logger:set_module_level(exometer_report_statsd, info), [] = register_all(counter, ?S1GW_COUNTERS), [] = register_all(gauge, ?S1GW_GAUGES). -- To view, visit https://gerrit.osmocom.org/c/erlang/osmo-s1gw/+/39865?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: erlang/osmo-s1gw Gerrit-Branch: master Gerrit-Change-Id: I317680252d052f6435d870d27ca51514ca5f0c09 Gerrit-Change-Number: 39865 Gerrit-PatchSet: 1 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de>

3 months, 1 week

1
0
0 0

2025

2024

2023

2022

gerrit-log March 2025