April 2025 - gerrit-log - lists.osmocom.org

[S] Change in osmo-ci[master]: jobs: free up runners from 01:00 - 03:00

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/40141?usp=email ) Change subject: jobs: free up runners from 01:00 - 03:00 ...................................................................... jobs: free up runners from 01:00 - 03:00 When looking at grafana, the build hosts (build4 and build5) are not busy during 00:00 and 03:00. Binary packages seem to be finished at 00:00 already. Shorten the time for "Jobs that don't need binary packages" by two hours to make room for running ttcn3-bts-test with less load in a follow-up patch. Change-Id: Iebfa58aca28bc8e575983aae9f4d34396a487a7e --- M jobs/README.md M jobs/osmocom-build-tags-against-master.yml M jobs/osmocom-list-commits.yml M jobs/registry-rebuild-upload-fpga-build.yml M jobs/registry-triggers.yml M jobs/simtester-sanitize.yml 6 files changed, 11 insertions(+), 11 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/41/40141/1 diff --git a/jobs/README.md b/jobs/README.md index 653978a..bc3b1c6 100644 --- a/jobs/README.md +++ b/jobs/README.md @@ -158,19 +158,19 @@ 19:XX osmocom-obs-wireshark 20:XX osmocom-obs (new binary packages start building on OBS!) -22:00 - 03:00 Jobs that don't need binary packages +22:00 - 01:00 Jobs that don't need binary packages 22:XX coverity 22:XX octsim_osmo-ccid-firmware 22:XX osmo-gsm-tester-runner (virtual) 23:XX build-kernels-testenv 23:XX master-builds-dahdi 00:XX osmocom-api + 00:XX osmocom-build-tags-against-master + 00:XX osmocom-list-commits + 00:XX registry-rebuild-upload-fpga-build (weekly) + 00:XX registry-triggers 00:XX registry-update-base-images - 01:XX osmocom-build-tags-against-master - 01:XX osmocom-list-commits - 01:XX simtester-sanitize - 02:XX registry-triggers - 02:XX registry-rebuild-upload-fpga-build (weekly) + 00:XX simtester-sanitize 03:00 - 18:00 Jobs that need binary packages 03:00 - 08:00 ttcn3-testsuites diff --git a/jobs/osmocom-build-tags-against-master.yml b/jobs/osmocom-build-tags-against-master.yml index 6953dfd..39d4b2b 100644 --- a/jobs/osmocom-build-tags-against-master.yml +++ b/jobs/osmocom-build-tags-against-master.yml @@ -46,7 +46,7 @@ git-config-name: 'Jenkins Builder' git-config-email: 'jenkins(a)osmocom.org' triggers: - - timed: "H 01 * * *" + - timed: "H 00 * * *" publishers: - archive: allow-empty: false diff --git a/jobs/osmocom-list-commits.yml b/jobs/osmocom-list-commits.yml index d367499..5084879 100644 --- a/jobs/osmocom-list-commits.yml +++ b/jobs/osmocom-list-commits.yml @@ -34,7 +34,7 @@ git-config-name: 'Jenkins Builder' git-config-email: 'jenkins(a)osmocom.org' triggers: - - timed: "H 01 * * *" + - timed: "H 00 * * *" publishers: - archive: allow-empty: false diff --git a/jobs/registry-rebuild-upload-fpga-build.yml b/jobs/registry-rebuild-upload-fpga-build.yml index 1846e49..9efa0ce 100644 --- a/jobs/registry-rebuild-upload-fpga-build.yml +++ b/jobs/registry-rebuild-upload-fpga-build.yml @@ -29,7 +29,7 @@ docker push $REGISTRY/$USER/fpga-build cd .. triggers: - - timed: "H 02 * * H" + - timed: "H 00 * * H" publishers: - email: diff --git a/jobs/registry-triggers.yml b/jobs/registry-triggers.yml index da7d402..8154ff6 100644 --- a/jobs/registry-triggers.yml +++ b/jobs/registry-triggers.yml @@ -27,7 +27,7 @@ - pollscm: cron: "H/5 * * * *" ignore-post-commit-hooks: false - - timed: "H 02 * * *" + - timed: "H 00 * * *" publishers: - trigger: project: '{obj:trigger}' diff --git a/jobs/simtester-sanitize.yml b/jobs/simtester-sanitize.yml index 7cb7f26..fce678a 100644 --- a/jobs/simtester-sanitize.yml +++ b/jobs/simtester-sanitize.yml @@ -33,7 +33,7 @@ branches: - '$BRANCH' triggers: - - timed: "H 01 * * *" + - timed: "H 00 * * *" publishers: - email: recipients: '$EMAIL_NOTIFICATIONS' -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40141?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Iebfa58aca28bc8e575983aae9f4d34396a487a7e Gerrit-Change-Number: 40141 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de>

3 months, 3 weeks

1
0
0 0

[M] Change in osmo-ci[master]: jobs: run ttcn3-bts-test-* between 01:00 - 06:00

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/40142?usp=email ) Change subject: jobs: run ttcn3-bts-test-* between 01:00 - 06:00 ...................................................................... jobs: run ttcn3-bts-test-* between 01:00 - 06:00 The automatic respawns of osmo-bts-trx have been intentionally removed in the related docker-playground patch, so we can better notice when osmo-bts-trx crashes instead of just restarting it and moving on with the testsuite. The result is that now it is even more important when ttcn3-bts-test jobs are running, because when the builders are under high load then faketrx cannot send the time quickly enough and it also causes osmo-bts-trx to shutdown. In a discussion with Vadim we've decided to schedule the ttcn3-bts-tests so that they don't only run at a time where no other jobs are running. But also that they don't run at the same time on the same builder anymore to give them the least unrelated load possible. Other solutions are also being investigated, but this should make the job in its current form more stable already. Move misc ttcn3 jobs to run from 06:00 to 08:00 and run ttcn3-bts-tests between 01:00 and 06:00. Pin them to specific runners to make sure we don't have both running on the same node. Related: docker-playground I770a0530f90c5e03d2073bfc1c4cfd0e840379c9 Change-Id: I0162f7299c8e37f893ffa10ddc4c8edece29ed7f --- M jobs/README.md M jobs/ttcn3-testsuites.yml 2 files changed, 33 insertions(+), 24 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/42/40142/1 diff --git a/jobs/README.md b/jobs/README.md index bc3b1c6..1997d33 100644 --- a/jobs/README.md +++ b/jobs/README.md @@ -172,8 +172,12 @@ 00:XX registry-update-base-images 00:XX simtester-sanitize -03:00 - 18:00 Jobs that need binary packages - 03:00 - 08:00 ttcn3-testsuites +01:00 - 06:00 ttcn3-bts-test (fail if the builders are under too much load!) + 01:30 ttcn3-bts-test, ttcn3-bts-test-io_uring + 03:30 ttcn3-bts-test-latest, ttcn3-bts-test-asan + +06:00 - 18:00 Other jobs that need binary packages + 06:00 - 08:00 ttcn3-testsuites (misc without testenv) 08:00 - 18:00 ttcn3-testsuites-testenv 04:XX osmocom-release-manuals 05:XX osmocom-release-tarballs diff --git a/jobs/ttcn3-testsuites.yml b/jobs/ttcn3-testsuites.yml index 0ac11b6..d842fff 100644 --- a/jobs/ttcn3-testsuites.yml +++ b/jobs/ttcn3-testsuites.yml @@ -21,44 +21,44 @@ job-name: # debian nightly - order by time - nplab-m3ua-test: # ~1 min - timer: H 03 * * * + timer: H 06 * * * - nplab-sua-test: # ~1 min - timer: H 03 * * * + timer: H 06 * * * - ttcn3-gbproxy-test: # ~15 min - timer: H 03 * * * + timer: H 06 * * * - ttcn3-gbproxy-test-fr: # ~26 min - timer: H 03 * * * + timer: H 06 * * * node: hdlc - ttcn3-ns-test-fr: # ~10 min - timer: H 04 * * * + timer: H 06 * * * node: hdlc - ttcn3-fr-test: # ~ 10 min - timer: H 04 * * * + timer: H 06 * * * node: hdlc - ttcn3-pgw-test: # ~7 min - timer: H 04 * * * + timer: H 06 * * * description: | Test open5gs-smfd + open5gs-upfd PGW main. email: jenkins-notifications(a)lists.osmocom.org acetcom(a)gmail.com - ttcn3-epdg-test: # ~4 min - timer: H 04 * * * + timer: H 06 * * * # osmo-epdg requires OTP and kernel >= debian12: node: build4-deb12build-ansible - ttcn3-asterisk-ims-ue-test: # ~19 min - timer: H 04 * * * + timer: H 06 * * * # debian latest stable - nplab-m3ua-test-latest: # ~1 min - timer: H 04 * * * + timer: H 07 * * * - nplab-sua-test-latest: # ~1 min - timer: H 04 * * * + timer: H 07 * * * - ttcn3-gbproxy-test-latest: # ~15 min - timer: H 05 * * * + timer: H 07 * * * - ttcn3-gbproxy-test-fr-latest: # ~23 min - timer: H 05 * * * + timer: H 07 * * * node: hdlc - ttcn3-epdg-test-latest: # ~5 min - timer: H 05 * * * + timer: H 07 * * * # osmo-epdg requires OTP and kernel >= debian12: node: build4-deb12build-ansible @@ -66,29 +66,34 @@ # centos7 nightly - TTCN3-centos7-pcap-client-test: # ~1 min - timer: H 05 * * * + timer: H 07 * * * # centos7 latest - TTCN3-centos7-pcap-client-test-latest: # ~1 min - timer: H 05 * * * + timer: H 07 * * * # With LIBOSMO_IO_BACKEND=IO_URING (OS#6357) - ttcn3-gbproxy-test-io_uring: # ~15 min - timer: H 05 * * * + timer: H 07 * * * # BTS testsuites tend to fail with timeouts on heavy loads. Group them - # here in a timeslot where jenkins nodes are otherwise not busy according + # here in timeslots where jenkins nodes are otherwise not busy according # to stats in grafana to hopefully make the test results more stable. + # See also jobs/README.md. # Once we can replace faketrx with a rewrite (OS#6672), this should not # be necessary anymore. - ttcn3-bts-test-io_uring: # ~135 min - timer: 0 06 * * * + timer: 30 01 * * * + node: build4-deb12build-ansible - ttcn3-bts-test: # ~135 min - timer: 0 06 * * * + timer: 30 01 * * * + node: build5-deb12build-ansible - ttcn3-bts-test-latest: # ~135 min - timer: 0 06 * * * + timer: 30 03 * * * + node: build4-deb12build-ansible - ttcn3-bts-test-asan: # ~135 min - timer: 0 06 * * * + timer: 30 03 * * * + node: build5-deb12build-ansible # With sanitizer (OS#5301) - ttcn3-gbproxy-test-asan: # ~15 min -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40142?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I0162f7299c8e37f893ffa10ddc4c8edece29ed7f Gerrit-Change-Number: 40142 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de>

3 months, 3 weeks

1
0
0 0

[M] Change in osmo-ttcn3-hacks[master]: testenv: support fetching coredumps in jenkins

by pespin

Attention is currently required from: fixeria, osmith. pespin has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email ) Change subject: testenv: support fetching coredumps in jenkins ...................................................................... Patch Set 2: (1 comment) File _testenv/testenv/coredump.py: https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140/comment/90def1a0_784f… : PS2, Line 35: def get_from_coredumpctl_lxc_host(): I'm missing in here some reference to the script providing this service. -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18 Gerrit-Change-Number: 40140 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Comment-Date: Thu, 24 Apr 2025 12:28:57 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No

3 months, 3 weeks

1
0
0 0

[XS] Change in osmo-ci[master]: jobs/ttcn3-testsuites-testenv: set core env var

by pespin

Attention is currently required from: osmith. pespin has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ci/+/40139?usp=email ) Change subject: jobs/ttcn3-testsuites-testenv: set core env var ...................................................................... Patch Set 1: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40139?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I8359b0faa1fed76b430749589916cd072a8a7753 Gerrit-Change-Number: 40139 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Comment-Date: Thu, 24 Apr 2025 12:26:02 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes

3 months, 3 weeks

1
0
0 0

[S] Change in osmo-ci[master]: ansible: add build-hosts (build4, build5)

by pespin

Attention is currently required from: fixeria, osmith. pespin has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ci/+/40137?usp=email ) Change subject: ansible: add build-hosts (build4, build5) ...................................................................... Patch Set 1: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40137?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I2aa4e774b288f4ac00fb38be40de32415c565733 Gerrit-Change-Number: 40137 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Comment-Date: Thu, 24 Apr 2025 12:25:43 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes

3 months, 3 weeks

1
0
0 0

[M] Change in osmo-ci[master]: ansible: build-hosts: add testenv-coredump-helper

by pespin

Attention is currently required from: osmith. pespin has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email ) Change subject: ansible: build-hosts: add testenv-coredump-helper ...................................................................... Patch Set 1: (2 comments) File ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.py: https://gerrit.osmocom.org/c/osmo-ci/+/40138/comment/32eaa762_a7281cee?usp=… : PS1, Line 4: # OS#6769: simple webserver to make Osmocom related coredumps available in LXCs I'm really lacking a big chunk of descriptions here regarding what this script does, where it is spected to run (host? lxc?) as most important, *how* it does it. I see some webs erver being involved for instance, but no mention anywhere on how that's supposed to be working or what are they exactly speaking. https://gerrit.osmocom.org/c/osmo-ci/+/40138/comment/74bc58ba_f78ef890?usp=… : PS1, Line 66: # Put coredump into a temporary file and return it this may be problematic if it ends up in tmpfs, a coredump can be veery large. -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I7e66c98106b7028a393e3b873e96ae2dcb412c48 Gerrit-Change-Number: 40138 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-CC: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Comment-Date: Thu, 24 Apr 2025 12:25:28 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No

3 months, 3 weeks

1
0
0 0

[XS] Change in libosmo-netif[master]: stream: Improve error logging sctp_recvmsg without sctp_sndrcvinfo

by pespin

pespin has submitted this change. ( https://gerrit.osmocom.org/c/libosmo-netif/+/40122?usp=email ) Change subject: stream: Improve error logging sctp_recvmsg without sctp_sndrcvinfo ...................................................................... stream: Improve error logging sctp_recvmsg without sctp_sndrcvinfo Related: OS#6775 Change-Id: I4ef03a2505ca7409d50fd34371ae4ec40bebb40f --- M src/stream.c 1 file changed, 5 insertions(+), 2 deletions(-) Approvals: osmith: Looks good to me, approved Jenkins Builder: Verified laforge: Looks good to me, but someone else must approve diff --git a/src/stream.c b/src/stream.c index fb15f93..f0902a4 100644 --- a/src/stream.c +++ b/src/stream.c @@ -266,8 +266,11 @@ return ret; } - if (OSMO_UNLIKELY(ret > 0 && !sinfo)) - LOGP(DLINP, LOGL_ERROR, "%s sctp_recvmsg without SNDRCV cmsg?!?\n", log_pfx); + if (OSMO_UNLIKELY(ret > 0 && !sinfo)) { + char buf[512]; + LOGP(DLINP, LOGL_ERROR, "%s sctp_recvmsg without SNDRCV cmsg?!? ret=%d, flags=0x%x, %s\n", + log_pfx, ret, flags, msgb_hexdump_buf(buf, sizeof(buf), msg)); + } return ret; } -- To view, visit https://gerrit.osmocom.org/c/libosmo-netif/+/40122?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: libosmo-netif Gerrit-Branch: master Gerrit-Change-Id: I4ef03a2505ca7409d50fd34371ae4ec40bebb40f Gerrit-Change-Number: 40122 Gerrit-PatchSet: 1 Gerrit-Owner: pespin <pespin(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>

3 months, 3 weeks

1
0
0 0

[M] Change in osmo-ttcn3-hacks[master]: testenv: support fetching coredumps in jenkins

by osmith

Attention is currently required from: fixeria, pespin. Hello Jenkins Builder, fixeria, pespin, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email to look at the new patch set (#2). The following approvals got outdated and were removed: Verified+1 by Jenkins Builder Change subject: testenv: support fetching coredumps in jenkins ...................................................................... testenv: support fetching coredumps in jenkins The Osmocom jenkins nodes run inside LXCs. When we get a coredump it appears on the host, fetch it from there via testenv-coredump-helper, which gets added to the hosts in the related patch. Related: osmo-ci I7e66c98106b7028a393e3b873e96ae2dcb412c48 Related: OS#6769 Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18 --- M _testenv/README.md M _testenv/testenv/coredump.py 2 files changed, 61 insertions(+), 1 deletion(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/40/40140/2 -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newpatchset Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18 Gerrit-Change-Number: 40140 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: pespin <pespin(a)sysmocom.de> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de>

3 months, 3 weeks

1
0
0 0

[M] Change in osmo-ttcn3-hacks[master]: testenv: support fetching coredumps in jenkins

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email ) Change subject: testenv: support fetching coredumps in jenkins ...................................................................... testenv: support fetching coredumps in jenkins The Osmocom jenkins nodes run inside LXCs. When we get a coredump it appears on the host, fetch it from there via testenv-coredump-helper, which gets added to the hosts in the related patch. Related: osmo-ci I7e66c98106b7028a393e3b873e96ae2dcb412c48 Related: OS#6769 Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18 --- M _testenv/README.md M _testenv/testenv/coredump.py 2 files changed, 60 insertions(+), 1 deletion(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/40/40140/1 diff --git a/_testenv/README.md b/_testenv/README.md index 11cd8bd..66158c4 100644 --- a/_testenv/README.md +++ b/_testenv/README.md @@ -215,6 +215,15 @@ available but doesn't work in podman. QEMU runs a bit slower when this is set. +* `TESTENV_COREDUMP_FROM_LXC_HOST`: + Instead of using coredumpctl to retrieve the coredump, assume testenv is + running inside an LXC and try to retrieve the coredump from the LXC host + (OS#6769). This is used in jenkins. + +* `TESTENV_COREDUMP_FROM_LXC_HOST_IP`: + Instead of attempting to automatically detect the LXC host IP, use this IP. + This can be set to 127.0.0.1 for testing. + ## Troubleshooting ### Timeout waiting for RESET-ACK after sending RESET diff --git a/_testenv/testenv/coredump.py b/_testenv/testenv/coredump.py index 0d56328..4c0d9d2 100644 --- a/_testenv/testenv/coredump.py +++ b/_testenv/testenv/coredump.py @@ -11,9 +11,52 @@ import testenv import testenv.daemons import testenv.testdir +import urllib +import urllib.request executable_path = None +lxc_netdev = "eth0" +lxc_ip_pattern = "10.0.*" +lxc_port = 8042 + + +def find_lxc_host_ip(): + cmd = ["ip", "-j", "-o", "-4", "addr", "show", "dev", lxc_netdev] + p = testenv.cmd.run(cmd, check=False, no_podman=True, capture_output=True, text=True) + ret = json.loads(p.stdout)[0]["addr_info"][0]["local"] + if fnmatch.fnmatch(ret, lxc_ip_pattern): + ret = ret.split(".") + ret = f"{ret[0]}.{ret[1]}.{ret[2]}.1" + return ret + return None + + +def get_from_coredumpctl_lxc_host(): + global executable_path + + logging.info("Looking for a coredump on lxc host") + + ip = os.environ.get("TESTENV_COREDUMP_FROM_LXC_HOST_IP") or find_lxc_host_ip() + if not ip: + logging.warning("Failed to get lxc host ip, can't look for coredump") + return + + try: + with urllib.request.urlopen(f"http://{ip}:{lxc_port}/core") as response: + executable_path = dict(response.getheaders())["X-Executable-Path"] + with open(f"{testenv.testdir.testdir}/core", "wb") as h: + shutil.copyfileobj(response, h) + logging.debug("Coredump found and copied to log dir") + except urllib.error.HTTPError as e: + executable_path = None + if e.code == 404: + logging.debug("No coredump found") + else: + logging.error(f"Unexpected error while attempting to fetch the coredump: {e}") + except urllib.error.URLError as e: + logging.error(f"Unexpected error while attempting to fetch the coredump: {e}") + def executable_is_relevant(exe): if testenv.args.binary_repo: @@ -32,7 +75,7 @@ return False -def get_from_coredumpctl(): +def get_from_coredumpctl_local(): global executable_path logging.info("Looking for a coredump") @@ -68,6 +111,13 @@ executable_path = coredump["exe"] +def get_from_coredumpctl(): + if os.environ.get("TESTENV_COREDUMP_FROM_LXC_HOST"): + get_from_coredumpctl_lxc_host() + else: + get_from_coredumpctl_local() + + def get_backtrace(): global executable_path -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18 Gerrit-Change-Number: 40140 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de>

3 months, 3 weeks

1
0
0 0

[M] Change in osmo-ci[master]: ansible: build-hosts: add testenv-coredump-helper

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email ) Change subject: ansible: build-hosts: add testenv-coredump-helper ...................................................................... ansible: build-hosts: add testenv-coredump-helper The Osmocom jenkins nodes run inside LXCs. When we get a coredump it appears on the host. Add a helper script to the hosts so the jenkins jobs can fetch the coredumps in case an Osmocom program crashes while running a ttcn3 testsuite. The helper script has the following safety features to ensure jenkins can't just fetch any coredump: * Only fetch coredumps within the last 3 seconds and only if the executable matches osmo-* or open5gs-* * Only listen on the lxc IP Related: OS#6769 Change-Id: I7e66c98106b7028a393e3b873e96ae2dcb412c48 --- A ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.py A ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.service A ansible/roles/testenv-coredump-helper/handlers/main.yml A ansible/roles/testenv-coredump-helper/tasks/main.yml M ansible/setup-build-host.yml 5 files changed, 160 insertions(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/38/40138/1 diff --git a/ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.py b/ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.py new file mode 100644 index 0000000..acbed17 --- /dev/null +++ b/ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +# Copyright 2025 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +# OS#6769: simple webserver to make Osmocom related coredumps available in LXCs +import datetime +import fnmatch +import http.server +import json +import os +import shutil +import signal +import socket +import socketserver +import subprocess +import sys +import tempfile + + +NETDEV = "lxcbr0" +IP_PATTERN = "10.0.*" +PORT = 8042 + + +def find_lxc_ip(): + cmd = ["ip", "-j", "-o", "-4", "addr", "show", "dev", NETDEV] + p = subprocess.run(cmd, capture_output=True, text=True, check=True) + ret = json.loads(p.stdout)[0]["addr_info"][0]["local"] + if not fnmatch.fnmatch(ret, IP_PATTERN): + print(f"ERROR: IP doesn't match pattern {IP_PATTERN}: {ret}") + sys.exit(1) + return ret + + +def executable_is_relevant(exe): + basename = os.path.basename(exe) + patterns = [ + "open5gs-*", + "osmo-*", + ] + + for pattern in patterns: + if fnmatch.fnmatch(basename, pattern): + return True + + return False + + +class CustomRequestHandler(http.server.SimpleHTTPRequestHandler): + def do_GET(self): + if self.path == "/core": + # Check for any coredump within last 3 seconds + since = (datetime.datetime.now() - datetime.timedelta(seconds=3)).strftime("%Y-%m-%d %H:%M:%S") + cmd = ["coredumpctl", "-q", "-S", since, "--json=short", "-n1"] + + p = subprocess.run(cmd, capture_output=True, text=True) + if p.returncode != 0: + self.send_error(404, "No coredump found") + return None + + # Check if the coredump executable is from osmo-*, open5gs-*, etc. + coredump = json.loads(p.stdout)[0] + if not executable_is_relevant(coredump["exe"]): + self.send_error(404, "No coredump found") + return None + + # Put coredump into a temporary file and return it + with tempfile.TemporaryDirectory() as tmpdirname: + core_path = os.path.join(tmpdirname, "core") + cmd = [ + "coredumpctl", + "dump", + "-q", + "-S", + since, + "-o", + core_path, + str(coredump["pid"]), + coredump["exe"], + ] + subprocess.run(cmd, stdout=subprocess.DEVNULL, check=True) + + with open(core_path, "rb") as f: + self.send_response(200) + self.send_header("X-Executable-Path", coredump["exe"]) + self.end_headers() + self.wfile.write(f.read()) + else: + self.send_error(404, "File Not Found") + + +def signal_handler(sig, frame): + sys.exit(0) + + +def main(): + if not shutil.which("coredumpctl"): + print("ERROR: coredumpctl not found!") + sys.exit(1) + + ip = os.environ.get("LXC_HOST_IP") or find_lxc_ip() + print(f"Listening on {ip}:{PORT}") + signal.signal(signal.SIGINT, signal_handler) + with socketserver.TCPServer((ip, PORT), CustomRequestHandler, False) as httpd: + httpd.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + httpd.server_bind() + httpd.server_activate() + httpd.serve_forever() + + +if __name__ == "__main__": + main() diff --git a/ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.service b/ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.service new file mode 100644 index 0000000..ef5a851 --- /dev/null +++ b/ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.service @@ -0,0 +1,12 @@ +[Unit] +Description=testenv coredump helper +After=lxc.service + +[Service] +Environment="PYTHONUNBUFFERED=1" +Type=simple +Restart=always +ExecStart=/opt/testenv-coredump-helper/testenv-coredump-helper + +[Install] +WantedBy=multi-user.target diff --git a/ansible/roles/testenv-coredump-helper/handlers/main.yml b/ansible/roles/testenv-coredump-helper/handlers/main.yml new file mode 100644 index 0000000..0f7ef57 --- /dev/null +++ b/ansible/roles/testenv-coredump-helper/handlers/main.yml @@ -0,0 +1,5 @@ +--- +- name: restart testenv-coredump-helper + service: + name: testenv-coredump-helper + state: restarted diff --git a/ansible/roles/testenv-coredump-helper/tasks/main.yml b/ansible/roles/testenv-coredump-helper/tasks/main.yml new file mode 100644 index 0000000..9ff2769 --- /dev/null +++ b/ansible/roles/testenv-coredump-helper/tasks/main.yml @@ -0,0 +1,31 @@ +--- +- name: install coredumpctl + apt: + name: + - systemd-coredump + cache_valid_time: 3600 + update_cache: yes + +- name: mkdir /opt/testenv-coredump-helper + ansible.builtin.file: + path: /opt/testenv-coredump-helper + state: directory + +- name: install testenv-coredump-helper + ansible.builtin.copy: + src: testenv-coredump-helper.py + dest: /opt/testenv-coredump-helper/testenv-coredump-helper + mode: '0755' + notify: restart testenv-coredump-helper + +- name: install testenv-coredump-helper service + ansible.builtin.copy: + src: testenv-coredump-helper.service + dest: /etc/systemd/system/testenv-coredump-helper.service + mode: '0644' + notify: restart testenv-coredump-helper + +- name: enable testenv-coredump-helper service + ansible.builtin.systemd_service: + name: testenv-coredump-helper + enabled: true diff --git a/ansible/setup-build-host.yml b/ansible/setup-build-host.yml index ed8def5..d1d9874 100644 --- a/ansible/setup-build-host.yml +++ b/ansible/setup-build-host.yml @@ -18,3 +18,4 @@ update_cache: yes roles: - name: apt-allow-relinfo-change + - name: testenv-coredump-helper -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I7e66c98106b7028a393e3b873e96ae2dcb412c48 Gerrit-Change-Number: 40138 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de>

3 months, 3 weeks

1
0
0 0

2025

2024

2023

2022

gerrit-log April 2025