Attention is currently required from: fixeria, pespin.
osmith has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email )
Change subject: testenv: support fetching coredumps in jenkins
......................................................................
Patch Set 3:
(1 comment)
File _testenv/testenv/coredump.py:
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140/comment/f7a4a14d_5f0e… :
PS2, Line 35: def get_from_coredumpctl_lxc_host():
> I'm missing in here some reference to the script providing this service.
Done
--
To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18
Gerrit-Change-Number: 40140
Gerrit-PatchSet: 3
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 24 Apr 2025 14:38:09 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: pespin <pespin(a)sysmocom.de>
Attention is currently required from: fixeria, osmith.
Hello Jenkins Builder, fixeria, pespin,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email
to look at the new patch set (#3).
The following approvals got outdated and were removed:
Verified+1 by Jenkins Builder
Change subject: testenv: support fetching coredumps in jenkins
......................................................................
testenv: support fetching coredumps in jenkins
The Osmocom jenkins nodes run inside LXCs. When we get a coredump it
appears on the host, fetch it from there via testenv-coredump-helper,
which gets added to the hosts in the related patch.
Related: osmo-ci I7e66c98106b7028a393e3b873e96ae2dcb412c48
Related: OS#6769
Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18
---
M _testenv/README.md
M _testenv/testenv/coredump.py
2 files changed, 62 insertions(+), 1 deletion(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/40/40140/3
--
To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18
Gerrit-Change-Number: 40140
Gerrit-PatchSet: 3
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: osmith <osmith(a)sysmocom.de>
Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de>
Attention is currently required from: pespin.
osmith has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email )
Change subject: ansible: build-hosts: add testenv-coredump-helper
......................................................................
Patch Set 2:
(2 comments)
File ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.py:
https://gerrit.osmocom.org/c/osmo-ci/+/40138/comment/cba2f35a_c80e1535?usp=… :
PS1, Line 4: # OS#6769: simple webserver to make Osmocom related coredumps available in LXCs
> I'm really lacking a big chunk of descriptions here regarding what this script does, where it is spe […]
added a README.md answering those questions.
https://gerrit.osmocom.org/c/osmo-ci/+/40138/comment/33245c0f_236bf052?usp=… :
PS1, Line 66: # Put coredump into a temporary file and return it
> this may be problematic if it ends up in tmpfs, a coredump can be veery large.
The temporary directory gets created in `/tmp`, e.g. `/tmp/tmptlrx_0jr`. The `/tmp` dir is not mounted as tmpfs on the build hosts. I was also curious about the max coredump size, it is set via `/etc/systemd/coredump.conf` and the default is 2GiB which should be reasonable. We can tweak it later if needed IMHO.
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I7e66c98106b7028a393e3b873e96ae2dcb412c48
Gerrit-Change-Number: 40138
Gerrit-PatchSet: 2
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-CC: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: pespin <pespin(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 24 Apr 2025 14:35:29 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: pespin <pespin(a)sysmocom.de>
Attention is currently required from: osmith.
Hello Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email
to look at the new patch set (#2).
The following approvals got outdated and were removed:
Verified+1 by Jenkins Builder
Change subject: ansible: build-hosts: add testenv-coredump-helper
......................................................................
ansible: build-hosts: add testenv-coredump-helper
The Osmocom jenkins nodes run inside LXCs. When we get a coredump it
appears on the host. Add a helper script to the hosts so the jenkins
jobs can fetch the coredumps in case an Osmocom program crashes while
running a ttcn3 testsuite.
The helper script has the following safety features to ensure jenkins
can't just fetch any coredump:
* Only fetch coredumps within the last 3 seconds and only if the
executable matches osmo-* or open5gs-*
* Only listen on the lxc IP
Related: OS#6769
Change-Id: I7e66c98106b7028a393e3b873e96ae2dcb412c48
---
A ansible/roles/testenv-coredump-helper/README.md
A ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.py
A ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.service
A ansible/roles/testenv-coredump-helper/handlers/main.yml
A ansible/roles/testenv-coredump-helper/tasks/main.yml
M ansible/setup-build-host.yml
6 files changed, 210 insertions(+), 0 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/38/40138/2
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I7e66c98106b7028a393e3b873e96ae2dcb412c48
Gerrit-Change-Number: 40138
Gerrit-PatchSet: 2
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-CC: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: osmith <osmith(a)sysmocom.de>
osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/40142?usp=email )
Change subject: jobs: run ttcn3-bts-test-* between 01:00 - 06:00
......................................................................
jobs: run ttcn3-bts-test-* between 01:00 - 06:00
The automatic respawns of osmo-bts-trx have been intentionally removed
in the related docker-playground patch, so we can better notice when
osmo-bts-trx crashes instead of just restarting it and moving on with
the testsuite.
The result is that now it is even more important when ttcn3-bts-test
jobs are running, because when the builders are under high load then
faketrx cannot send the time quickly enough and it also causes
osmo-bts-trx to shutdown.
In a discussion with Vadim we've decided to schedule the ttcn3-bts-tests
so that they don't only run at a time where no other jobs are running.
But also that they don't run at the same time on the same builder
anymore to give them the least unrelated load possible. Other solutions
are also being investigated, but this should make the job in its current
form more stable already.
Move misc ttcn3 jobs to run from 06:00 to 08:00 and run ttcn3-bts-tests
between 01:00 and 06:00. Pin them to specific runners to make sure we
don't have both running on the same node.
Related: docker-playground I770a0530f90c5e03d2073bfc1c4cfd0e840379c9
Change-Id: I0162f7299c8e37f893ffa10ddc4c8edece29ed7f
---
M jobs/README.md
M jobs/ttcn3-testsuites.yml
2 files changed, 33 insertions(+), 24 deletions(-)
Approvals:
pespin: Looks good to me, but someone else must approve
laforge: Looks good to me, but someone else must approve
Jenkins Builder: Verified
osmith: Looks good to me, approved
diff --git a/jobs/README.md b/jobs/README.md
index bc3b1c6..1997d33 100644
--- a/jobs/README.md
+++ b/jobs/README.md
@@ -172,8 +172,12 @@
00:XX registry-update-base-images
00:XX simtester-sanitize
-03:00 - 18:00 Jobs that need binary packages
- 03:00 - 08:00 ttcn3-testsuites
+01:00 - 06:00 ttcn3-bts-test (fail if the builders are under too much load!)
+ 01:30 ttcn3-bts-test, ttcn3-bts-test-io_uring
+ 03:30 ttcn3-bts-test-latest, ttcn3-bts-test-asan
+
+06:00 - 18:00 Other jobs that need binary packages
+ 06:00 - 08:00 ttcn3-testsuites (misc without testenv)
08:00 - 18:00 ttcn3-testsuites-testenv
04:XX osmocom-release-manuals
05:XX osmocom-release-tarballs
diff --git a/jobs/ttcn3-testsuites.yml b/jobs/ttcn3-testsuites.yml
index 0ac11b6..d842fff 100644
--- a/jobs/ttcn3-testsuites.yml
+++ b/jobs/ttcn3-testsuites.yml
@@ -21,44 +21,44 @@
job-name:
# debian nightly - order by time
- nplab-m3ua-test: # ~1 min
- timer: H 03 * * *
+ timer: H 06 * * *
- nplab-sua-test: # ~1 min
- timer: H 03 * * *
+ timer: H 06 * * *
- ttcn3-gbproxy-test: # ~15 min
- timer: H 03 * * *
+ timer: H 06 * * *
- ttcn3-gbproxy-test-fr: # ~26 min
- timer: H 03 * * *
+ timer: H 06 * * *
node: hdlc
- ttcn3-ns-test-fr: # ~10 min
- timer: H 04 * * *
+ timer: H 06 * * *
node: hdlc
- ttcn3-fr-test: # ~ 10 min
- timer: H 04 * * *
+ timer: H 06 * * *
node: hdlc
- ttcn3-pgw-test: # ~7 min
- timer: H 04 * * *
+ timer: H 06 * * *
description: |
Test open5gs-smfd + open5gs-upfd PGW main.
email: jenkins-notifications(a)lists.osmocom.org acetcom(a)gmail.com
- ttcn3-epdg-test: # ~4 min
- timer: H 04 * * *
+ timer: H 06 * * *
# osmo-epdg requires OTP and kernel >= debian12:
node: build4-deb12build-ansible
- ttcn3-asterisk-ims-ue-test: # ~19 min
- timer: H 04 * * *
+ timer: H 06 * * *
# debian latest stable
- nplab-m3ua-test-latest: # ~1 min
- timer: H 04 * * *
+ timer: H 07 * * *
- nplab-sua-test-latest: # ~1 min
- timer: H 04 * * *
+ timer: H 07 * * *
- ttcn3-gbproxy-test-latest: # ~15 min
- timer: H 05 * * *
+ timer: H 07 * * *
- ttcn3-gbproxy-test-fr-latest: # ~23 min
- timer: H 05 * * *
+ timer: H 07 * * *
node: hdlc
- ttcn3-epdg-test-latest: # ~5 min
- timer: H 05 * * *
+ timer: H 07 * * *
# osmo-epdg requires OTP and kernel >= debian12:
node: build4-deb12build-ansible
@@ -66,29 +66,34 @@
# centos7 nightly
- TTCN3-centos7-pcap-client-test: # ~1 min
- timer: H 05 * * *
+ timer: H 07 * * *
# centos7 latest
- TTCN3-centos7-pcap-client-test-latest: # ~1 min
- timer: H 05 * * *
+ timer: H 07 * * *
# With LIBOSMO_IO_BACKEND=IO_URING (OS#6357)
- ttcn3-gbproxy-test-io_uring: # ~15 min
- timer: H 05 * * *
+ timer: H 07 * * *
# BTS testsuites tend to fail with timeouts on heavy loads. Group them
- # here in a timeslot where jenkins nodes are otherwise not busy according
+ # here in timeslots where jenkins nodes are otherwise not busy according
# to stats in grafana to hopefully make the test results more stable.
+ # See also jobs/README.md.
# Once we can replace faketrx with a rewrite (OS#6672), this should not
# be necessary anymore.
- ttcn3-bts-test-io_uring: # ~135 min
- timer: 0 06 * * *
+ timer: 30 01 * * *
+ node: build4-deb12build-ansible
- ttcn3-bts-test: # ~135 min
- timer: 0 06 * * *
+ timer: 30 01 * * *
+ node: build5-deb12build-ansible
- ttcn3-bts-test-latest: # ~135 min
- timer: 0 06 * * *
+ timer: 30 03 * * *
+ node: build4-deb12build-ansible
- ttcn3-bts-test-asan: # ~135 min
- timer: 0 06 * * *
+ timer: 30 03 * * *
+ node: build5-deb12build-ansible
# With sanitizer (OS#5301)
- ttcn3-gbproxy-test-asan: # ~15 min
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40142?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: merged
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I0162f7299c8e37f893ffa10ddc4c8edece29ed7f
Gerrit-Change-Number: 40142
Gerrit-PatchSet: 2
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Gerrit-Reviewer: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>