osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/40142?usp=email )
Change subject: jobs: run ttcn3-bts-test-* between 01:00 - 06:00
......................................................................
jobs: run ttcn3-bts-test-* between 01:00 - 06:00
The automatic respawns of osmo-bts-trx have been intentionally removed
in the related docker-playground patch, so we can better notice when
osmo-bts-trx crashes instead of just restarting it and moving on with
the testsuite.
The result is that now it is even more important when ttcn3-bts-test
jobs are running, because when the builders are under high load then
faketrx cannot send the time quickly enough and it also causes
osmo-bts-trx to shutdown.
In a discussion with Vadim we've decided to schedule the ttcn3-bts-tests
so that they don't only run at a time where no other jobs are running.
But also that they don't run at the same time on the same builder
anymore to give them the least unrelated load possible. Other solutions
are also being investigated, but this should make the job in its current
form more stable already.
Move misc ttcn3 jobs to run from 06:00 to 08:00 and run ttcn3-bts-tests
between 01:00 and 06:00. Pin them to specific runners to make sure we
don't have both running on the same node.
Related: docker-playground I770a0530f90c5e03d2073bfc1c4cfd0e840379c9
Change-Id: I0162f7299c8e37f893ffa10ddc4c8edece29ed7f
---
M jobs/README.md
M jobs/ttcn3-testsuites.yml
2 files changed, 33 insertions(+), 24 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/42/40142/1
diff --git a/jobs/README.md b/jobs/README.md
index bc3b1c6..1997d33 100644
--- a/jobs/README.md
+++ b/jobs/README.md
@@ -172,8 +172,12 @@
00:XX registry-update-base-images
00:XX simtester-sanitize
-03:00 - 18:00 Jobs that need binary packages
- 03:00 - 08:00 ttcn3-testsuites
+01:00 - 06:00 ttcn3-bts-test (fail if the builders are under too much load!)
+ 01:30 ttcn3-bts-test, ttcn3-bts-test-io_uring
+ 03:30 ttcn3-bts-test-latest, ttcn3-bts-test-asan
+
+06:00 - 18:00 Other jobs that need binary packages
+ 06:00 - 08:00 ttcn3-testsuites (misc without testenv)
08:00 - 18:00 ttcn3-testsuites-testenv
04:XX osmocom-release-manuals
05:XX osmocom-release-tarballs
diff --git a/jobs/ttcn3-testsuites.yml b/jobs/ttcn3-testsuites.yml
index 0ac11b6..d842fff 100644
--- a/jobs/ttcn3-testsuites.yml
+++ b/jobs/ttcn3-testsuites.yml
@@ -21,44 +21,44 @@
job-name:
# debian nightly - order by time
- nplab-m3ua-test: # ~1 min
- timer: H 03 * * *
+ timer: H 06 * * *
- nplab-sua-test: # ~1 min
- timer: H 03 * * *
+ timer: H 06 * * *
- ttcn3-gbproxy-test: # ~15 min
- timer: H 03 * * *
+ timer: H 06 * * *
- ttcn3-gbproxy-test-fr: # ~26 min
- timer: H 03 * * *
+ timer: H 06 * * *
node: hdlc
- ttcn3-ns-test-fr: # ~10 min
- timer: H 04 * * *
+ timer: H 06 * * *
node: hdlc
- ttcn3-fr-test: # ~ 10 min
- timer: H 04 * * *
+ timer: H 06 * * *
node: hdlc
- ttcn3-pgw-test: # ~7 min
- timer: H 04 * * *
+ timer: H 06 * * *
description: |
Test open5gs-smfd + open5gs-upfd PGW main.
email: jenkins-notifications(a)lists.osmocom.org acetcom(a)gmail.com
- ttcn3-epdg-test: # ~4 min
- timer: H 04 * * *
+ timer: H 06 * * *
# osmo-epdg requires OTP and kernel >= debian12:
node: build4-deb12build-ansible
- ttcn3-asterisk-ims-ue-test: # ~19 min
- timer: H 04 * * *
+ timer: H 06 * * *
# debian latest stable
- nplab-m3ua-test-latest: # ~1 min
- timer: H 04 * * *
+ timer: H 07 * * *
- nplab-sua-test-latest: # ~1 min
- timer: H 04 * * *
+ timer: H 07 * * *
- ttcn3-gbproxy-test-latest: # ~15 min
- timer: H 05 * * *
+ timer: H 07 * * *
- ttcn3-gbproxy-test-fr-latest: # ~23 min
- timer: H 05 * * *
+ timer: H 07 * * *
node: hdlc
- ttcn3-epdg-test-latest: # ~5 min
- timer: H 05 * * *
+ timer: H 07 * * *
# osmo-epdg requires OTP and kernel >= debian12:
node: build4-deb12build-ansible
@@ -66,29 +66,34 @@
# centos7 nightly
- TTCN3-centos7-pcap-client-test: # ~1 min
- timer: H 05 * * *
+ timer: H 07 * * *
# centos7 latest
- TTCN3-centos7-pcap-client-test-latest: # ~1 min
- timer: H 05 * * *
+ timer: H 07 * * *
# With LIBOSMO_IO_BACKEND=IO_URING (OS#6357)
- ttcn3-gbproxy-test-io_uring: # ~15 min
- timer: H 05 * * *
+ timer: H 07 * * *
# BTS testsuites tend to fail with timeouts on heavy loads. Group them
- # here in a timeslot where jenkins nodes are otherwise not busy according
+ # here in timeslots where jenkins nodes are otherwise not busy according
# to stats in grafana to hopefully make the test results more stable.
+ # See also jobs/README.md.
# Once we can replace faketrx with a rewrite (OS#6672), this should not
# be necessary anymore.
- ttcn3-bts-test-io_uring: # ~135 min
- timer: 0 06 * * *
+ timer: 30 01 * * *
+ node: build4-deb12build-ansible
- ttcn3-bts-test: # ~135 min
- timer: 0 06 * * *
+ timer: 30 01 * * *
+ node: build5-deb12build-ansible
- ttcn3-bts-test-latest: # ~135 min
- timer: 0 06 * * *
+ timer: 30 03 * * *
+ node: build4-deb12build-ansible
- ttcn3-bts-test-asan: # ~135 min
- timer: 0 06 * * *
+ timer: 30 03 * * *
+ node: build5-deb12build-ansible
# With sanitizer (OS#5301)
- ttcn3-gbproxy-test-asan: # ~15 min
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40142?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I0162f7299c8e37f893ffa10ddc4c8edece29ed7f
Gerrit-Change-Number: 40142
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Attention is currently required from: fixeria, osmith.
pespin has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email )
Change subject: testenv: support fetching coredumps in jenkins
......................................................................
Patch Set 2:
(1 comment)
File _testenv/testenv/coredump.py:
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140/comment/90def1a0_784f… :
PS2, Line 35: def get_from_coredumpctl_lxc_host():
I'm missing in here some reference to the script providing this service.
--
To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18
Gerrit-Change-Number: 40140
Gerrit-PatchSet: 2
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: osmith <osmith(a)sysmocom.de>
Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 24 Apr 2025 12:28:57 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Attention is currently required from: osmith.
pespin has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ci/+/40139?usp=email )
Change subject: jobs/ttcn3-testsuites-testenv: set core env var
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40139?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I8359b0faa1fed76b430749589916cd072a8a7753
Gerrit-Change-Number: 40139
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: osmith <osmith(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 24 Apr 2025 12:26:02 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Attention is currently required from: osmith.
pespin has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email )
Change subject: ansible: build-hosts: add testenv-coredump-helper
......................................................................
Patch Set 1:
(2 comments)
File ansible/roles/testenv-coredump-helper/files/testenv-coredump-helper.py:
https://gerrit.osmocom.org/c/osmo-ci/+/40138/comment/32eaa762_a7281cee?usp=… :
PS1, Line 4: # OS#6769: simple webserver to make Osmocom related coredumps available in LXCs
I'm really lacking a big chunk of descriptions here regarding what this script does, where it is spected to run (host? lxc?) as most important, *how* it does it.
I see some webs erver being involved for instance, but no mention anywhere on how that's supposed to be working or what are they exactly speaking.
https://gerrit.osmocom.org/c/osmo-ci/+/40138/comment/74bc58ba_f78ef890?usp=… :
PS1, Line 66: # Put coredump into a temporary file and return it
this may be problematic if it ends up in tmpfs, a coredump can be veery large.
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40138?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I7e66c98106b7028a393e3b873e96ae2dcb412c48
Gerrit-Change-Number: 40138
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-CC: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: osmith <osmith(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 24 Apr 2025 12:25:28 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Attention is currently required from: fixeria, pespin.
Hello Jenkins Builder, fixeria, pespin,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email
to look at the new patch set (#2).
The following approvals got outdated and were removed:
Verified+1 by Jenkins Builder
Change subject: testenv: support fetching coredumps in jenkins
......................................................................
testenv: support fetching coredumps in jenkins
The Osmocom jenkins nodes run inside LXCs. When we get a coredump it
appears on the host, fetch it from there via testenv-coredump-helper,
which gets added to the hosts in the related patch.
Related: osmo-ci I7e66c98106b7028a393e3b873e96ae2dcb412c48
Related: OS#6769
Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18
---
M _testenv/README.md
M _testenv/testenv/coredump.py
2 files changed, 61 insertions(+), 1 deletion(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/40/40140/2
--
To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40140?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Change-Id: I3784b4cbcef08b26f77b6f6f7a70a830d9c81a18
Gerrit-Change-Number: 40140
Gerrit-PatchSet: 2
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de>