osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/40142?usp=email ) Change subject: jobs: run ttcn3-bts-test-* between 01:00 - 06:00 ...................................................................... jobs: run ttcn3-bts-test-* between 01:00 - 06:00 The automatic respawns of osmo-bts-trx have been intentionally removed in the related docker-playground patch, so we can better notice when osmo-bts-trx crashes instead of just restarting it and moving on with the testsuite. The result is that now it is even more important when ttcn3-bts-test jobs are running, because when the builders are under high load then faketrx cannot send the time quickly enough and it also causes osmo-bts-trx to shutdown. In a discussion with Vadim we've decided to schedule the ttcn3-bts-tests so that they don't only run at a time where no other jobs are running. But also that they don't run at the same time on the same builder anymore to give them the least unrelated load possible. Other solutions are also being investigated, but this should make the job in its current form more stable already. Move misc ttcn3 jobs to run from 06:00 to 08:00 and run ttcn3-bts-tests between 01:00 and 06:00. Pin them to specific runners to make sure we don't have both running on the same node. Related: docker-playground I770a0530f90c5e03d2073bfc1c4cfd0e840379c9 Change-Id: I0162f7299c8e37f893ffa10ddc4c8edece29ed7f --- M jobs/README.md M jobs/ttcn3-testsuites.yml 2 files changed, 33 insertions(+), 24 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/42/40142/1 diff --git a/jobs/README.md b/jobs/README.md index bc3b1c6..1997d33 100644 --- a/jobs/README.md +++ b/jobs/README.md @@ -172,8 +172,12 @@ 00:XX registry-update-base-images 00:XX simtester-sanitize -03:00 - 18:00 Jobs that need binary packages - 03:00 - 08:00 ttcn3-testsuites +01:00 - 06:00 ttcn3-bts-test (fail if the builders are under too much load!) + 01:30 ttcn3-bts-test, ttcn3-bts-test-io_uring + 03:30 ttcn3-bts-test-latest, ttcn3-bts-test-asan + +06:00 - 18:00 Other jobs that need binary packages + 06:00 - 08:00 ttcn3-testsuites (misc without testenv) 08:00 - 18:00 ttcn3-testsuites-testenv 04:XX osmocom-release-manuals 05:XX osmocom-release-tarballs diff --git a/jobs/ttcn3-testsuites.yml b/jobs/ttcn3-testsuites.yml index 0ac11b6..d842fff 100644 --- a/jobs/ttcn3-testsuites.yml +++ b/jobs/ttcn3-testsuites.yml @@ -21,44 +21,44 @@ job-name: # debian nightly - order by time - nplab-m3ua-test: # ~1 min - timer: H 03 * * * + timer: H 06 * * * - nplab-sua-test: # ~1 min - timer: H 03 * * * + timer: H 06 * * * - ttcn3-gbproxy-test: # ~15 min - timer: H 03 * * * + timer: H 06 * * * - ttcn3-gbproxy-test-fr: # ~26 min - timer: H 03 * * * + timer: H 06 * * * node: hdlc - ttcn3-ns-test-fr: # ~10 min - timer: H 04 * * * + timer: H 06 * * * node: hdlc - ttcn3-fr-test: # ~ 10 min - timer: H 04 * * * + timer: H 06 * * * node: hdlc - ttcn3-pgw-test: # ~7 min - timer: H 04 * * * + timer: H 06 * * * description: | Test open5gs-smfd + open5gs-upfd PGW main. email: jenkins-notifications(a)lists.osmocom.org acetcom(a)gmail.com - ttcn3-epdg-test: # ~4 min - timer: H 04 * * * + timer: H 06 * * * # osmo-epdg requires OTP and kernel >= debian12: node: build4-deb12build-ansible - ttcn3-asterisk-ims-ue-test: # ~19 min - timer: H 04 * * * + timer: H 06 * * * # debian latest stable - nplab-m3ua-test-latest: # ~1 min - timer: H 04 * * * + timer: H 07 * * * - nplab-sua-test-latest: # ~1 min - timer: H 04 * * * + timer: H 07 * * * - ttcn3-gbproxy-test-latest: # ~15 min - timer: H 05 * * * + timer: H 07 * * * - ttcn3-gbproxy-test-fr-latest: # ~23 min - timer: H 05 * * * + timer: H 07 * * * node: hdlc - ttcn3-epdg-test-latest: # ~5 min - timer: H 05 * * * + timer: H 07 * * * # osmo-epdg requires OTP and kernel >= debian12: node: build4-deb12build-ansible @@ -66,29 +66,34 @@ # centos7 nightly - TTCN3-centos7-pcap-client-test: # ~1 min - timer: H 05 * * * + timer: H 07 * * * # centos7 latest - TTCN3-centos7-pcap-client-test-latest: # ~1 min - timer: H 05 * * * + timer: H 07 * * * # With LIBOSMO_IO_BACKEND=IO_URING (OS#6357) - ttcn3-gbproxy-test-io_uring: # ~15 min - timer: H 05 * * * + timer: H 07 * * * # BTS testsuites tend to fail with timeouts on heavy loads. Group them - # here in a timeslot where jenkins nodes are otherwise not busy according + # here in timeslots where jenkins nodes are otherwise not busy according # to stats in grafana to hopefully make the test results more stable. + # See also jobs/README.md. # Once we can replace faketrx with a rewrite (OS#6672), this should not # be necessary anymore. - ttcn3-bts-test-io_uring: # ~135 min - timer: 0 06 * * * + timer: 30 01 * * * + node: build4-deb12build-ansible - ttcn3-bts-test: # ~135 min - timer: 0 06 * * * + timer: 30 01 * * * + node: build5-deb12build-ansible - ttcn3-bts-test-latest: # ~135 min - timer: 0 06 * * * + timer: 30 03 * * * + node: build4-deb12build-ansible - ttcn3-bts-test-asan: # ~135 min - timer: 0 06 * * * + timer: 30 03 * * * + node: build5-deb12build-ansible # With sanitizer (OS#5301) - ttcn3-gbproxy-test-asan: # ~15 min -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/40142?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I0162f7299c8e37f893ffa10ddc4c8edece29ed7f Gerrit-Change-Number: 40142 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de>