August 2024 - gerrit-log - lists.osmocom.org

[XL] Change in osmo-ttcn3-hacks[master]: testenv: add test environment script

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37694?usp=email ) Change subject: testenv: add test environment script ...................................................................... testenv: add test environment script Add a new testenv.py script that builds/installs all components needed for a testsuite, builds the testsuite from source and runs it. Features: * --binary-repo argument to install packages from osmocom:latest or any other repository from the Osmocom OBS instead of building from source * without --binary-repo, the test components are built with osmo-dev, cloning the missing source git repositories and building them in the right order * --podman argument to run the testsuite and its components inside a container (using podman instead of docker so it runs rootless) * Simple testenv.cfg file to specify components for running testsuites * Iterative compilation of components and testsuite * Using ccache * Testsuite doesn't start if any of the components fail to start (e.g. because of a config error) * Testsuite gets stopped if any of the components crash * ^C stops the testsuite + all components * Test component output logs to stdout in addition to a log file (turn off with --no-tee) * --test argument to only run one specific test * --shell argument to run an interactive shell before teardown to inspect the test environment while components are still running This script unifies the use cases of running a testsuite without containers (for local development), and with containers (as jenkins runs it, but can also be used for local development e.g. to get a clean pcap). Previously jenkins used a different set of configurations from docker-playground.git and many different containers instead of just one. Related: OS#6494 Change-Id: If9f8b79dd6e5b4f06be4e5ff73db97759c3acfb2 --- M .gitignore M README.md A _testenv/README.md A _testenv/data/osmo-dev/osmo-bts-trx.opts A _testenv/data/podman/Dockerfile A _testenv/data/podman/obs.key A _testenv/data/scripts/log_format.sh A _testenv/data/scripts/rename_junit_xml_classname.sh A _testenv/data/scripts/respawn.sh A _testenv/data/scripts/testenv-podman-main.sh A _testenv/pyproject.toml A _testenv/testenv.py A _testenv/testenv/__init__.py A _testenv/testenv/cmd.py A _testenv/testenv/daemons.py A _testenv/testenv/osmo_dev.py A _testenv/testenv/podman.py A _testenv/testenv/podman_install.py A _testenv/testenv/requirements.py A _testenv/testenv/testdir.py A _testenv/testenv/testenv_cfg.py A _testenv/testenv/testsuite.py A testenv.py 23 files changed, 2,240 insertions(+), 0 deletions(-) Approvals: pespin: Looks good to me, but someone else must approve osmith: Looks good to me, approved laforge: Looks good to me, but someone else must approve Jenkins Builder: Verified diff --git a/.gitignore b/.gitignore index e77dfa7..f03b0b3 100644 --- a/.gitignore +++ b/.gitignore @@ -22,3 +22,4 @@ sms.db sms.db-shm sms.db-wal +__pycache__ diff --git a/README.md b/README.md index 907ebb1..c84f31c 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,14 @@ network elements, from 2G, 3G, 4G to 5G. The individual test-suites are in sub-directories, while some shared library code is in *library*. +Running Testsuites +------------------ + +Use the `testenv.py` script to run the testsuites, e.g.: + +``` +$ ./testenv.py run mgw +``` Continuous Integration ---------------------- diff --git a/_testenv/README.md b/_testenv/README.md new file mode 100644 index 0000000..2c5a181 --- /dev/null +++ b/_testenv/README.md @@ -0,0 +1,142 @@ +# testenv + +Build everything needed for running Osmocom TTCN-3 testsuites and execute them. + +## testenv.cfg + +The `testenv.cfg` file has one `[testsuite]` section, typically with one or +more sections for test components. + +### Example + +```ini +[testsuite] +program=MGCP_Test +config=MGCP_Test.cfg + +[mgw] +program=osmo-mgw +make=osmo-mgw +package=osmo-mgw +copy=osmo-mgw.cfg +``` + +### Keys + +#### Testsuite section + +* `program=` the executable for starting the testsuite, without arguments. + +* `config=`: the testsuite configuration file. + +* `copy=`: additional file(s) to copy from the testsuite directory to the test + directory, useful for include files mentioned in the config. Multiple values + are separated by spaces. + +* `clean=`: optional script to run before running the testsuite and on exit. + This can be used to clean up network devices for example, or to fix name + collisions when running a test with multiple configs + (`rename_junit_xml_classname.sh`). See below for `PATH`. A + `TESTENV_CLEAN_REASON` env var is set to `prepare`, `crashed` or `finished` + depending on when the script runs. + +#### Component section + +* `program=`: executable for starting a test component, may contain arguments. + See below for `PATH`. + +* `copy=`: file(s) to copy from the testsuite directory to the test directory, + like `.cfg` and `.confmerge` files. Multiple values are separated by spaces. + +* `make=`: osmo-dev make target for building from source, if running without + `--binary-repo`. This is usually the name of the git repository, but could + also be e.g. `.make.osmocom-bb.clone` to only clone and not build + `osmocom-bb.git` for faketrx. Set `make=no` to not build/clone anything. + +* `package=`: debian package(s) to be installed for running a test component + and the script in `prepare=`. Multiple values separated by spaces. Set to + `package=no` to not install any package. + +* `prepare=`: optional script to run before staring the program (after files + are copied to the test directory). Typically this is used to create configs + with `osmo-config-merge`. See below for `PATH`. + +* `setup=`: optional script to run after the program was started. Execution of + the next program / the testsuite will wait until the setup script has quit. + This can be used to wait until the program is ready or to fill a test + database for example. See below for `PATH`. + +* `clean=`: optional script to run before `prepare=` and on exit. This can be + used to clean up network devices for example, or to fix name collisions when + running a test with multiple configs (`rename_junit_xml_classname.sh`). See + below for `PATH`. A `TESTENV_CLEAN_REASON` env var is set to `prepare`, + `crashed` or `finished` depending on when the script runs. + +### PATH + +Executables mentioned in `program=`, `prepare=`, `setup=` and `clean=` run +with a `PATH` environment variable containing: + +* The directory of the testsuite +* The directory for binaries built from source +* The directory `_testenv/data/scripts` (which has e.g. `respawn.sh`) + +### Latest configs + +Sometimes we need to run test components and/or testsuites with different +configurations depending on whether we are currently testing the nightly/master +versions of test components or the latest (stable) versions. For example, when +a new feature gets introduced that we need to configure and test, but which is +not available in the latest stable version. + +For this purpose, it is possible to add configuration keys ending in `_latest` +to `testenv.cfg`. These keys will override the original keys if `testenv.py` +is running with a binary repository ending in `:latest` or with `--latest`. + +It is also possible to take different code paths or exclude tests in the +TTCN-3 code if the latest configs are (not) used. This is done with +`f_osmo_repo_is` in `library/Misc_Helpers.ttcn`. + +### Multiple testenv.cfg files + +Usually each testsuite has only one `testenv.cfg` file, e.g. `mgw/testenv.cfg`. +For some testsuites it is necessary to run them multiple times with slightly +different configurations. In that case, we have multiple `testenv.cfg` files, +typically one `testenv_generic.cfg` and additional `testenv_<NAME>.cfg` files. + +For example: +* `bts/testenv_generic.cfg` +* `bts/testenv_hopping.cfg` +* `bts/testenv_oml.cfg` + +## Environment variables + +* `TESTENV_SRC_DIR`: + Set the directory for sources of Osmocom components. The default is the + directory above your osmo-ttcn3-hacks.git clone. + +* `TESTENV_NO_IMAGE_UP_TO_DATE_CHECK`: + Do not compare the timestamp of `data/podman/Dockerfile` with the date of the + podman image. This check does not work on jenkins where we always have + a fresh clone. + +* `TESTENV_COLOR_{DEBUG|INFO|WARNING|ERROR|CRITICAL|RESET}`: + Change the colors for different log levels (we use this in Jenkins, which + prints the output on white background). Find the defaults in + `testenv/__init__.py:ColorFormatter()`. + +* `TESTENV_SOURCE_HIGHLIGHT_COLORS`: + The argument to pass to `source-highlight` for formatting the junit log xml + files. Jenkins sets this to `esc` instead of `esc256` for better contrast on + white background. + +## Troubleshooting + +### Timeout waiting for RESET-ACK after sending RESET + +This can happen if another Osmocom program is started before OsmoSTP, then +tries to connect to OsmoSTP and fails, and waits several seconds before +retrying. Check the logs to confirm this is the case. To fix this, adjust your +`testenv.cfg` to start OsmoSTP before other Osmocom programs. The testenv +scripts will wait a bit to give OsmoSTP enough time to start up, before +starting the other test components. diff --git a/_testenv/data/osmo-dev/osmo-bts-trx.opts b/_testenv/data/osmo-dev/osmo-bts-trx.opts new file mode 100644 index 0000000..8acec2d --- /dev/null +++ b/_testenv/data/osmo-dev/osmo-bts-trx.opts @@ -0,0 +1 @@ +osmo-bts --enable-trx diff --git a/_testenv/data/podman/Dockerfile b/_testenv/data/podman/Dockerfile new file mode 100644 index 0000000..95a8c66 --- /dev/null +++ b/_testenv/data/podman/Dockerfile @@ -0,0 +1,123 @@ +ARG REGISTRY=docker.io +ARG DISTRO=debian:12 +FROM ${REGISTRY}/${DISTRO} + +# Arguments used after FROM must be specified again +ARG OSMOCOM_REPO_TESTSUITE_MIRROR="https://downloads.osmocom.org" +ARG OSMOCOM_REPO="$OSMOCOM_REPO_TESTSUITE_MIRROR/packages/osmocom:/latest/Debian_12/" + +# Copy from common dir +COPY obs.key /obs.key + +# Install packages from Debian repositories (alphabetic order) +ENV DEBIAN_FRONTEND=noninteractive +RUN set -x && \ + apt-get update && \ + apt-get install -y --no-install-recommends \ + autoconf \ + automake \ + bison \ + build-essential \ + ca-certificates \ + ccache \ + cmake \ + flex \ + git \ + iproute2 \ + iputils-ping \ + libbson-dev \ + libc-ares-dev \ + libcsv-dev \ + libcurl4-gnutls-dev \ + libdbd-sqlite3 \ + libdbi-dev \ + libgcrypt-dev \ + libgnutls28-dev \ + libidn11-dev \ + libjansson-dev \ + libmicrohttpd-dev \ + libmnl-dev \ + libmongoc-dev \ + libnghttp2-dev \ + libortp-dev \ + libpcap-dev \ + libpcsclite-dev \ + libsctp-dev \ + libsofia-sip-ua-glib-dev \ + libsqlite3-dev \ + libssl-dev \ + libtalloc-dev \ + libtins-dev \ + libtool \ + libulfius-dev \ + liburing-dev \ + libusb-1.0-0-dev \ + libyaml-dev \ + libzmq3-dev \ + meson \ + netcat-openbsd \ + pcscd \ + pkg-config \ + procps \ + psmisc \ + python3-pip \ + rebar3 \ + rsync \ + source-highlight \ + sqlite3 \ + sudo \ + tcpdump \ + vim \ + wget \ + wireshark-common \ + && \ + apt-get clean + +# Ccache is installed above so it can be optionally used when rebuilding the +# testsuites inside the docker containers. Don't use it by default. +ENV USE_CCACHE=0 + +# Binary-only transcoding library for RANAP/RUA/HNBAP to work around TITAN only implementing BER +RUN set -x && \ + export DPKG_ARCH="$(dpkg --print-architecture)" && \ + wget https://ftp.osmocom.org/binaries/libfftranscode/libfftranscode0_0.5_${DPKG_… && \ + wget https://ftp.osmocom.org/binaries/libfftranscode/libfftranscode-dev_0.5_${DP… && \ + dpkg -i ./libfftranscode0_0.5_${DPKG_ARCH}.deb ./libfftranscode-dev_0.5_${DPKG_ARCH}.deb && \ + apt-get install --fix-broken && \ + rm libfftranscode*.deb + +# Install osmo-python-tests (for obtaining talloc reports from SUT) +ADD https://gerrit.osmocom.org/plugins/gitiles/python/osmo-python-tests/+/maste… /tmp/osmo-python-tests-commit +RUN set -x && \ + git clone --depth=1 https://gerrit.osmocom.org/python/osmo-python-tests osmo-python-tests && \ + pip3 install ./osmo-python-tests --break-system-packages && \ + rm -rf osmo-python-tests + +# Add eclipse-titan from osmocom:latest, invalidate cache when :latest changes +RUN echo "deb [signed-by=/obs.key] $OSMOCOM_REPO ./" \ + > /etc/apt/sources.list.d/osmocom-latest.list +ADD $OSMOCOM_REPO/Release /tmp/Release +RUN set -x && \ + apt-get update && \ + apt-get install -y --no-install-recommends \ + eclipse-titan \ + && \ + apt-get clean && \ + rm /etc/apt/sources.list.d/osmocom-latest.list + +# Add mongodb for open5gs-hss. Using the package from bullseye since bookworm +# mongodb-org package is not available. Furthermore, manually install required +# libssl1.1. +RUN set -x && \ + mkdir -p /tmp/mongodb && \ + cd /tmp/mongodb && \ + wget "https://pgp.mongodb.com/server-5.0.asc" -O "/mongodb.key" && \ + wget "http://security.debian.org/debian-security/pool/updates/main/o/openssl/libs…" && \ + dpkg -i "libssl1.1_1.1.1n-0+deb10u6_amd64.deb" && \ + echo "deb [signed-by=/mongodb.key] http://repo.mongodb.org/apt/debian bullseye/mongodb-org/5.0 main" \ + > /etc/apt/sources.list.d/mongodb-org.list && \ + apt-get update && \ + apt-get install -y mongodb-org && \ + apt-get clean && \ + cd / && \ + rm -rf /tmp/mongodb diff --git a/_testenv/data/podman/obs.key b/_testenv/data/podman/obs.key new file mode 100644 index 0000000..ecca084 --- /dev/null +++ b/_testenv/data/podman/obs.key @@ -0,0 +1,26 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- +Version: GnuPG v1.4.5 (GNU/Linux) + +mQENBGKzE1QBCADFcM3ZzggvgxNRNNqDGWf5xIDAiK5qzFLdGes7L6F9VCHdaPy0 +RAOB5bFb/Q1tSDFNEBLtaauXKz+4iGL6qMVjZcyjzpB5w4jKN+kkrFRhjDNUv/SH +BX6d+P7v5WBGSNArNgA8D1BGzckp5a99EZ0okMJFEqIcN40PD6OGugpq5XnVV5Nk +e93fLa2Cu8vhFBcVn6CuHeEhsmuMf6NLbQRNfNNCEEUYaZn7beMYtpZ7t1djsKx5 +1xGm50OzI22FLu8lELQ9d7qMVGRG3WHYawX9BDteRybiyqxfwUHm1haWazRJtlGt +UWyzvwAb80BK1J2Nu5fbAa3w5CoEPAbUuCyrABEBAAG0JW9zbW9jb20gT0JTIFBy +b2plY3QgPG9zbW9jb21Ab3Ntb2NvbT6JAVQEEwEIAD4WIQRrKp83ktFetw1Oao+G +pzC2U3JZcwUCYrMV4wIbAwUJBB6yjwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAK +CRCGpzC2U3JZc4FRCACQQkKIrnvQ7n2u7GSmyVZa3I+oLoFXSGqaGyey5TW/nrMm +vFDKU3qliHiuNSmUY35SnAhXUsvqOYppxVRoO1MLrqUvzMOnIWqkJpf8mtjGUnsW +jyVeto7Rsjs75y2i1Hk+e7ljb/V65J3NlfrfEYWbqR9AKd53ReNXTdrQ0J05A38N +GdI4Ld/2lNISAwaBmGhqdeKsLHpQw/JERU1TApVJR1whFiIwDF1rOCg9GPnNKIk7 +yRZdK267XzztrainX/cbPILyzUZEDhYs6wQuyACyQ1YUxZIxrwVfk7PMNay8CrLH +z42B73Ne5IAj8+op/3iJafFONLm7YXiDUFN+QDYAiQEzBBMBCAAdFiEExoiYhHND +S7aVYlnqa51NyAUyjdsFAmKzE1UACgkQa51NyAUyjdvuZgf+OXmr//i7u7Gg7eWB +7e0qUsyCId9lXS8J437x3K6ciJfD7/6RSy8TFW5Nglm/uSkbyq582I8t+SoOirMD +E6cg9U/5+h5s46bAf+Kd2XS/6tLGeNLM18i4el8CP06NpFzDrsKu76uYFpyRiiHD +otBdtgxeLJ83LugGfZslF+/5cigJkAJMhAdVvGO8h85R6fba8ZSOKtMKkaQRfi76 +nhyOrJPlLuS+DLEnHwdkOFgtKnxHdjM97K+Tx0gisb6uwaWroXfSLnhP8RTLLZZy +Z+noU1Hw3c+mn4c/NYbcC/uwHYHKRzuf9gHnQ3dGgv0Z5sbeLRVo92hjGj7Ftlyd +4hmKBg== +=HxK4 +-----END PGP PUBLIC KEY BLOCK----- diff --git a/_testenv/data/scripts/log_format.sh b/_testenv/data/scripts/log_format.sh new file mode 100755 index 0000000..a74affe --- /dev/null +++ b/_testenv/data/scripts/log_format.sh @@ -0,0 +1,8 @@ +#!/bin/sh -e +# Run ttcn3_logformat on all merged log files. + +for i in *.merged; do + temp="$i.temp" + ttcn3_logformat -o "$temp" "$i" + mv "$temp" "$i" +done diff --git a/_testenv/data/scripts/rename_junit_xml_classname.sh b/_testenv/data/scripts/rename_junit_xml_classname.sh new file mode 100755 index 0000000..031ef90 --- /dev/null +++ b/_testenv/data/scripts/rename_junit_xml_classname.sh @@ -0,0 +1,37 @@ +#!/bin/sh -e +# Some jenkins jobs run the same tests multiple times with different configs. +# Then we have a conflict of test names ("classname") in the test result files. +# This script renames the classnames in the test result files to avoid the +# conflict. Jenkins will them show them separated in the test result analyzer. + +SUFFIX="$1" + +if [ -z "$SUFFIX" ]; then + echo "usage: rename_junit_xml_classname.sh SUFFIX" + echo "example: rename_junit_xml_classname ':hopping'" + exit 1 +fi + +xmls="$(find -name 'junit-xml*.log')" + +if [ -z "$xmls" ]; then + case "$TESTENV_CLEAN_REASON" in + prepare|crashed) + # No xml files is expected + exit 0 + ;; + finished) + echo "ERROR: could not find any junit-xml*.log files!" + exit 1 + ;; + *) + echo "ERROR: invalid TESTENV_CLEAN_REASON: $TESTENV_CLEAN_REASON" + exit 1 + ;; + esac +fi + +for i in $xmls; do + echo "Adding '$SUFFIX' to classnames in: $i" + sed -i "s/classname='$[^']\+$'/classname='\1$SUFFIX'/g" $i +done diff --git a/_testenv/data/scripts/respawn.sh b/_testenv/data/scripts/respawn.sh new file mode 100755 index 0000000..3b079c7 --- /dev/null +++ b/_testenv/data/scripts/respawn.sh @@ -0,0 +1,22 @@ +#!/bin/sh +# Restart a given test component while the testsuite is running. + +trap "kill 0" EXIT + +SLEEP_BEFORE_RESPAWN=${SLEEP_BEFORE_RESPAWN:-0} + +i=0 +max_i=500 +while [ $i -lt $max_i ]; do + echo "respawn: $i: starting: $*" + $* & + LAST_PID=$! + wait $LAST_PID + echo "respawn: $i: stopped pid $LAST_PID with status $?" + if [ $SLEEP_BEFORE_RESPAWN -gt 0 ]; then + echo "respawn: sleeping $SLEEP_BEFORE_RESPAWN seconds..." + sleep $SLEEP_BEFORE_RESPAWN + fi + i=$(expr $i + 1) +done +echo "respawn: exiting after $max_i runs" diff --git a/_testenv/data/scripts/testenv-podman-main.sh b/_testenv/data/scripts/testenv-podman-main.sh new file mode 100755 index 0000000..d0bc586 --- /dev/null +++ b/_testenv/data/scripts/testenv-podman-main.sh @@ -0,0 +1,20 @@ +#!/bin/sh -e +# Simple watchdog script that exits if either: +# * testenv doesn't create /tmp/watchdog every 10s +# * 4 hours have passed +# This ensures the podman container stops a few seconds after a jenkins job was +# aborted, or if a test is stuck in a loop for hours. + +stop_time=$(($(date +%s) + 3600 * 4)) + +while [ $(date +%s) -lt $stop_time ]; do + sleep 10 + + if ! [ -e /tmp/watchdog ]; then + break + fi + + rm /tmp/watchdog +done + +exit 1 diff --git a/_testenv/pyproject.toml b/_testenv/pyproject.toml new file mode 100644 index 0000000..3719b74 --- /dev/null +++ b/_testenv/pyproject.toml @@ -0,0 +1,2 @@ +[tool.ruff] +line-length=120 diff --git a/_testenv/testenv.py b/_testenv/testenv.py new file mode 100755 index 0000000..6ed1e51 --- /dev/null +++ b/_testenv/testenv.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import logging +import os +import sys +import testenv +import testenv.cmd +import testenv.daemons +import testenv.osmo_dev +import testenv.podman +import testenv.podman_install +import testenv.requirements +import testenv.testdir +import testenv.testenv_cfg +import testenv.testsuite + + +def run(): + testenv.testenv_cfg.init() + + if not testenv.args.binary_repo: + testenv.osmo_dev.check_init_needed() + + testenv.requirements.check() + testenv.podman_install.init() + testenv.cmd.init_env() + testenv.testdir.init() + testenv.daemons.init() + + if testenv.args.podman: + testenv.podman.init() + testenv.podman.start() + + if not testenv.args.binary_repo: + testenv.osmo_dev.init() + + testenv.testsuite.init() + testenv.testsuite.build() + + # Run prepare functions of testsuites (may enable extra repos) + for cfg_name, cfg in testenv.testenv_cfg.cfgs.items(): + testenv.testenv_cfg.set_current(cfg_name) + testenv.testsuite.run_prepare_script(cfg) + + # Build all components first + if not testenv.args.binary_repo: + for cfg_name, cfg in testenv.testenv_cfg.cfgs.items(): + testenv.testenv_cfg.set_current(cfg_name) + testenv.osmo_dev.make(cfg) + + # Run the components + testsuite + cfg_count = 0 + for cfg_name, cfg in testenv.testenv_cfg.cfgs.items(): + testenv.testenv_cfg.set_current(cfg_name) + + if testenv.args.binary_repo: + testenv.podman.enable_binary_repo() + testenv.podman_install.packages(cfg, cfg_name) + + testenv.testdir.prepare(cfg_name, cfg) + testenv.daemons.start(cfg) + testenv.testsuite.run(cfg) + testenv.daemons.stop() + testenv.testdir.clean_run_scripts("finished") + testenv.testsuite.cat_junit_logs() + + cfg_count += 1 + testenv.set_log_prefix("[testenv]") + + # Restart podman container before running with another config + if testenv.args.podman: + restart = cfg_count < len(testenv.testenv_cfg.cfgs) + testenv.podman.stop(restart) + + +def init_podman(): + testenv.podman.init_image_name_distro() + testenv.podman.image_build() + + +def init_osmo_dev(): + testenv.osmo_dev.init_clone() + + +def clean(): + cache_dirs = [ + "git", + "host", + "podman", + ] + for cache_dir in cache_dirs: + path = os.path.join(testenv.cache_dir_default, cache_dir) + if os.path.exists(path): + testenv.cmd.run(["rm", "-rf", path]) + + +def main(): + testenv.init_logging() + testenv.init_args() + + action = testenv.args.action + if action == "run": + run() + elif action == "init": + if testenv.args.runtime == "osmo-dev": + init_osmo_dev() + elif testenv.args.runtime == "podman": + init_podman() + elif action == "clean": + clean() + + +try: + main() +except testenv.NoTraceException as e: + logging.error(e) + testenv.podman.stop() + sys.exit(2) +except KeyboardInterrupt: + print("") # new line + testenv.podman.stop() + sys.exit(3) +except: + testenv.podman.stop() + raise diff --git a/_testenv/testenv/__init__.py b/_testenv/testenv/__init__.py new file mode 100644 index 0000000..fe9a612 --- /dev/null +++ b/_testenv/testenv/__init__.py @@ -0,0 +1,218 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import argparse +import logging +import os.path + + +class NoTraceException(Exception): + pass + + +args = None + +src_dir = os.environ.get("TESTENV_SRC_DIR", os.path.realpath(f"{__file__}/../../../..")) +data_dir = os.path.join(os.path.realpath(f"{__file__}/../.."), "data") +distro_default = "debian:bookworm" +cache_dir_default = os.path.join(os.path.expanduser("~/.cache"), "osmo-ttcn3-testenv") +ccache_dir_default = os.path.join(cache_dir_default, "ccache") + +log_prefix = "[testenv]" + + +def resolve_testsuite_name_alias(name): + mapping = { + "ggsn": "ggsn_tests", + } + + if name in mapping: + logging.debug(f"Using testsuite {mapping[name]} (via alias {name})") + return mapping[name] + + return name + + +def parse_args(): + global args + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description="Build/install everything for a testsuite and run it.\n" + "\n" + "examples:\n" + " ./testenv.py run mgw\n" + " ./testenv.py run mgw --test TC_crcx\n" + " ./testenv.py run mgw --podman --binary-repo osmocom:latest\n" + " ./testenv.py run mgw --io-uring\n" + " ./testenv.py run bts --config oml\n", + ) + + sub = parser.add_subparsers(title="action", dest="action", required=True) + + sub_init = sub.add_parser("init", help="initialize osmo-dev/podman") + sub_init_runtime = sub_init.add_subparsers(title="runtime", required=True, dest="runtime") + sub_init_runtime.add_parser( + "osmo-dev", + help="prepare osmo-dev (top-level makefile scripts, for building test" + " components from source when using 'run' without '--binary-repo')", + ) + + sub_podman = sub_init_runtime.add_parser("podman", help="prepare the podman image (for 'run --podman')") + sub_podman.add_argument( + "-f", + "--force", + action="store_true", + help="build image even if it is up-to-date", + ) + + sub_run = sub.add_parser("run", help="build components and run a testsuite") + + group = sub_run.add_argument_group("testsuite options") + group.add_argument("testsuite", help="a directory in osmo-ttcn3-hacks.git (msc, bsc, mgw, ...)") + group.add_argument( + "-t", + "--test", + help="only run one specific test (e.g. TC_selftest, BTS_Tests_OML.TC_wrong_mdisc)", + ) + group.add_argument( + "-c", + "--config", + action="append", + help="which testenv.cfg to use, in case the testsuite has multiple (e.g. generic|oml|hopping for bts)", + ) + group.add_argument("-i", "--io-uring", action="store_true", help="set LIBOSMO_IO_BACKEND=IO_URING") + + group = sub_run.add_argument_group("source/binary options", "All components are built from source by default.") + group = group.add_mutually_exclusive_group() + group.add_argument( + "-b", + "--binary-repo", + metavar="OBS_PROJECT", + help="use binary packages from this Osmocom OBS project instead (e.g. osmocom:nightly)", + ) + group = sub_run.add_argument_group( + "config file options", + "Testsuite and test component configs" + " for nightly/master versions of test" + " components are used, unless a binary" + " repository ending in :latest is set" + " or --latest is used.", + ) + group.add_argument("--latest", action="store_true", help="use latest configs") + + group = sub_run.add_argument_group("podman options", "All components are run directly on the host by default.") + group.add_argument("-p", "--podman", action="store_true", help="run all components inside podman") + group.add_argument( + "-d", + "--distro", + default=distro_default, + help=f"distribution for podman (default: {distro_default})", + ) + group.add_argument( + "-s", + "--shell", + action="store_true", + help="run an interactive shell before stopping daemons/container", + ) + + group = sub_run.add_argument_group("output options") + group.add_argument("-l", "--log-dir", help="log here instead of a random dir in /tmp") + group.add_argument( + "-n", + "--no-tee", + dest="tee", + action="store_false", + help="don't send test component's output to stdout", + ) + + group = sub_run.add_argument_group("cache options") + group.add_argument( + "--cache", + help=f"cache path (default: {cache_dir_default})", + default=cache_dir_default, + ) + group.add_argument( + "--ccache", + help=f"ccache path (default: {ccache_dir_default})", + default=ccache_dir_default, + ) + + sub.add_parser("clean", help="clean previous build artifacts") + + args = parser.parse_args() + + if args.action == "run": + args.testsuite = resolve_testsuite_name_alias(args.testsuite) + if args.binary_repo and args.binary_repo.endswith(":latest"): + logging.debug("Binary repository ends in :latest, using latest configs") + args.latest = True + + +def verify_args_run(): + if args.action != "run": + return + + if args.binary_repo and not args.podman: + raise NoTraceException("--binary-repo requires --podman") + + ttcn3_hacks_dir_src = os.path.realpath(f"{__file__}/../../..") + testsuite_dir = os.path.join(ttcn3_hacks_dir_src, args.testsuite) + if not os.path.exists(testsuite_dir): + raise NoTraceException(f"testsuite dir not found: {testsuite_dir}") + + +def init_args(): + parse_args() + verify_args_run() + + +class ColorFormatter(logging.Formatter): + colors = { + "debug": "\033[37m", # light gray + "info": "\033[94m", # blue + "warning": "\033[93m", # yellow + "error": "\033[91m", # red + "critical": "\033[91m", # red + "reset": "\033[0m", + } + + def __init__(self): + for color in self.colors.keys(): + env_var = f"TESTENV_COLOR_{color.upper()}" + if env_var in os.environ: + self.colors[color] = os.environ.get(env_var) + + super().__init__() + + def format(self, record): + if record.levelno == logging.DEBUG: + color = "debug" + elif record.levelno == logging.INFO: + color = "info" + elif record.levelno == logging.WARNING: + color = "warning" + elif record.levelno == logging.ERROR: + color = "error" + elif record.levelno == logging.CRITICAL: + color = "critical" + + self._style._fmt = f"{self.colors[color]}{log_prefix} %(msg)s{self.colors['reset']}" + result = logging.Formatter.format(self, record) + + return result + + +def init_logging(): + formatter = ColorFormatter() + root_logger = logging.getLogger() + root_logger.setLevel(logging.DEBUG) + root_logger.handlers = [] + + handler = logging.StreamHandler() + handler.setFormatter(formatter) + root_logger.addHandler(handler) + + +def set_log_prefix(new): + global log_prefix + + log_prefix = new diff --git a/_testenv/testenv/cmd.py b/_testenv/testenv/cmd.py new file mode 100644 index 0000000..93e4e36 --- /dev/null +++ b/_testenv/testenv/cmd.py @@ -0,0 +1,103 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import logging +import os +import os.path +import subprocess +import testenv +import testenv.testsuite + +env_extra = {} +usr_dir = None + + +def init_env(): + global env_extra + global usr_dir + + if testenv.args.podman: + if not testenv.args.binary_repo: + usr_dir = os.path.join(testenv.args.cache, "podman", "usr") + else: + usr_dir = os.path.join(testenv.args.cache, "host", "usr") + + if usr_dir: + pkg_config_path = os.path.join(usr_dir, "lib/pkgconfig") + if "PKG_CONFIG_PATH" in os.environ: + pkg_config_path += f":{os.environ.get('PKG_CONFIG_PATH')}" + pkg_config_path += ":/usr/lib/pkgconfig" + + ld_library_path = os.path.join(usr_dir, "lib") + if "LD_LIBRARY_PATH" in os.environ: + ld_library_path += f":{os.environ.get('LD_LIBRARY_PATH')}" + ld_library_path += ":/usr/lib" + + env_extra["PKG_CONFIG_PATH"] = pkg_config_path + env_extra["LD_LIBRARY_PATH"] = ld_library_path + + env_extra["CCACHE_DIR"] = testenv.args.ccache + env_extra["TESTENV_CACHE_DIR"] = testenv.args.cache + env_extra["TESTENV_SRC_DIR"] = testenv.src_dir + + env_extra["TERM"] = os.environ.get("TERM", "dumb") + + if testenv.args.binary_repo: + env_extra["TESTENV_GIT_DIR"] = testenv.podman_install.git_dir + else: + if testenv.args.podman: + env_extra["OSMO_DEV_MAKE_DIR"] = os.path.join(testenv.args.cache, "podman", "make") + else: + env_extra["OSMO_DEV_MAKE_DIR"] = os.path.join(testenv.args.cache, "host", "make") + + +def exit_error_cmd(completed, error_msg): + """:param completed: return from run_cmd() below""" + + logging.error(error_msg) + logging.debug(f"Command: {completed.args}") + logging.debug(f"Returncode: {completed.returncode}") + raise RuntimeError("shell command related error, find details right above this python trace") + + +def generate_env(env={}, podman=False): + ret = dict(env_extra) + path = os.path.join(testenv.data_dir, "scripts") + if testenv.testsuite.ttcn3_hacks_dir: + path += f":{os.path.join(testenv.testsuite.ttcn3_hacks_dir, testenv.args.testsuite)}" + + if usr_dir: + path += f":{os.path.join(usr_dir, 'bin')}" + + if podman: + path += ":/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" + else: + path += f":{os.environ.get('PATH')}" + + ret["PATH"] = path + ret["HOME"] = os.environ.get("HOME") + + for var in env: + ret[var] = env[var] + return ret + + +def run(cmd, check=True, env={}, no_podman=False, stdin=subprocess.DEVNULL, *args, **kwargs): + if not no_podman and testenv.podman.is_running(): + return testenv.podman.exec_cmd(cmd, check=check, env=env, *args, **kwargs) + + logging.debug(f"+ {cmd}") + + # Set stdin to /dev/null by default so we can still capture ^C with testenv + p = subprocess.run( + cmd, + env=generate_env(env), + shell=isinstance(cmd, str), + stdin=stdin, + *args, + **kwargs, + ) + + if p.returncode == 0 or not check: + return p + + exit_error_cmd(p, "Command failed unexpectedly") diff --git a/_testenv/testenv/daemons.py b/_testenv/testenv/daemons.py new file mode 100644 index 0000000..792341c --- /dev/null +++ b/_testenv/testenv/daemons.py @@ -0,0 +1,120 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import atexit +import logging +import os +import os.path +import shlex +import subprocess +import testenv +import testenv.testdir +import time + +daemons = {} +run_shell_on_stop = False + + +def init(): + global run_shell_on_stop + + if not testenv.args.podman: + atexit.register(stop) + if testenv.args.shell: + run_shell_on_stop = True + + +def start(cfg): + global daemons + + for section in cfg: + if section in ["testenv", "DEFAULT"]: + continue + + section_data = cfg[section] + if "program" not in section_data: + continue + if section == "testsuite": + # Runs in the foreground with testenv.testsuite.run() + continue + + program = section_data["program"] + logging.info(f"Running {section}") + + cwd = os.path.join(testenv.testdir.testdir, section) + os.makedirs(cwd, exist_ok=True) + + log = os.path.join(testenv.testdir.testdir, section, f"{section}.log") + + if testenv.args.tee: + pipe = f"2>&1 | tee {shlex.quote(log)}" + else: + pipe = f">{shlex.quote(log)} 2>&1" + cmd = ["sh", "-c", f"{program} 2>&1 {pipe}"] + + env = {} + if testenv.args.io_uring: + env["LIBOSMO_IO_BACKEND"] = "IO_URING" + + if testenv.podman.is_running(): + daemons[section] = testenv.podman.exec_cmd_background(cmd, cwd=cwd, env=env) + else: + logging.debug(f"+ {cmd}") + daemons[section] = subprocess.Popen(cmd, cwd=cwd, env=testenv.cmd.generate_env(env)) + + # Wait 200ms and check if it is still running + time.sleep(0.2) + if daemons[section].poll() is not None: + raise testenv.NoTraceException(f"program failed to start: {program}") + + # Run setup script + if "setup" in section_data: + setup = section_data["setup"] + logging.info(f"Running {section} setup script") + testenv.cmd.run(setup, cwd=cwd) + + +def kill_process_tree(pid, ppids): + subprocess.run( + ["kill", "-9", str(pid)], + check=False, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + + for child_pid, child_ppid in ppids: + if child_ppid == str(pid): + kill_process_tree(child_pid, ppids) + + +def kill(pid): + cmd = ["ps", "-e", "-o", "pid,ppid"] + ret = subprocess.run(cmd, check=True, stdout=subprocess.PIPE) + ppids = [] + proc_entries = ret.stdout.decode("utf-8").rstrip().split("\n")[1:] + for row in proc_entries: + items = row.split() + if len(items) != 2: + raise RuntimeError("Unexpected ps output: " + row) + ppids.append(items) + + kill_process_tree(pid, ppids) + + +def stop(): + global daemons + global run_shell_on_stop + + if run_shell_on_stop: + logging.info("Running interactive shell before stopping daemons (--shell)") + + # stdin=None: override stdin=/dev/null, so we can type into the shell + testenv.cmd.run(["bash"], cwd=testenv.testdir.testdir, stdin=None, check=False) + + run_shell_on_stop = False + + for daemon in daemons: + pid = daemons[daemon].pid + logging.info(f"Stopping {daemon} ({pid})") + kill(pid) + + daemons = {} diff --git a/_testenv/testenv/osmo_dev.py b/_testenv/testenv/osmo_dev.py new file mode 100644 index 0000000..00782cd --- /dev/null +++ b/_testenv/testenv/osmo_dev.py @@ -0,0 +1,135 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import logging +import os +import sys +import testenv +import testenv.cmd + +make_dir = None +init_done = False + + +def get_osmo_dev_dir(): + # Users may have used osmo-dev to clone osmo-ttcn3-hacks: + # osmo-dev/src/osmo-ttcn3-hacks + alt_path = os.path.realpath(os.path.join(testenv.src_dir, "../")) + if os.path.exists(os.path.join(alt_path, "gen_makefile.py")): + return alt_path + + # Assume osmo-dev is next to osmo-ttcn3-hacks: + # src_dir + # ├── osmo-dev + # └── osmo-ttcn3-hacks + return os.path.join(testenv.src_dir, "osmo-dev") + + +def init_clone(): + osmo_dev_dir = get_osmo_dev_dir() + + if os.path.exists(osmo_dev_dir): + logging.debug(f"osmo-dev found, nothing to do: {osmo_dev_dir}") + return + + testenv.cmd.run(["git", "clone", "https://gerrit.osmocom.org/osmo-dev"], cwd=testenv.src_dir) + + +def check_init_needed(): + osmo_dev_dir = get_osmo_dev_dir() + + if os.path.exists(osmo_dev_dir): + logging.debug(f"osmo-dev dir: {osmo_dev_dir}") + return + + logging.error("Missing osmo-dev for building test components from source.") + logging.error("Run 'testenv.py init osmo-dev' first.") + logging.error("") + logging.error("osmo-dev and other Osmocom repositories (if they don't already exist) will be cloned to:") + logging.error(testenv.src_dir) + logging.error("") + logging.error("Set the environment variable TESTENV_SRC_DIR to use a different path.") + sys.exit(1) + + +def init(): + global init_done + global make_dir + + if init_done: + return + + extra_opts = [] + if testenv.args.podman: + make_dir = os.path.join(testenv.args.cache, "podman", "make") + extra_opts = [ + "--install-prefix", + os.path.join(testenv.args.cache, "podman/usr"), + ] + else: + make_dir = os.path.join(testenv.args.cache, "host", "make") + extra_opts = [ + "--install-prefix", + os.path.join(testenv.args.cache, "host/usr"), + ] + + cmd = [ + "./gen_makefile.py", + "--build-debug", + "--make-dir", + make_dir, + "--no-ldconfig", + "--src-dir", + testenv.src_dir, + "default.opts", + "ccache.opts", + "iu.opts", + "no_dahdi.opts", + "no_doxygen.opts", + "no_systemd.opts", + "werror.opts", + os.path.join(testenv.data_dir, "osmo-dev/osmo-bts-trx.opts"), + ] + extra_opts + + testenv.cmd.run(cmd, cwd=get_osmo_dev_dir()) + init_done = True + + +def make(cfg, limit_section=None): + targets = [] + + for section in cfg: + section_data = cfg[section] + if section == "testsuite": + # Gets built with testenv.testsuite.build() + continue + if limit_section and limit_section != section: + # When called from testenv.podman.install_packages as fallback to + # not having a package available, then we only want to run make + # for the target of one specific config section + continue + + if "make" in section_data and section_data["make"] != "no" and section_data["make"] not in targets: + targets += [section_data["make"]] + + if not targets: + logging.debug("No osmo-dev make targets found in testenv.cfg") + return + + makefile_path = os.path.join(make_dir, "Makefile") + with open(makefile_path) as f: + makefile = f.read() + + for target in targets: + if f"\n{target}:" in makefile: + continue + logging.error(f"Could not find make target: {target}") + logging.error("Add it to osmo-dev by adjusting:") + logging.error("* all.deps") + logging.error("* all.buildsystems (if buildsystem != autotools)") + logging.error("* all.urls (if the project is not on gerrit.osmocom.org)") + logging.error("Location of your osmo-dev.git clone:") + logging.error(os.path.join(testenv.src_dir, "osmo-dev")) + sys.exit(1) + + logging.info("Building test components") + testenv.cmd.run(["make"] + targets, cwd=make_dir) diff --git a/_testenv/testenv/podman.py b/_testenv/testenv/podman.py new file mode 100644 index 0000000..bda6f9d --- /dev/null +++ b/_testenv/testenv/podman.py @@ -0,0 +1,287 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import atexit +import datetime +import json +import logging +import multiprocessing +import os +import shlex +import subprocess +import testenv.cmd +import testenv.testdir +import time + +image_name = None +distro = None +container_name = None # instance of image +apt_dir_var_cache = None +apt_dir_var_lib = None +feed_watchdog_process = None +run_shell_on_stop = False + + +def image_exists(): + return testenv.cmd.run(["podman", "image", "exists", image_name], check=False).returncode == 0 + + +def image_up_to_date(): + history = testenv.cmd.run( + ["podman", "history", image_name, "--format", "json"], + capture_output=True, + text=True, + ) + created = json.loads(history.stdout)[0]["created"].split(".", 1)[0] + created = datetime.datetime.strptime(created, "%Y-%m-%dT%H:%M:%S") + logging.debug(f"Image creation date: {created}") + + # On a local development system, we can say that the podman image is + # outdated if the Dockerfile is newer than the creation date. But this does + # not work for jenkins where Dockerfile may just be from a new git + # checkout. So allow to skip this check. + if os.environ.get("TESTENV_NO_IMAGE_UP_TO_DATE_CHECK"): + logging.debug("Assuming the podman image is up-to-date") + return True + + dockerfile = os.path.join(testenv.data_dir, "podman/Dockerfile") + mtime = os.stat(dockerfile).st_mtime + mtime = datetime.datetime.utcfromtimestamp(mtime) + logging.debug(f"Dockerfile last modified: {str(mtime).split('.')[0]}") + + return mtime < created + + +def image_build(): + if image_exists() and image_up_to_date(): + logging.debug(f"Podman image is up-to-date: {image_name}") + if testenv.args.force: + logging.debug("Building anyway since --force was used") + else: + return + + logging.info(f"Building podman image: {image_name}") + testenv.cmd.run( + [ + "buildah", + "build", + "--build-arg", + f"DISTRO={distro}", + "-t", + image_name, + os.path.join(testenv.data_dir, "podman"), + ] + ) + + +def generate_env_podman(env={}): + ret = [] + + for key, val in testenv.cmd.generate_env(env, True).items(): + ret += ["-e", f"{key}={val}"] + + return ret + + +def init_image_name_distro(): + global image_name + global distro + + distro = getattr(testenv.args, "distro", testenv.distro_default) + image_name = f"{distro}-osmo-ttcn3-testenv" + image_name = image_name.replace(":", "-").replace("_", "-") + + +def init(): + global apt_dir_var_cache + global apt_dir_var_lib + global run_shell_on_stop + + apt_dir_var_cache = os.path.join(testenv.args.cache, "podman", "var-cache-apt") + apt_dir_var_lib = os.path.join(testenv.args.cache, "podman", "var-lib-apt") + + os.makedirs(apt_dir_var_cache, exist_ok=True) + os.makedirs(apt_dir_var_lib, exist_ok=True) + os.makedirs(testenv.args.ccache, exist_ok=True) + + init_image_name_distro() + + if not image_exists(): + raise testenv.NoTraceException("Missing podman image, run 'testenv.py init podman' first to build it") + if not image_up_to_date(): + logging.warning("The podman image might be outdated, consider running 'testenv.py init podman' to rebuild it") + + atexit.register(stop) + + if testenv.args.shell: + run_shell_on_stop = True + + +def exec_cmd(cmd, podman_opts=[], cwd=None, env={}, *args, **kwargs): + podman_opts = list(podman_opts) + podman_opts += generate_env_podman(env) + # Attach a fake tty (eclipse-titan won't print colored output otherwise) + podman_opts += ["-t"] + + if cwd: + podman_opts += ["-w", cwd] + + if isinstance(cmd, str): + cmd = ["sh", "-c", cmd] + + testenv.cmd.run( + ["podman", "exec"] + podman_opts + [container_name] + cmd, + no_podman=True, + *args, + **kwargs, + ) + + +def exec_cmd_background(cmd, podman_opts=[], cwd=None, env={}): + podman_opts = list(podman_opts) + generate_env_podman(env) + + if cwd: + podman_opts += ["-w", cwd] + + if isinstance(cmd, str): + cmd = ["sh", "-c", cmd] + + cmd = ["podman", "exec"] + podman_opts + [container_name] + cmd + logging.debug(f"+ {cmd}") + + return subprocess.Popen(cmd) + + +def feed_watchdog_loop(): + # The script testenv-podman-main.sh checks every 10s for /tmp/watchdog and + # deletes the file. Create it here every 5s so the container keeps running. + # This ensures that if we run in jenkins and the job gets aborted, the + # container will terminate after a few seconds. + try: + while True: + time.sleep(5) + p = subprocess.run(["podman", "exec", container_name, "touch", "/tmp/watchdog"]) + if p.returncode: + logging.error("podman container crashed!") + return + except KeyboardInterrupt: + pass + + +def start(): + global container_name + global feed_watchdog_process + + testdir_topdir = testenv.testdir.testdir_topdir + osmo_dev_dir = testenv.osmo_dev.get_osmo_dev_dir() + container_name = testenv.testdir.prefix + # Custom seccomp profile that allows io_uring + seccomp = os.path.join(testenv.data_dir, "podman/seccomp.json") + + cmd = [ + "podman", + "run", + "--rm", + "--name", + container_name, + "--detach", + f"--security-opt=seccomp={seccomp}", + "--cap-add=NET_ADMIN", # for dumpcap, tun devices, osmo-pcap-client + "--cap-add=NET_RAW", # for dumpcap, osmo-pcap-client + "--device=/dev/net/tun", # for e.g. ggsn_tests + "--volume", + f"{apt_dir_var_cache}:/var/cache/apt", + "--volume", + f"{apt_dir_var_lib}:/var/lib/apt", + ] + + if not testenv.args.binary_repo: + cmd += [ + "--volume", + f"{osmo_dev_dir}:{osmo_dev_dir}", + ] + + cmd += [ + "--volume", + f"{testdir_topdir}:{testdir_topdir}", + "--volume", + f"{testenv.args.cache}:{testenv.args.cache}", + "--volume", + f"{testenv.args.ccache}:{testenv.args.ccache}", + "--volume", + f"{testenv.src_dir}:{testenv.src_dir}", + image_name, + os.path.join(testenv.data_dir, "scripts/testenv-podman-main.sh"), + ] + + testenv.cmd.run(cmd, no_podman=True) + + feed_watchdog_process = multiprocessing.Process(target=feed_watchdog_loop) + feed_watchdog_process.start() + + exec_cmd(["rm", "/etc/apt/apt.conf.d/docker-clean"]) + + pkgcache = os.path.join(apt_dir_var_cache, "pkgcache.bin") + if not os.path.exists(pkgcache): + exec_cmd(["apt-get", "-q", "update"]) + + +def distro_to_repo_dir(distro): + if distro == "debian:bookworm": + return "Debian_12" + raise RuntimeError(f"Can't translate distro {distro} to repo_dir!") + + +def enable_binary_repo(): + config = "deb [signed-by=/obs.key]" + config += " https://downloads.osmocom.org/packages/" + config += testenv.args.binary_repo.replace(":", ":/") + config += "/" + config += distro_to_repo_dir(distro) + config += "/ ./" + + path = "/etc/apt/sources.list.d/osmocom.list" + + exec_cmd(["sh", "-c", f"echo {shlex.quote(config)} > {path}"]) + exec_cmd(["apt-get", "-q", "update"]) + + +def is_running(): + if container_name is None: + return False + + cmd = ["podman", "ps", "-q", "--filter", f"name={container_name}"] + if not subprocess.run(cmd, capture_output=True, text=True).stdout: + return False + + return True + + +def stop(restart=False): + global container_name + global run_shell_on_stop + + if not is_running(): + return + + if not restart and run_shell_on_stop: + logging.info("Running interactive shell before stopping container (--shell)") + + # stdin=None: override stdin=/dev/null, so we can type into the shell + exec_cmd(["bash"], ["-i"], cwd=testenv.testdir.testdir, stdin=None, check=False) + + run_shell_on_stop = False + + restart_msg = " (restart)" if restart else "" + logging.info(f"Stopping podman container{restart_msg}") + testenv.cmd.run(["podman", "kill", container_name], no_podman=True) + + if feed_watchdog_process: + feed_watchdog_process.terminate() + + if restart: + testenv.cmd.run(["podman", "wait", container_name], no_podman=True, check=False) + + container_name = None + + if restart: + start() diff --git a/_testenv/testenv/podman_install.py b/_testenv/testenv/podman_install.py new file mode 100644 index 0000000..50ebea8 --- /dev/null +++ b/_testenv/testenv/podman_install.py @@ -0,0 +1,153 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import logging +import multiprocessing +import os +import sys +import testenv.cmd +import testenv.podman + +git_dir = None +bb_dir = None +trxcon_dir = None +sccp_dir = None +jobs = None + + +def init(): + global git_dir + global bb_dir + global trxcon_dir + global sccp_dir + global jobs + + git_dir = os.path.join(testenv.args.cache, "git") + bb_dir = os.path.join(git_dir, "osmocom-bb") + trxcon_dir = os.path.join(bb_dir, "src/host/trxcon") + sccp_dir = os.path.join(git_dir, "libosmo-sccp") + jobs = multiprocessing.cpu_count() + 1 + + os.makedirs(git_dir, exist_ok=True) + + +def apt_install(pkgs): + if not pkgs: + return + + # Remove duplicates + pkgs = list(set(pkgs)) + + logging.info(f"Installing packages: {', '.join(pkgs)}") + testenv.cmd.run(["apt-get", "-q", "install", "-y", "--no-install-recommends"] + pkgs) + + +def clone_osmocom_bb(): + if os.path.exists(bb_dir): + logging.debug("osmocom-bb: already cloned") + return + + testenv.cmd.run( + [ + "git", + "-C", + git_dir, + "clone", + "--depth", + "1", + "https://gerrit.osmocom.org/osmocom-bb", + ] + ) + + +def clone_libosmo_sccp(): + if os.path.exists(sccp_dir): + logging.debug("libosmo-sccp: already cloned") + return + + testenv.cmd.run( + [ + "git", + "-C", + git_dir, + "clone", + "--depth", + "1", + "https://gerrit.osmocom.org/libosmo-sccp", + ] + ) + + +def from_source_trxcon(): + trxcon_in_srcdir = os.path.join(trxcon_dir, "src/trxcon") + + if not os.path.exists(trxcon_in_srcdir): + clone_osmocom_bb() + apt_install(["libosmocore-dev"]) + logging.info("Building trxcon") + testenv.cmd.run(["autoreconf", "-fi"], cwd=trxcon_dir) + testenv.cmd.run(["./configure"], cwd=trxcon_dir) + testenv.cmd.run(["make", "-j", f"{jobs}"], cwd=trxcon_dir) + + testenv.cmd.run(["ln", "-s", trxcon_in_srcdir, "/usr/local/bin/trxcon"]) + + +def from_source_sccp_demo_user(): + sccp_demo_user_path = os.path.join(sccp_dir, "examples/sccp_demo_user") + + # Install libraries even if not building sccp_demo_user, because it gets + # linked dynamically against them. + apt_install( + [ + "libosmo-netif-dev", + "libosmocore-dev", + ] + ) + + if not os.path.exists(sccp_demo_user_path): + clone_libosmo_sccp() + logging.info("Building sccp_demo_user") + testenv.cmd.run(["autoreconf", "-fi"], cwd=sccp_dir) + testenv.cmd.run(["./configure"], cwd=sccp_dir) + testenv.cmd.run( + ["make", "-j", f"{jobs}", "libosmo-sigtran.la"], + cwd=os.path.join(sccp_dir, "src"), + ) + testenv.cmd.run( + ["make", "-j", f"{jobs}", "sccp_demo_user"], + cwd=os.path.join(sccp_dir, "examples"), + ) + + testenv.cmd.run(["ln", "-s", sccp_demo_user_path, "/usr/local/bin/sccp_demo_user"]) + + +def from_source(cfg, cfg_name, section): + program = cfg[section]["program"] + if program == "trxcon": + return from_source_trxcon() + if program == "run_fake_trx.sh": + return clone_osmocom_bb() + if program == "run_sccp_demo_user.sh": + return from_source_sccp_demo_user() + + logging.error(f"Can't install {section}! Fix this by either:") + logging.error(f"* Adding package= to [{section}] in {cfg_name}") + logging.error(" (if it can be installed from binary packages)") + logging.error("* Editing from_source() in testenv/podman_install.py") + sys.exit(1) + + +def packages(cfg, cfg_name): + packages = [] + + for section in cfg: + if section in ["DEFAULT", "testsuite"]: + continue + section_data = cfg[section] + if "package" in section_data: + if section_data["package"] == "no": + continue + packages += section_data["package"].split(" ") + else: + from_source(cfg, cfg_name, section) + + apt_install(packages) diff --git a/_testenv/testenv/requirements.py b/_testenv/testenv/requirements.py new file mode 100644 index 0000000..7badbe9 --- /dev/null +++ b/_testenv/testenv/requirements.py @@ -0,0 +1,83 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import logging +import os.path +import shutil +import sys +import testenv +import testenv.cmd +import testenv.testsuite + + +def check_programs(): + programs = [ + "git", + ] + + if testenv.args.podman: + programs += [ + "buildah", + "podman", + "rsync", + ] + else: + programs += [ + "autoconf", + "automake", + "ccache", + "dumpcap", + "g++", + "gcc", + "make", + "pkg-config", + "setcap", + "ttcn3_compiler", + "wget", + ] + + abort = False + for program in programs: + if not shutil.which(program): + logging.error(f"Missing program: {program}") + + if program == "ttcn3_compiler": + logging.error(" Install eclipse-titan, e.g. from osmocom:latest:") + logging.error(" https://osmocom.org/projects/cellular-infrastructure/wiki/Latest_Builds") + abort = True + + if abort: + sys.exit(1) + + +def check_fftranscode(): + cmd = [ + "grep", + "-q", + "fftranscode", + os.path.join( + testenv.testsuite.ttcn3_hacks_dir_src, + testenv.args.testsuite, + "regen_makefile.sh", + ), + ] + if testenv.cmd.run(cmd, check=False).returncode == 1: + return + + cmd = ["pkg-config", "--modversion", "libfftranscode"] + if testenv.cmd.run(cmd, check=False).returncode == 0: + return + + logging.error("Missing library: libfftranscode") + logging.error( + " https://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Tests…" + ) + logging.error(" Consider installing it from here:") + logging.error(" https://ftp.osmocom.org/binaries/libfftranscode/") + sys.exit(1) + + +def check(): + check_programs() + + if not testenv.args.podman: + check_fftranscode() diff --git a/_testenv/testenv/testdir.py b/_testenv/testenv/testdir.py new file mode 100644 index 0000000..f7abe2e --- /dev/null +++ b/_testenv/testenv/testdir.py @@ -0,0 +1,177 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import atexit +import datetime +import glob +import logging +import os +import os.path +import tempfile +import testenv +import testenv.cmd +import testenv.testsuite +import uuid + + +# Some testsuites have multiple configurations (like bts: generic, hopping, …). +# For each configuration, prepare() gets called. testdir is the one for the +# current configuration. +testdir = None +testdir_topdir = None +prefix = None +clean_scripts = {} + + +def init(): + global testdir_topdir + global prefix + + prefix = f"testenv-{testenv.args.testsuite}-" + if testenv.args.config: + prefix += f"{'-'.join(testenv.args.config)}-" + if testenv.args.binary_repo: + prefix += f"{testenv.args.binary_repo.replace(':','-')}-" + prefix += datetime.datetime.now().strftime("%Y%m%d-%H%M") + prefix += f"-{str(uuid.uuid4()).split('-', 1)[0]}" + + if testenv.args.log_dir: + testdir_topdir = testenv.args.log_dir + os.makedirs(testdir_topdir) + else: + testdir_topdir = tempfile.mkdtemp(prefix=f"{prefix}-") + + atexit.register(clean) + + logging.info(f"Logging to: {testdir_topdir}") + + # Add a convenience symlink + if not testenv.args.log_dir: + if os.path.exists("/tmp/logs"): + testenv.cmd.run(["rm", "/tmp/logs"], no_podman=True) + testenv.cmd.run(["ln", "-sf", testdir_topdir, "/tmp/logs"], no_podman=True) + + +def prepare(cfg_name, cfg): + global testdir + global clean_scripts + + if len(testenv.testenv_cfg.cfgs) == 1: + testdir = testdir_topdir + else: + testdir = os.path.join(testdir_topdir, cfg_name.replace("testenv_", "").replace(".cfg", "")) + + logging.info(f"Preparing testdir: {testdir}") + testsuite_dir = os.path.join(testenv.testsuite.ttcn3_hacks_dir, testenv.args.testsuite) + + atexit.register(clean_run_scripts) + + for section in cfg: + if section in ["DEFAULT"]: + continue + + section_data = cfg[section] + section_dir = os.path.join(testdir, section) + os.makedirs(section_dir) + + if "config" in section_data and section == "testsuite": + file = section_data["config"] + path = os.path.join(testsuite_dir, file) + path_dest = os.path.join(section_dir, file) + testenv.cmd.run(["install", "-Dm644", path, path_dest]) + + if "copy" in section_data: + for file in section_data["copy"].split(" "): + path = os.path.join(testsuite_dir, file) + path_dest = os.path.join(section_dir, file) + mode = 755 if os.access(path, os.X_OK) else 644 + testenv.cmd.run(["install", f"-Dm{mode}", path, path_dest]) + + if "clean" in section_data: + logging.info(f"Running {section} clean script (reason: prepare)") + clean_scripts[section] = section_data["clean"] + env = {"TESTENV_CLEAN_REASON": "prepare"} + testenv.cmd.run(section_data["clean"], cwd=section_dir, env=env) + + if "prepare" in section_data: + logging.info(f"Running {section} prepare script") + testenv.cmd.run(section_data["prepare"], cwd=section_dir) + + # Referenced in testsuite cfgs: *.default + pattern = os.path.join(testsuite_dir, "*.default") + for path in glob.glob(pattern): + path_dest = os.path.join(testdir, "testsuite", os.path.basename(path)) + testenv.cmd.run(["install", "-Dm644", path, path_dest]) + + # Referenced in testsuite cfgs: Common.cfg + common_cfg = os.path.join(testdir, "testsuite", "Common.cfg") + path = os.path.join(testenv.testsuite.ttcn3_hacks_dir, "Common.cfg") + testenv.cmd.run(["install", "-Dm644", path, common_cfg]) + testenv.cmd.run( + [ + "sed", + "-i", + f's#TTCN3_HACKS_PATH := .*#TTCN3_HACKS_PATH := "{testenv.testsuite.ttcn3_hacks_dir}"#', + common_cfg, + ] + ) + + # Adjust testsuite config: set mp_osmo_repo, set Common.cfg path in the + # testsuite's config and in all configs it may include + mp_osmo_repo = "latest" if testenv.args.latest else "nightly" + line = f'Misc_Helpers.mp_osmo_repo := "{mp_osmo_repo}"' + + patterns = [ + os.path.join(testdir, "testsuite/**/*.cfg"), + os.path.join(testdir, "testsuite/**/*.default"), + ] + for pattern in patterns: + for cfg_file in glob.glob(pattern, recursive=True): + logging.debug(f"Adjusting testsuite config: {cfg_file}") + testenv.cmd.run( + [ + "sed", + "-i", + "-e", + f"s/\\[MODULE_PARAMETERS\\]/\\[MODULE_PARAMETERS\\]\\n{line}/g", + "-e", + "s#../Common.cfg#Common.cfg#", + cfg_file, + ] + ) + + +def clean(): + """Don't leave behind an empty testdir_topdir, e.g. if testenv.py aborted + during build of components.""" + # Show log dir path/link if it isn't empty + if os.listdir(testdir_topdir): + msg = "Logs saved to:" + + url = os.environ.get("BUILD_URL") # Jenkins sets this + if url: + # Add a space at the end, so jenkins can transform this into a link + # without adding the color reset escape code to it + msg += f" {url}artifact/logs/ " + else: + msg += f" {testdir_topdir}" + if not testenv.args.log_dir: + msg += " (symlink: /tmp/logs)" + logging.info(msg) + return + + logging.debug("Remving empty log dir") + testenv.cmd.run(["rm", "-d", testdir_topdir], no_podman=True) + + # Remove broken symlink + if not testenv.args.log_dir and os.path.lexists("/tmp/logs") and not os.path.exists("/tmp/logs"): + testenv.cmd.run(["rm", "/tmp/logs"], no_podman=True) + + +def clean_run_scripts(reason="crashed"): + global clean_scripts + + for section, script in clean_scripts.items(): + logging.info(f"Running {section} clean script (reason: {reason})") + env = {"TESTENV_CLEAN_REASON": reason} + testenv.cmd.run(script, cwd=os.path.join(testdir, section), env=env) + clean_scripts = {} diff --git a/_testenv/testenv/testenv_cfg.py b/_testenv/testenv/testenv_cfg.py new file mode 100644 index 0000000..64f2423 --- /dev/null +++ b/_testenv/testenv/testenv_cfg.py @@ -0,0 +1,189 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import configparser +import glob +import logging +import os.path +import sys +import testenv +import testenv.testsuite + +cfgs = {} +current = None + + +def set_current(cfg_name): + global current + current = cfg_name + + if cfg_name == "testenv.cfg": + testenv.set_log_prefix("[testenv]") + else: + cfg_name = cfg_name.replace("testenv_", "") + cfg_name = cfg_name.replace(".cfg", "") + testenv.set_log_prefix(f"[testenv][{cfg_name}]") + + +def exit_error_readme(): + readme = os.path.join(testenv.testsuite.ttcn3_hacks_dir_src, "_testenv/README.md") + logging.error(f"More information: {readme}") + sys.exit(1) + + +def handle_latest(cfg, path): + """Remove _latest keys from cfg or use them instead of the regular keys, + if --latest is set.""" + + for section in cfg: + for key in cfg[section]: + if not key.endswith("_latest"): + continue + + if testenv.args.latest: + key_regular = key.replace("_latest", "") + logging.debug(f"{path}: [{section}]: using {key} instead of {key_regular} (--latest is set)") + cfg[section][key_regular] = cfg[section][key] + else: + logging.debug(f"{path}: [{section}]: ignoring {key} (--latest is not set)") + + del cfg[section][key] + + +def verify(cfg, path): + keys_valid_testsuite = [ + "clean", + "config", + "copy", + "program", + ] + keys_valid_component = [ + "clean", + "copy", + "make", + "package", + "prepare", + "program", + "setup", + ] + keys_invalid = { + "configs": "config", + "packages": "package", + "programs": "program", + } + + if "testsuite" not in cfg: + logging.error(f"{path}: missing [testsuite] section") + exit_error_readme() + if "program" not in cfg["testsuite"]: + logging.error(f"{path}: missing program= in [testsuite]") + exit_error_readme() + if " " in cfg["testsuite"]["program"]: + logging.error(f"{path}: program= in [testsuite] must not have arguments") + exit_error_readme() + if " " in cfg["testsuite"]["config"]: + logging.error(f"{path}: config= in [testsuite] must not have spaces") + exit_error_readme() + if "config" not in cfg["testsuite"]: + logging.error(f"{path}: missing config= in [testsuite]") + exit_error_readme() + + for section in cfg: + for key in cfg[section].keys(): + valid = keys_valid_component + if section == "testsuite": + valid = keys_valid_testsuite + + if key in valid: + continue + + msg = f"{path}: [{section}]: {key}= is invalid" + if key in keys_invalid and keys_invalid[key] in valid: + msg += f", did you mean {keys_invalid[key]}=?" + + logging.error(msg) + exit_error_readme() + + if section not in ["DEFAULT", "testsuite"] and "make" not in cfg[section]: + logging.error(f"{path}: missing make= in section [{section}].") + logging.error("If this is on purpose, set make=no.") + exit_error_readme() + + +def raise_error_config_arg(glob_result): + valid = [] + for path in glob_result: + basename = os.path.basename(path) + if basename != "testenv.cfg": + valid += [basename.split("_", 1)[1].split(".", -1)[0]] + + msg = f"Invalid parameter for --config: {testenv.args.config}" + + if valid: + msg += f" (valid: all, {', '.join(valid)})" + else: + msg += f" (the {testenv.args.testsuite} testsuite only has one testenv.cfg file, therefore just omit --config)" + + raise testenv.NoTraceException(msg) + + +def find_configs(): + dir_testsuite = os.path.join(testenv.testsuite.ttcn3_hacks_dir_src, testenv.args.testsuite) + pattern = os.path.join(dir_testsuite, "testenv*.cfg") + ret = glob.glob(pattern) + + if not ret: + logging.error(f"Missing testenv.cfg in: {dir_testsuite}") + exit_error_readme() + sys.exit(1) + + if len(ret) > 1 and os.path.exists(os.path.join(dir_testsuite, "testenv.cfg")): + logging.error("Found multiple testenv*.cfg, and a testenv.cfg.") + logging.error("The testenv.cfg file must be renamed, consider naming it testenv_generic.cfg.") + sys.exit(1) + + if len(ret) == 1 and not os.path.exists(os.path.join(dir_testsuite, "testenv.cfg")): + logging.error("There is only one testenv*.cfg file, so please rename it:") + logging.error(f"$ mv {os.path.basename(ret[0])} testenv.cfg") + sys.exit(1) + + if len(ret) > 1 and not testenv.args.config: + logging.error("Found multiple testenv.cfg files:") + for path in ret: + logging.error(f" * {os.path.basename(path)}") + example = os.path.basename(ret[0]).split("_", 1)[1].split(".cfg", 1)[0] + logging.error(f"Select a specific config (e.g. '-c {example}') or all ('-c all')") + sys.exit(1) + + return ret + + +def init(): + global cfgs + + config_paths = find_configs() + + for path in config_paths: + basename = os.path.basename(path) + if basename != "testenv.cfg" and not basename.startswith("testenv_"): + raise testenv.NoTraceException( + f"Invalid filename, expected either testenv.cfg or testenv_*.cfg: {basename}" + ) + if basename == "testenv_all.cfg": + raise testenv.NoTraceException(f"Invalid filename: {basename}") + + cfg = configparser.ConfigParser() + cfg.read(path) + handle_latest(cfg, path) + verify(cfg, path) + + if not testenv.args.config: + cfgs[basename] = cfg + continue + + for config_arg in testenv.args.config: + if config_arg == "all" or f"testenv_{config_arg}.cfg" == basename: + cfgs[basename] = cfg + break + + if not cfgs: + raise_error_config_arg(config_paths) diff --git a/_testenv/testenv/testsuite.py b/_testenv/testenv/testsuite.py new file mode 100644 index 0000000..88ea467 --- /dev/null +++ b/_testenv/testenv/testsuite.py @@ -0,0 +1,218 @@ +# Copyright 2024 sysmocom - s.f.m.c. GmbH +# SPDX-License-Identifier: GPL-3.0-or-later +import atexit +import glob +import logging +import multiprocessing +import os +import os.path +import shlex +import shutil +import subprocess +import testenv +import testenv.cmd +import time + +ttcn3_hacks_dir = None +ttcn3_hacks_dir_src = os.path.realpath(f"{__file__}/../../..") +testsuite_proc = None + + +def update_deps(): + deps_marker = os.path.join(testenv.args.cache, "ttcn3-deps-updated") + if os.path.exists(deps_marker): + return + + logging.info("Updating osmo-ttcn3-hacks/deps") + deps_dir = os.path.join(ttcn3_hacks_dir_src, "deps") + testenv.cmd.run(["make", "-C", deps_dir]) + testenv.cmd.run(["touch", deps_marker]) + + +def copy_ttcn3_hacks_dir(): + """Copy source files of osmo-ttcn3-hacks.git to the cache dir, so we don't + mix binary objects from host and inside podman that are very likely to + be incompatible""" + global ttcn3_hacks_dir + + ttcn3_hacks_dir = os.path.join(testenv.args.cache, "podman", "osmo-ttcn3-hacks") + + logging.info(f"Copying osmo-ttcn3-hacks sources to: {ttcn3_hacks_dir}") + + # Rsync can't directly parse the .gitignore with ! rules, so create a list + # of files to be copied with git + copy_list = os.path.join(testenv.args.cache, "podman", "ttcn3-copy-list") + testenv.cmd.run( + f"git ls-files -o -c --exclude-standard > {shlex.quote(copy_list)}", + cwd=ttcn3_hacks_dir_src, + no_podman=True, + ) + + # Copy source files, excluding binary objects + testenv.cmd.run( + [ + "rsync", + "--links", + "--recursive", + f"--files-from={copy_list}", + f"{ttcn3_hacks_dir_src}/", + f"{ttcn3_hacks_dir}/", + ], + no_podman=True, + ) + + # The "deps" dir is in gitignore, copy it separately + testenv.cmd.run( + [ + "rsync", + "--links", + "--recursive", + "--exclude", + "/.git", + f"{ttcn3_hacks_dir_src}/deps/", + f"{ttcn3_hacks_dir}/deps/", + ], + no_podman=True, + ) + + +def prepare_testsuite_dir(): + testsuite_dir = f"{ttcn3_hacks_dir}/{testenv.args.testsuite}" + logging.info(f"Generating links and Makefile for {testenv.args.testsuite}") + testenv.cmd.run(["./gen_links.sh"], cwd=testsuite_dir) + testenv.cmd.run("USE_CCACHE=1 ./regen_makefile.sh", cwd=testsuite_dir) + + +def init(): + global ttcn3_hacks_dir + + atexit.register(stop) + + update_deps() + + if testenv.args.podman: + copy_ttcn3_hacks_dir() + else: + ttcn3_hacks_dir = ttcn3_hacks_dir_src + + prepare_testsuite_dir() + + +def build(): + logging.info("Building testsuite") + testsuite_dir = f"{ttcn3_hacks_dir}/{testenv.args.testsuite}" + testenv.cmd.run(["make", "compile"], cwd=testsuite_dir) + + jobs = multiprocessing.cpu_count() + 1 + testenv.cmd.run(["make", "-j", f"{jobs}"], cwd=testsuite_dir) + + +def is_running(pid): + # Check if a process is still running, or if it is dead / a zombie. We + # can't just use proc.poll() because this gets called from another thread. + cmdline = f"/proc/{pid}/cmdline" + if not os.path.exists(cmdline): + return False + + # The cmdline file is empty if it is a zombie + with open(cmdline) as f: + return f.read() != "" + + +def merge_log_files(cfg): + section_data = cfg["testsuite"] + cwd = os.path.join(testenv.testdir.testdir, "testsuite") + + logging.info("Merging log files") + log_merge = os.path.join(ttcn3_hacks_dir, "log_merge.sh") + # stdout of this script is very verbose, making it harder to see the output + # that matters (tests failed or not), so redirect it to /dev/null + cmd = f"{shlex.quote(log_merge)} {shlex.quote(section_data['program'])} --rm >/dev/null" + testenv.cmd.run(cmd, cwd=cwd) + + +def format_log_files(cfg): + cwd = os.path.join(testenv.testdir.testdir, "testsuite") + + logging.info("Formatting log files") + cmd = os.path.join(testenv.data_dir, "scripts/log_format.sh") + testenv.cmd.run(cmd, cwd=cwd) + + +def cat_junit_logs(): + tool = "cat" + + if testenv.args.podman or shutil.which("source-highlight"): + colors = os.environ.get("TESTENV_SOURCE_HIGHLIGHT_COLORS", "esc256") + tool = f"source-highlight -f {shlex.quote(colors)} -s xml -i" + + pattern = os.path.join(testenv.testdir.testdir, "testsuite", "junit-*.log") + for path in glob.glob(pattern): + cmd = f"echo && {tool} {shlex.quote(path)} && echo" + logging.info(f"Showing {os.path.basename(path)}") + testenv.cmd.run(cmd) + + +def run(cfg): + global testsuite_proc + + section_data = cfg["testsuite"] + + cwd = os.path.join(testenv.testdir.testdir, "testsuite") + start_testsuite = os.path.join(ttcn3_hacks_dir, "start-testsuite.sh") + suite = os.path.join(ttcn3_hacks_dir, testenv.args.testsuite, section_data["program"]) + + env = { + "TTCN3_PCAP_PATH": os.path.join(testenv.testdir.testdir, "testsuite"), + } + + cmd = [start_testsuite, suite, section_data["config"]] + + test_arg = testenv.args.test + if test_arg: + if "." in test_arg: + cmd += [test_arg] + else: + cmd += [f"{section_data['program']}.{test_arg}"] + + logging.info("Running testsuite") + + if testenv.podman.is_running(): + testsuite_proc = testenv.podman.exec_cmd_background(cmd, cwd=cwd, env=env) + else: + logging.debug(f"+ {cmd}") + testsuite_proc = subprocess.Popen(cmd, cwd=cwd, env=env) + + # Ensure all daemons run until the testsuite stops + while True: + time.sleep(1) + + if not is_running(testsuite_proc.pid): + logging.debug("Testsuite is done") + stop() + break + + for daemon_name, daemon_proc in testenv.daemons.daemons.items(): + if not is_running(daemon_proc.pid): + raise testenv.NoTraceException(f"{daemon_name} crashed!") + + merge_log_files(cfg) + format_log_files(cfg) + + +def run_prepare_script(cfg): + section_data = cfg["testsuite"] + if "prepare" not in section_data: + return + + logging.info("Running testsuite prepare script") + testenv.cmd.run(section_data["prepare"]) + + +def stop(): + global testsuite_proc + + if testsuite_proc: + logging.info(f"Stopping testsuite ({testsuite_proc.pid})") + testenv.daemons.kill(testsuite_proc.pid) + testsuite_proc = None diff --git a/testenv.py b/testenv.py new file mode 120000 index 0000000..36c795e --- /dev/null +++ b/testenv.py @@ -0,0 +1 @@ +_testenv/testenv.py \ No newline at end of file -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37694?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: If9f8b79dd6e5b4f06be4e5ff73db97759c3acfb2 Gerrit-Change-Number: 37694 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: daniel <dwillmann(a)sysmocom.de> Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

11 months

1
0
0 0

[S] Change in osmo-ttcn3-hacks[master]: hlr: initial testenv.cfg

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37695?usp=email ) Change subject: hlr: initial testenv.cfg ...................................................................... hlr: initial testenv.cfg Change-Id: If937f021a33bf81b4e7296c2efdbd13a73cdaca9 --- A hlr/testenv.cfg 1 file changed, 18 insertions(+), 0 deletions(-) Approvals: laforge: Looks good to me, but someone else must approve pespin: Looks good to me, approved Jenkins Builder: Verified diff --git a/hlr/testenv.cfg b/hlr/testenv.cfg new file mode 100644 index 0000000..0fe2f86 --- /dev/null +++ b/hlr/testenv.cfg @@ -0,0 +1,9 @@ +[testsuite] +program=HLR_Tests +config=HLR_Tests.cfg + +[hlr] +program=osmo-hlr +make=osmo-hlr +package=osmo-hlr +copy=osmo-hlr.cfg -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37695?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: If937f021a33bf81b4e7296c2efdbd13a73cdaca9 Gerrit-Change-Number: 37695 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

11 months

1
0
0 0

[S] Change in osmo-ttcn3-hacks[master]: mgw: initial testenv.cfg

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37696?usp=email ) Change subject: mgw: initial testenv.cfg ...................................................................... mgw: initial testenv.cfg Change-Id: Ifad2a6e25816c2dd7d27206751597b486acdd096 --- A mgw/testenv.cfg 1 file changed, 18 insertions(+), 0 deletions(-) Approvals: pespin: Looks good to me, approved Jenkins Builder: Verified laforge: Looks good to me, but someone else must approve diff --git a/mgw/testenv.cfg b/mgw/testenv.cfg new file mode 100644 index 0000000..c69c422 --- /dev/null +++ b/mgw/testenv.cfg @@ -0,0 +1,9 @@ +[testsuite] +program=MGCP_Test +config=MGCP_Test.cfg + +[mgw] +program=osmo-mgw +make=osmo-mgw +package=osmo-mgw +copy=osmo-mgw.cfg -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37696?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: Ifad2a6e25816c2dd7d27206751597b486acdd096 Gerrit-Change-Number: 37696 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

11 months

1
0
0 0

[S] Change in osmo-ttcn3-hacks[master]: sip: initial testenv.cfg

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37697?usp=email ) Change subject: sip: initial testenv.cfg ...................................................................... sip: initial testenv.cfg Change-Id: I2b6df9240d8cb56c363c8c1a175946af183497da --- A sip/testenv.cfg 1 file changed, 18 insertions(+), 0 deletions(-) Approvals: pespin: Looks good to me, approved laforge: Looks good to me, but someone else must approve Jenkins Builder: Verified diff --git a/sip/testenv.cfg b/sip/testenv.cfg new file mode 100644 index 0000000..fc3a83e --- /dev/null +++ b/sip/testenv.cfg @@ -0,0 +1,9 @@ +[testsuite] +program=SIP_Tests +config=SIP_Tests.cfg + +[sip-connector] +program=osmo-sip-connector +make=osmo-sip-connector +package=osmo-sip-connector +copy=osmo-sip-connector.cfg -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37697?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: I2b6df9240d8cb56c363c8c1a175946af183497da Gerrit-Change-Number: 37697 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

11 months

1
0
0 0

[XL] Change in osmo-ttcn3-hacks[master]: testenv: add test environment script

by osmith

Attention is currently required from: daniel, fixeria, pespin. osmith has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37694?usp=email ) Change subject: testenv: add test environment script ...................................................................... Patch Set 3: Code-Review+2 (2 comments) Patchset: PS3: 1+1=2 File _testenv/data/podman/Dockerfile: https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37694/comment/f3a1d93c_6425… PS3, Line 78: ENV USE_CCACHE=0 > shouldn't you move this further down until it is used? otherwise all the steps below are rerun if on […] CCACHE is enabled on demand when running commands inside the container by setting the environment variable, not by changing it here in the Dockerfile. It is not useful to have it enabled while building the image since we don't build any of the testsuite or SUT code while building the image. -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37694?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: If9f8b79dd6e5b4f06be4e5ff73db97759c3acfb2 Gerrit-Change-Number: 37694 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: daniel <dwillmann(a)sysmocom.de> Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: pespin <pespin(a)sysmocom.de> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Attention: daniel <dwillmann(a)sysmocom.de> Gerrit-Comment-Date: Fri, 02 Aug 2024 13:17:36 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Comment-In-Reply-To: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: comment

11 months

1
0
0 0

[XL] Change in docker-playground[master]: Remove ttcn3 testsuites ported to testenv, part 1

by pespin

Attention is currently required from: osmith. pespin has posted comments on this change. ( https://gerrit.osmocom.org/c/docker-playground/+/37714?usp=email ) Change subject: Remove ttcn3 testsuites ported to testenv, part 1 ...................................................................... Patch Set 1: (1 comment) Patchset: PS1: Let's wait until the testenv patches are merged and people try them out for a few days before dropping this. -- To view, visit https://gerrit.osmocom.org/c/docker-playground/+/37714?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: docker-playground Gerrit-Branch: master Gerrit-Change-Id: I7182c7e8eb00a9323fe1b6d27f630d253b4d34f6 Gerrit-Change-Number: 37714 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-CC: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Comment-Date: Fri, 02 Aug 2024 13:15:36 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment

11 months

1
0
0 0

[L] Change in osmo-ttcn3-hacks[master]: mme: initial testenv.cfg

by pespin

Attention is currently required from: osmith. pespin has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37709?usp=email ) Change subject: mme: initial testenv.cfg ...................................................................... Patch Set 3: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37709?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: I780584cc7722d92baa213049ee0a35e16a0e087c Gerrit-Change-Number: 37709 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Comment-Date: Fri, 02 Aug 2024 13:14:22 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

11 months

1
0
0 0

[L] Change in osmo-ttcn3-hacks[master]: hss: initial testenv.cfg

by pespin

Attention is currently required from: osmith. pespin has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37708?usp=email ) Change subject: hss: initial testenv.cfg ...................................................................... Patch Set 3: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37708?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: I8a095f51f1a9516967d5188587537a2f90927f6c Gerrit-Change-Number: 37708 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Comment-Date: Fri, 02 Aug 2024 13:14:08 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

11 months

1
0
0 0

[M] Change in osmo-ttcn3-hacks[master]: cbc: initial testenv.cfg

by pespin

Attention is currently required from: osmith. pespin has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37707?usp=email ) Change subject: cbc: initial testenv.cfg ...................................................................... Patch Set 3: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37707?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: I8578bf7a2e627c93cc7480534cc2ddbba3ad6ff1 Gerrit-Change-Number: 37707 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Comment-Date: Fri, 02 Aug 2024 13:13:49 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

11 months

1
0
0 0

[S] Change in osmo-ttcn3-hacks[master]: smlc: initial testenv.cfg

by pespin

Attention is currently required from: osmith. pespin has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37706?usp=email ) Change subject: smlc: initial testenv.cfg ...................................................................... Patch Set 3: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37706?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: Ib6db49f149e9382b802fceb40b307c5a286f92a1 Gerrit-Change-Number: 37706 Gerrit-PatchSet: 3 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Comment-Date: Fri, 02 Aug 2024 13:13:40 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

11 months

1
0
0 0

2025

2024

2023

2022

gerrit-log August 2024