[XL] Change in osmo-ttcn3-hacks[master]: testenv: add test environment script

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37694?usp=email ) Change subject: testenv: add test environment script ...................................................................... testenv: add test environment script Add a new testenv.py script that builds/installs all components needed for a testsuite, builds the testsuite from source and runs it. Features: * --binary-repo argument to install packages from osmocom:latest or any other repository from the Osmocom OBS instead of building from source * without --binary-repo, the test components are built with osmo-dev, cloning the missing source git repositories and building them in the right order * --podman argument to run the testsuite and its components inside a container (using podman instead of docker so it runs rootless) * Simple testenv.cfg file to specify components for running testsuites * Iterative compilation of components and testsuite * Using ccache * Testsuite doesn't start if any of the components fail to start (e.g. because of a config error) * Testsuite gets stopped if any of the components crash * ^C stops the testsuite + all components * Test component output logs to stdout in addition to a log file (turn off with --no-tee) * --test argument to only run one specific test * --shell argument to run an interactive shell before teardown to inspect the test environment while components are still running This script unifies the use cases of running a testsuite without containers (for local development), and with containers (as jenkins runs it, but can also be used for local development e.g. to get a clean pcap). Previously jenkins used a different set of configurations from docker-playground.git and many different containers instead of just one. This patch contains the seccomp_profile.json for enabling io_uring, from docker-playground I27567c2a5d9543c3509c316226c082ab950c5ebc. Related: OS#6494 Change-Id: If9f8b79dd6e5b4f06be4e5ff73db97759c3acfb2 --- M .gitignore M README.md A _testenv/README.md A _testenv/data/osmo-dev/osmo-bts-trx.opts A _testenv/data/podman/Dockerfile A _testenv/data/podman/obs.key A _testenv/data/podman/seccomp_profile.json A _testenv/data/scripts/log_format.sh A _testenv/data/scripts/rename_junit_xml_classname.sh A _testenv/data/scripts/respawn.sh A _testenv/data/scripts/testenv-podman-main.sh A _testenv/testenv.py A _testenv/testenv/__init__.py A _testenv/testenv/cmd.py A _testenv/testenv/daemons.py A _testenv/testenv/osmo_dev.py A _testenv/testenv/podman.py A _testenv/testenv/podman_install.py A _testenv/testenv/requirements.py A _testenv/testenv/testdir.py A _testenv/testenv/testenv_cfg.py A _testenv/testenv/testsuite.py A testenv.py 23 files changed, 2,999 insertions(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/94/37694/1 diff --git a/.gitignore b/.gitignore index e77dfa7..f03b0b3 100644 --- a/.gitignore +++ b/.gitignore @@ -22,3 +22,4 @@ sms.db sms.db-shm sms.db-wal +__pycache__ diff --git a/README.md b/README.md index 907ebb1..c84f31c 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,14 @@ network elements, from 2G, 3G, 4G to 5G. The individual test-suites are in sub-directories, while some shared library code is in *library*. +Running Testsuites +------------------ + +Use the `testenv.py` script to run the testsuites, e.g.: + +``` +$ ./testenv.py run mgw +``` Continuous Integration ---------------------- diff --git a/_testenv/README.md b/_testenv/README.md new file mode 100644 index 0000000..ed3a0a0 --- /dev/null +++ b/_testenv/README.md @@ -0,0 +1,141 @@ +# testenv + +Build everything needed for running Osmocom TTCN-3 testsuites and execute them. + +## testenv.cfg + +The `testenv.cfg` file has one `[testsuite]` section, typically with one or +more sections for test components. + +### Example + +```ini +[testsuite] +program=MGCP_Test +config=MGCP_Test.cfg + +[mgw] +program=osmo-mgw +make=osmo-mgw +package=osmo-mgw +copy=osmo-mgw.cfg +``` + +### Keys + +#### Testsuite section + +* `program=` the executable for starting the testsuite, without arguments. + +* `config=`: the testsuite configuration file. + +* `copy=`: additional file(s) to copy from the testsuite directory to the test + directory, useful for include files mentioned in the config. Multiple values + are separated by spaces. + +* `clean=`: optional script to run before `prepare=` and on exit. This can be + used to clean up network devices for example, or to fix name collisions when + running a test with multiple configs (`rename_junit_xml_classname.sh`). See + below for `PATH`. A `TESTENV_CLEAN_REASON` env var is set to `prepare`, + `crashed` or `finished` depending on when the script runs. + +#### Component section + +* `program=`: executable for starting a test component, may contain arguments. + See below for `PATH`. + +* `copy=`: file(s) to copy from the testsuite directory to the test directory, + like `.cfg` and `.confmerge` files. Multiple values are separated by spaces. + +* `make=`: osmo-dev make target for building from source, if running without + `--binary-repo`. This is usually the name of the git repository, but could + also be e.g. `.make.osmocom-bb.clone` to only clone and not build + `osmocom-bb.git` for faketrx. Set `make=no` to not build/clone anything. + +* `package=`: debian package(s) to be installed for running a test component + and the script in `prepare=`. Multiple values separated by spaces. Set to + `package=no` to not install any package. + +* `prepare=`: optional script to run before staring the program (after files + are copied to the test directory). Typically this is used to create configs + with `osmo-config-merge`. See below for `PATH`. + +* `setup=`: optional script to run after the program was started. Execution of + the next program / the testsuite will wait until the setup script has quit. + This can be used to wait until the program is ready or to fill a test + database for example. See below for `PATH`. + +* `clean=`: optional script to run before `prepare=` and on exit. This can be + used to clean up network devices for example, or to fix name collisions when + running a test with multiple configs (`rename_junit_xml_classname.sh`). See + below for `PATH`. A `TESTENV_CLEAN_REASON` env var is set to `prepare`, + `crashed` or `finished` depending on when the script runs. + +### PATH + +Executables mentioned in `program=`, `prepare=`, `setup=` and `clean=` run +with a `PATH` environment variable containing: + +* The directory of the testsuite +* The directory for binaries built from source +* The directory `_testenv/data/scripts` (which has e.g. `respawn.sh`) + +### Latest configs + +Sometimes we need to run test components and/or testsuites with different +configurations depending on whether we are currently testing the nightly/master +versions of test components or the latest (stable) versions. For example, when +a new feature gets introduced that we need to configure and test, but which is +not available in the latest stable version. + +For this purpose, it is possible to add configuration keys ending in `_latest` +to `testenv.cfg`. These keys will override the original keys if `testenv.py` +is running with a binary repository ending in `:latest` or with `--latest`. + +It is also possible to take different code paths or exclude tests in the +TTCN-3 code if the latest configs are (not) used. This is done with +`f_osmo_repo_is` in `library/Misc_Helpers.ttcn`. + +### Multiple testenv.cfg files + +Usually each testsuite has only one `testenv.cfg` file, e.g. `mgw/testenv.cfg`. +For some testsuites it is necessary to run them multiple times with slightly +different configurations. In that case, we have multiple `testenv.cfg` files, +typically one `testenv_generic.cfg` and additional `testenv_<NAME>.cfg` files. + +For example: +* `bts/testenv_generic.cfg` +* `bts/testenv_hopping.cfg` +* `bts/testenv_oml.cfg` + +## Environment variables + +* `TESTENV_SRC_DIR`: + Set the directory for sources of Osmocom components. The default is the + directory above your osmo-ttcn3-hacks.git clone. + +* `TESTENV_NO_IMAGE_UP_TO_DATE_CHECK`: + Do not compare the timestamp of `data/podman/Dockerfile` with the date of the + podman image. This check does not work on jenkins where we always have + a fresh clone. + +* `TESTENV_COLOR_{DEBUG|INFO|WARNING|ERROR|CRITICAL|RESET}`: + Change the colors for different log levels (we use this in Jenkins, which + prints the output on white background). Find the defaults in + `testenv/__init__.py:ColorFormatter()`. + +* `TESTENV_SOURCE_HIGHLIGHT_COLORS`: + The argument to pass to `source-highlight` for formatting the junit log xml + files. Jenkins sets this to `esc` instead of `esc256` for better contrast on + white background. + +## Troubleshooting + +### Timeout waiting for RESET-ACK after sending RESET + +This can happen if another Osmocom program is started before OsmoSTP, then +tries to connect to OsmoSTP and fails, and waits several seconds before +retrying. Check the logs to confirm this is the case. To fix this, adjust your +`testenv.cfg` to start OsmoSTP before other Osmocom programs. The testenv +scripts will wait a bit to give OsmoSTP enough time to start up, before +starting the other test components. diff --git a/_testenv/data/osmo-dev/osmo-bts-trx.opts b/_testenv/data/osmo-dev/osmo-bts-trx.opts new file mode 100644 index 0000000..8acec2d --- /dev/null +++ b/_testenv/data/osmo-dev/osmo-bts-trx.opts @@ -0,0 +1 @@ +osmo-bts --enable-trx diff --git a/_testenv/data/podman/Dockerfile b/_testenv/data/podman/Dockerfile new file mode 100644 index 0000000..6cae472 --- /dev/null +++ b/_testenv/data/podman/Dockerfile @@ -0,0 +1,125 @@ +ARG REGISTRY=docker.io +ARG DISTRO=debian:12 +FROM ${REGISTRY}/${DISTRO} + +# Arguments used after FROM must be specified again +ARG OSMOCOM_REPO_TESTSUITE_MIRROR="https://downloads.osmocom.org" +ARG OSMOCOM_REPO="$OSMOCOM_REPO_TESTSUITE_MIRROR/packages/osmocom:/latest/Debian_12/" + +# Copy from common dir +COPY obs.key /obs.key + +# Install packages from Debian repositories (alphabetic order) +ENV DEBIAN_FRONTEND=noninteractive +RUN set -x && \ + apt-get update && \ + apt-get install -y --no-install-recommends \ + autoconf \ + automake \ + bison \ + build-essential \ + ca-certificates \ + ccache \ + cmake \ + flex \ + git \ + iproute2 \ + iputils-ping \ + libbson-dev \ + libc-ares-dev \ + libcsv-dev \ + libcurl4-gnutls-dev \ + libdbd-sqlite3 \ + libdbi-dev \ + libgcrypt-dev \ + libgnutls28-dev \ + libidn11-dev \ + libjansson-dev \ + libmicrohttpd-dev \ + libmnl-dev \ + libmongoc-dev \ + libnghttp2-dev \ + libortp-dev \ + libpcap-dev \ + libpcsclite-dev \ + libsctp-dev \ + libsofia-sip-ua-glib-dev \ + libsqlite3-dev \ + libssl-dev \ + libtalloc-dev \ + libtins-dev \ + libtool \ + libulfius-dev \ + liburing-dev \ + libusb-1.0-0-dev \ + libyaml-dev \ + libzmq3-dev \ + meson \ + netcat-openbsd \ + pcscd \ + pkg-config \ + procps \ + psmisc \ + python3-setuptools \ + rebar3 \ + rsync \ + source-highlight \ + sqlite3 \ + sudo \ + tcpdump \ + vim \ + wget \ + wireshark-common \ + && \ + apt-get clean + +# Ccache is installed above so it can be optionally used when rebuilding the +# testsuites inside the docker containers. Don't use it by default. +ENV USE_CCACHE=0 + +# Binary-only transcoding library for RANAP/RUA/HNBAP to work around TITAN only implementing BER +RUN set -x && \ + export DPKG_ARCH="$(dpkg --print-architecture)" && \ + wget https://ftp.osmocom.org/binaries/libfftranscode/libfftranscode0_0.5_${DPKG_… && \ + wget https://ftp.osmocom.org/binaries/libfftranscode/libfftranscode-dev_0.5_${DP… && \ + dpkg -i ./libfftranscode0_0.5_${DPKG_ARCH}.deb ./libfftranscode-dev_0.5_${DPKG_ARCH}.deb && \ + apt install --fix-broken && \ + rm libfftranscode*.deb + +# Install osmo-python-tests (for obtaining talloc reports from SUT) +ADD https://gerrit.osmocom.org/plugins/gitiles/python/osmo-python-tests/+/maste… /tmp/osmo-python-tests-commit +RUN set -x && \ + git clone --depth=1 https://gerrit.osmocom.org/python/osmo-python-tests osmo-python-tests && \ + cd osmo-python-tests && \ + python3 setup.py clean build install && \ + cd .. && \ + rm -rf osmo-python-tests + +# Add eclipse-titan from osmocom:latest, invalidate cache when :latest changes +RUN echo "deb [signed-by=/obs.key] $OSMOCOM_REPO ./" \ + > /etc/apt/sources.list.d/osmocom-latest.list +ADD $OSMOCOM_REPO/Release /tmp/Release +RUN set -x && \ + apt-get update && \ + apt-get install -y --no-install-recommends \ + eclipse-titan \ + && \ + apt-get clean && \ + rm /etc/apt/sources.list.d/osmocom-latest.list + +# Add mongodb for open5gs-hss. Using the package from bullseye since bookworm +# mongodb-org package is not available. Furthermore, manually install required +# libssl1.1. +RUN set -x && \ + mkdir -p /tmp/mongodb && \ + cd /tmp/mongodb && \ + wget "https://pgp.mongodb.com/server-5.0.asc" -O "/mongodb.key" && \ + wget "http://security.debian.org/debian-security/pool/updates/main/o/openssl/libssl1.1_1.1.1n-0+deb10u6_amd64.deb" && \ + dpkg -i "libssl1.1_1.1.1n-0+deb10u6_amd64.deb" && \ + echo "deb [signed-by=/mongodb.key] http://repo.mongodb.org/apt/debian bullseye/mongodb-org/5.0 main" \ + > /etc/apt/sources.list.d/mongodb-org.list && \ + apt-get update && \ + apt-get install -y mongodb-org && \ + apt-get clean && \ + cd / && \ + rm -rf /tmp/mongodb diff --git a/_testenv/data/podman/obs.key b/_testenv/data/podman/obs.key new file mode 100644 index 0000000..ecca084 --- /dev/null +++ b/_testenv/data/podman/obs.key @@ -0,0 +1,26 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- +Version: GnuPG v1.4.5 (GNU/Linux) + +mQENBGKzE1QBCADFcM3ZzggvgxNRNNqDGWf5xIDAiK5qzFLdGes7L6F9VCHdaPy0 +RAOB5bFb/Q1tSDFNEBLtaauXKz+4iGL6qMVjZcyjzpB5w4jKN+kkrFRhjDNUv/SH +BX6d+P7v5WBGSNArNgA8D1BGzckp5a99EZ0okMJFEqIcN40PD6OGugpq5XnVV5Nk +e93fLa2Cu8vhFBcVn6CuHeEhsmuMf6NLbQRNfNNCEEUYaZn7beMYtpZ7t1djsKx5 +1xGm50OzI22FLu8lELQ9d7qMVGRG3WHYawX9BDteRybiyqxfwUHm1haWazRJtlGt +UWyzvwAb80BK1J2Nu5fbAa3w5CoEPAbUuCyrABEBAAG0JW9zbW9jb20gT0JTIFBy +b2plY3QgPG9zbW9jb21Ab3Ntb2NvbT6JAVQEEwEIAD4WIQRrKp83ktFetw1Oao+G +pzC2U3JZcwUCYrMV4wIbAwUJBB6yjwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAK +CRCGpzC2U3JZc4FRCACQQkKIrnvQ7n2u7GSmyVZa3I+oLoFXSGqaGyey5TW/nrMm +vFDKU3qliHiuNSmUY35SnAhXUsvqOYppxVRoO1MLrqUvzMOnIWqkJpf8mtjGUnsW +jyVeto7Rsjs75y2i1Hk+e7ljb/V65J3NlfrfEYWbqR9AKd53ReNXTdrQ0J05A38N +GdI4Ld/2lNISAwaBmGhqdeKsLHpQw/JERU1TApVJR1whFiIwDF1rOCg9GPnNKIk7 +yRZdK267XzztrainX/cbPILyzUZEDhYs6wQuyACyQ1YUxZIxrwVfk7PMNay8CrLH +z42B73Ne5IAj8+op/3iJafFONLm7YXiDUFN+QDYAiQEzBBMBCAAdFiEExoiYhHND +S7aVYlnqa51NyAUyjdsFAmKzE1UACgkQa51NyAUyjdvuZgf+OXmr//i7u7Gg7eWB +7e0qUsyCId9lXS8J437x3K6ciJfD7/6RSy8TFW5Nglm/uSkbyq582I8t+SoOirMD +E6cg9U/5+h5s46bAf+Kd2XS/6tLGeNLM18i4el8CP06NpFzDrsKu76uYFpyRiiHD +otBdtgxeLJ83LugGfZslF+/5cigJkAJMhAdVvGO8h85R6fba8ZSOKtMKkaQRfi76 +nhyOrJPlLuS+DLEnHwdkOFgtKnxHdjM97K+Tx0gisb6uwaWroXfSLnhP8RTLLZZy +Z+noU1Hw3c+mn4c/NYbcC/uwHYHKRzuf9gHnQ3dGgv0Z5sbeLRVo92hjGj7Ftlyd +4hmKBg== +=HxK4 +-----END PGP PUBLIC KEY BLOCK----- diff --git a/_testenv/data/podman/seccomp_profile.json b/_testenv/data/podman/seccomp_profile.json new file mode 100644 index 0000000..70a0b63 --- /dev/null +++ b/_testenv/data/podman/seccomp_profile.json @@ -0,0 +1,836 @@ +{ + "defaultAction": "SCMP_ACT_ERRNO", + "defaultErrnoRet": 1, + "archMap": [ + { + "architecture": "SCMP_ARCH_X86_64", + "subArchitectures": [ + "SCMP_ARCH_X86", + "SCMP_ARCH_X32" + ] + }, + { + "architecture": "SCMP_ARCH_AARCH64", + "subArchitectures": [ + "SCMP_ARCH_ARM" + ] + }, + { + "architecture": "SCMP_ARCH_MIPS64", + "subArchitectures": [ + "SCMP_ARCH_MIPS", + "SCMP_ARCH_MIPS64N32" + ] + }, + { + "architecture": "SCMP_ARCH_MIPS64N32", + "subArchitectures": [ + "SCMP_ARCH_MIPS", + "SCMP_ARCH_MIPS64" + ] + }, + { + "architecture": "SCMP_ARCH_MIPSEL64", + "subArchitectures": [ + "SCMP_ARCH_MIPSEL", + "SCMP_ARCH_MIPSEL64N32" + ] + }, + { + "architecture": "SCMP_ARCH_MIPSEL64N32", + "subArchitectures": [ + "SCMP_ARCH_MIPSEL", + "SCMP_ARCH_MIPSEL64" + ] + }, + { + "architecture": "SCMP_ARCH_S390X", + "subArchitectures": [ + "SCMP_ARCH_S390" + ] + }, + { + "architecture": "SCMP_ARCH_RISCV64", + "subArchitectures": null + } + ], + "syscalls": [ + { + "names": [ + "accept", + "accept4", + "access", + "adjtimex", + "alarm", + "bind", + "brk", + "cachestat", + "capget", + "capset", + "chdir", + "chmod", + "chown", + "chown32", + "clock_adjtime", + "clock_adjtime64", + "clock_getres", + "clock_getres_time64", + "clock_gettime", + "clock_gettime64", + "clock_nanosleep", + "clock_nanosleep_time64", + "close", + "close_range", + "connect", + "copy_file_range", + "creat", + "dup", + "dup2", + "dup3", + "epoll_create", + "epoll_create1", + "epoll_ctl", + "epoll_ctl_old", + "epoll_pwait", + "epoll_pwait2", + "epoll_wait", + "epoll_wait_old", + "eventfd", + "eventfd2", + "execve", + "execveat", + "exit", + "exit_group", + "faccessat", + "faccessat2", + "fadvise64", + "fadvise64_64", + "fallocate", + "fanotify_mark", + "fchdir", + "fchmod", + "fchmodat", + "fchmodat2", + "fchown", + "fchown32", + "fchownat", + "fcntl", + "fcntl64", + "fdatasync", + "fgetxattr", + "flistxattr", + "flock", + "fork", + "fremovexattr", + "fsetxattr", + "fstat", + "fstat64", + "fstatat64", + "fstatfs", + "fstatfs64", + "fsync", + "ftruncate", + "ftruncate64", + "futex", + "futex_requeue", + "futex_time64", + "futex_wait", + "futex_waitv", + "futex_wake", + "futimesat", + "getcpu", + "getcwd", + "getdents", + "getdents64", + "getegid", + "getegid32", + "geteuid", + "geteuid32", + "getgid", + "getgid32", + "getgroups", + "getgroups32", + "getitimer", + "getpeername", + "getpgid", + "getpgrp", + "getpid", + "getppid", + "getpriority", + "getrandom", + "getresgid", + "getresgid32", + "getresuid", + "getresuid32", + "getrlimit", + "get_robust_list", + "getrusage", + "getsid", + "getsockname", + "getsockopt", + "get_thread_area", + "gettid", + "gettimeofday", + "getuid", + "getuid32", + "getxattr", + "inotify_add_watch", + "inotify_init", + "inotify_init1", + "inotify_rm_watch", + "io_cancel", + "ioctl", + "io_destroy", + "io_getevents", + "io_pgetevents", + "io_pgetevents_time64", + "ioprio_get", + "ioprio_set", + "io_setup", + "io_submit", + "io_uring_enter", + "io_uring_register", + "io_uring_setup", + "ipc", + "kill", + "landlock_add_rule", + "landlock_create_ruleset", + "landlock_restrict_self", + "lchown", + "lchown32", + "lgetxattr", + "link", + "linkat", + "listen", + "listxattr", + "llistxattr", + "_llseek", + "lremovexattr", + "lseek", + "lsetxattr", + "lstat", + "lstat64", + "madvise", + "map_shadow_stack", + "membarrier", + "memfd_create", + "memfd_secret", + "mincore", + "mkdir", + "mkdirat", + "mknod", + "mknodat", + "mlock", + "mlock2", + "mlockall", + "mmap", + "mmap2", + "mprotect", + "mq_getsetattr", + "mq_notify", + "mq_open", + "mq_timedreceive", + "mq_timedreceive_time64", + "mq_timedsend", + "mq_timedsend_time64", + "mq_unlink", + "mremap", + "msgctl", + "msgget", + "msgrcv", + "msgsnd", + "msync", + "munlock", + "munlockall", + "munmap", + "name_to_handle_at", + "nanosleep", + "newfstatat", + "_newselect", + "open", + "openat", + "openat2", + "pause", + "pidfd_open", + "pidfd_send_signal", + "pipe", + "pipe2", + "pkey_alloc", + "pkey_free", + "pkey_mprotect", + "poll", + "ppoll", + "ppoll_time64", + "prctl", + "pread64", + "preadv", + "preadv2", + "prlimit64", + "process_mrelease", + "pselect6", + "pselect6_time64", + "pwrite64", + "pwritev", + "pwritev2", + "read", + "readahead", + "readlink", + "readlinkat", + "readv", + "recv", + "recvfrom", + "recvmmsg", + "recvmmsg_time64", + "recvmsg", + "remap_file_pages", + "removexattr", + "rename", + "renameat", + "renameat2", + "restart_syscall", + "rmdir", + "rseq", + "rt_sigaction", + "rt_sigpending", + "rt_sigprocmask", + "rt_sigqueueinfo", + "rt_sigreturn", + "rt_sigsuspend", + "rt_sigtimedwait", + "rt_sigtimedwait_time64", + "rt_tgsigqueueinfo", + "sched_getaffinity", + "sched_getattr", + "sched_getparam", + "sched_get_priority_max", + "sched_get_priority_min", + "sched_getscheduler", + "sched_rr_get_interval", + "sched_rr_get_interval_time64", + "sched_setaffinity", + "sched_setattr", + "sched_setparam", + "sched_setscheduler", + "sched_yield", + "seccomp", + "select", + "semctl", + "semget", + "semop", + "semtimedop", + "semtimedop_time64", + "send", + "sendfile", + "sendfile64", + "sendmmsg", + "sendmsg", + "sendto", + "setfsgid", + "setfsgid32", + "setfsuid", + "setfsuid32", + "setgid", + "setgid32", + "setgroups", + "setgroups32", + "setitimer", + "setpgid", + "setpriority", + "setregid", + "setregid32", + "setresgid", + "setresgid32", + "setresuid", + "setresuid32", + "setreuid", + "setreuid32", + "setrlimit", + "set_robust_list", + "setsid", + "setsockopt", + "set_thread_area", + "set_tid_address", + "setuid", + "setuid32", + "setxattr", + "shmat", + "shmctl", + "shmdt", + "shmget", + "shutdown", + "sigaltstack", + "signalfd", + "signalfd4", + "sigprocmask", + "sigreturn", + "socketcall", + "socketpair", + "splice", + "stat", + "stat64", + "statfs", + "statfs64", + "statx", + "symlink", + "symlinkat", + "sync", + "sync_file_range", + "syncfs", + "sysinfo", + "tee", + "tgkill", + "time", + "timer_create", + "timer_delete", + "timer_getoverrun", + "timer_gettime", + "timer_gettime64", + "timer_settime", + "timer_settime64", + "timerfd_create", + "timerfd_gettime", + "timerfd_gettime64", + "timerfd_settime", + "timerfd_settime64", + "times", + "tkill", + "truncate", + "truncate64", + "ugetrlimit", + "umask", + "uname", + "unlink", + "unlinkat", + "utime", + "utimensat", + "utimensat_time64", + "utimes", + "vfork", + "vmsplice", + "wait4", + "waitid", + "waitpid", + "write", + "writev" + ], + "action": "SCMP_ACT_ALLOW" + }, + { + "names": [ + "process_vm_readv", + "process_vm_writev", + "ptrace" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "minKernel": "4.8" + } + }, + { + "names": [ + "socket" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 40, + "op": "SCMP_CMP_NE" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 0, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 8, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 131072, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 131080, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 4294967295, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "sync_file_range2", + "swapcontext" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "ppc64le" + ] + } + }, + { + "names": [ + "arm_fadvise64_64", + "arm_sync_file_range", + "sync_file_range2", + "breakpoint", + "cacheflush", + "set_tls" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "arm", + "arm64" + ] + } + }, + { + "names": [ + "arch_prctl" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "amd64", + "x32" + ] + } + }, + { + "names": [ + "modify_ldt" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "amd64", + "x32", + "x86" + ] + } + }, + { + "names": [ + "s390_pci_mmio_read", + "s390_pci_mmio_write", + "s390_runtime_instr" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "s390", + "s390x" + ] + } + }, + { + "names": [ + "riscv_flush_icache" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "riscv64" + ] + } + }, + { + "names": [ + "open_by_handle_at" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_DAC_READ_SEARCH" + ] + } + }, + { + "names": [ + "bpf", + "clone", + "clone3", + "fanotify_init", + "fsconfig", + "fsmount", + "fsopen", + "fspick", + "lookup_dcookie", + "mount", + "mount_setattr", + "move_mount", + "open_tree", + "perf_event_open", + "quotactl", + "quotactl_fd", + "setdomainname", + "sethostname", + "setns", + "syslog", + "umount", + "umount2", + "unshare" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_ADMIN" + ] + } + }, + { + "names": [ + "clone" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 2114060288, + "op": "SCMP_CMP_MASKED_EQ" + } + ], + "excludes": { + "caps": [ + "CAP_SYS_ADMIN" + ], + "arches": [ + "s390", + "s390x" + ] + } + }, + { + "names": [ + "clone" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 1, + "value": 2114060288, + "op": "SCMP_CMP_MASKED_EQ" + } + ], + "comment": "s390 parameter ordering for clone is different", + "includes": { + "arches": [ + "s390", + "s390x" + ] + }, + "excludes": { + "caps": [ + "CAP_SYS_ADMIN" + ] + } + }, + { + "names": [ + "clone3" + ], + "action": "SCMP_ACT_ERRNO", + "errnoRet": 38, + "excludes": { + "caps": [ + "CAP_SYS_ADMIN" + ] + } + }, + { + "names": [ + "reboot" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_BOOT" + ] + } + }, + { + "names": [ + "chroot" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_CHROOT" + ] + } + }, + { + "names": [ + "delete_module", + "init_module", + "finit_module" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_MODULE" + ] + } + }, + { + "names": [ + "acct" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_PACCT" + ] + } + }, + { + "names": [ + "kcmp", + "pidfd_getfd", + "process_madvise", + "process_vm_readv", + "process_vm_writev", + "ptrace" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_PTRACE" + ] + } + }, + { + "names": [ + "iopl", + "ioperm" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_RAWIO" + ] + } + }, + { + "names": [ + "settimeofday", + "stime", + "clock_settime", + "clock_settime64" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_TIME" + ] + } + }, + { + "names": [ + "vhangup" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_TTY_CONFIG" + ] + } + }, + { + "names": [ + "get_mempolicy", + "mbind", + "set_mempolicy", + "set_mempolicy_home_node" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_NICE" + ] + } + }, + { + "names": [ + "syslog" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYSLOG" + ] + } + }, + { + "names": [ + "bpf" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_BPF" + ] + } + }, + { + "names": [ + "perf_event_open" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_PERFMON" + ] + } + } + ] +} diff --git a/_testenv/data/scripts/log_format.sh b/_testenv/data/scripts/log_format.sh new file mode 100755 index 0000000..b181e10 --- /dev/null +++ b/_testenv/data/scripts/log_format.sh @@ -0,0 +1,7 @@ +#!/bin/sh -e + +for i in *.merged; do + temp="$i.temp" + ttcn3_logformat -o "$temp" "$i" + mv "$temp" "$i" +done diff --git a/_testenv/data/scripts/rename_junit_xml_classname.sh b/_testenv/data/scripts/rename_junit_xml_classname.sh new file mode 100755 index 0000000..64d0062 --- /dev/null +++ b/_testenv/data/scripts/rename_junit_xml_classname.sh @@ -0,0 +1,32 @@ +#!/bin/sh -e +SUFFIX="$1" + +if [ -z "$SUFFIX" ]; then + echo "usage: rename_junit_xml_classname.sh SUFFIX" + echo "example: rename_junit_xml_classname ':hopping'" + exit 1 +fi + +xmls="$(find -name 'junit-xml*.log')" + +if [ -z "$xmls" ]; then + case "$TESTENV_CLEAN_REASON" in + prepare|crashed) + # No xml files is expected + exit 0 + ;; + finished) + echo "ERROR: could not find any junit-xml*.log files!" + exit 1 + ;; + *) + echo "ERROR: invalid TESTENV_CLEAN_REASON: $TESTENV_CLEAN_REASON" + exit 1 + ;; + esac +fi + +for i in $xmls; do + echo "Adding '$SUFFIX' to classnames in: $i" + sed -i "s/classname='\([^']\+\)'/classname='\1$SUFFIX'/g" $i +done diff --git a/_testenv/data/scripts/respawn.sh b/_testenv/data/scripts/respawn.sh new file mode 100755 index 0000000..80e4055 --- /dev/null +++ b/_testenv/data/scripts/respawn.sh @@ -0,0 +1,21 @@ +#!/bin/sh + +trap "kill 0" EXIT + +SLEEP_BEFORE_RESPAWN=${SLEEP_BEFORE_RESPAWN:-0} + +i=0 +max_i=500 +while [ $i -lt $max_i ]; do + echo "respawn: $i: starting: $*" + $* & + LAST_PID=$! + wait $LAST_PID + echo "respawn: $i: stopped pid $LAST_PID with status $?" + if [ $SLEEP_BEFORE_RESPAWN -gt 0 ]; then + echo "respawn: sleeping $SLEEP_BEFORE_RESPAWN seconds..." + sleep $SLEEP_BEFORE_RESPAWN + fi + i=$(expr $i + 1) +done +echo "respawn: exiting after $max_i runs" diff --git a/_testenv/data/scripts/testenv-podman-main.sh b/_testenv/data/scripts/testenv-podman-main.sh new file mode 100755 index 0000000..d0bc586 --- /dev/null +++ b/_testenv/data/scripts/testenv-podman-main.sh @@ -0,0 +1,20 @@ +#!/bin/sh -e +# Simple watchdog script that exits if either: +# * testenv doesn't create /tmp/watchdog every 10s +# * 4 hours have passed +# This ensures the podman container stops a few seconds after a jenkins job was +# aborted, or if a test is stuck in a loop for hours. + +stop_time=$(($(date +%s) + 3600 * 4)) + +while [ $(date +%s) -lt $stop_time ]; do + sleep 10 + + if ! [ -e /tmp/watchdog ]; then + break + fi + + rm /tmp/watchdog +done + +exit 1 diff --git a/_testenv/testenv.py b/_testenv/testenv.py new file mode 100755 index 0000000..bbf9ada --- /dev/null +++ b/_testenv/testenv.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +import logging +import os +import sys +import testenv +import testenv.cmd +import testenv.daemons +import testenv.osmo_dev +import testenv.podman +import testenv.podman_install +import testenv.requirements +import testenv.testdir +import testenv.testenv_cfg +import testenv.testsuite + + +def run(): + testenv.testenv_cfg.init() + + if not testenv.args.binary_repo: + testenv.osmo_dev.check_init_needed() + + testenv.requirements.check() + testenv.podman_install.init() + testenv.cmd.init_env() + testenv.testdir.init() + testenv.daemons.init() + + if testenv.args.podman: + testenv.podman.init() + testenv.podman.start() + + if not testenv.args.binary_repo: + testenv.osmo_dev.init() + + testenv.testsuite.init() + testenv.testsuite.build() + + # Run prepare functions of testsuites (may enable extra repos) + for cfg_name, cfg in testenv.testenv_cfg.cfgs.items(): + testenv.testenv_cfg.set_current(cfg_name) + testenv.testsuite.run_prepare_script(cfg) + + # Build all components first + if not testenv.args.binary_repo: + for cfg_name, cfg in testenv.testenv_cfg.cfgs.items(): + testenv.testenv_cfg.set_current(cfg_name) + testenv.osmo_dev.make(cfg) + + # Run the components + testsuite + cfg_count = 0 + for cfg_name, cfg in testenv.testenv_cfg.cfgs.items(): + testenv.testenv_cfg.set_current(cfg_name) + + if testenv.args.binary_repo: + testenv.podman.enable_binary_repo() + testenv.podman_install.packages(cfg, cfg_name) + + testenv.testdir.prepare(cfg_name, cfg) + testenv.daemons.start(cfg) + testenv.testsuite.run(cfg) + testenv.daemons.stop() + testenv.testdir.clean_run_scripts("finished") + testenv.testsuite.cat_junit_logs() + + cfg_count += 1 + testenv.set_log_prefix("[testenv]") + + # Restart podman container before running with another config + if testenv.args.podman: + restart = cfg_count < len(testenv.testenv_cfg.cfgs) + testenv.podman.stop(restart) + + +def init_podman(): + testenv.podman.init_image_name_distro() + testenv.podman.image_build() + + +def init_osmo_dev(): + testenv.osmo_dev.init_clone() + + +def clean(): + cache_dirs = [ + "git", + "host", + "podman", + ] + for cache_dir in cache_dirs: + path = os.path.join(testenv.cache_dir_default, cache_dir) + if os.path.exists(path): + testenv.cmd.run(["rm", "-rf", path]) + + +def main(): + testenv.init_logging() + testenv.init_args() + + action = testenv.args.action + if action == "run": + run() + elif action == "init": + if testenv.args.runtime == "osmo-dev": + init_osmo_dev() + elif testenv.args.runtime == "podman": + init_podman() + elif action == "clean": + clean() + + +try: + main() +except testenv.NoTraceException as e: + logging.error(e) + testenv.podman.stop() + sys.exit(2) +except KeyboardInterrupt: + print("") # new line + testenv.podman.stop() + sys.exit(3) +except: + testenv.podman.stop() + raise diff --git a/_testenv/testenv/__init__.py b/_testenv/testenv/__init__.py new file mode 100644 index 0000000..0c3066d --- /dev/null +++ b/_testenv/testenv/__init__.py @@ -0,0 +1,199 @@ +import argparse +import logging +import os.path + + +class NoTraceException(Exception): + pass + +args = None + +src_dir = os.environ.get("TESTENV_SRC_DIR", os.path.realpath(f"{__file__}/../../../..")) +data_dir = os.path.join(os.path.realpath(f"{__file__}/../.."), "data") +distro_default = "debian:bookworm" +cache_dir_default = os.path.join(os.path.expanduser("~/.cache"), "osmo-ttcn3-testenv") +ccache_dir_default = os.path.join(cache_dir_default, "ccache") + +log_prefix = "[testenv]" + + +def resolve_testsuite_name_alias(name): + mapping = { + "ggsn": "ggsn_tests", + } + + if name in mapping: + logging.debug(f"Using testsuite {mapping[name]} (via alias {name})") + return mapping[name] + + return name + + +def parse_args(): + global args + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description="Build/install everything for a testsuite and run it.\n" + "\n" + "examples:\n" + " ./testenv.py run mgw\n" + " ./testenv.py run mgw --test TC_crcx\n" + " ./testenv.py run mgw --podman --binary-repo osmocom:latest\n" + " ./testenv.py run mgw --io-uring\n" + " ./testenv.py run bts --config oml\n" + ) + + sub = parser.add_subparsers(title="action", dest="action", required=True) + + sub_init = sub.add_parser("init", help="initialize osmo-dev/podman") + sub_init_runtime = sub_init.add_subparsers(title="runtime", required=True, + dest="runtime") + sub_osmo_dev = sub_init_runtime.add_parser("osmo-dev", + help="prepare osmo-dev (top-level makefile scripts, for building test" + " components from source when using 'run' without '--binary-repo')") + + sub_podman = sub_init_runtime.add_parser("podman", + help="prepare the podman image (for 'run --podman')") + sub_podman.add_argument("-f", "--force", action="store_true", + help="build image even if it is up-to-date") + + sub_run = sub.add_parser("run", help="build components and run a testsuite") + + group = sub_run.add_argument_group("testsuite options") + group.add_argument("testsuite", + help="a directory in osmo-ttcn3-hacks.git (msc, bsc," + " mgw, ...)") + group.add_argument("-t", "--test", + help="only run one specific test (e.g. TC_selftest," + " BTS_Tests_OML.TC_wrong_mdisc)") + group.add_argument("-c", "--config", action="append", + help="which testenv.cfg to use, in case the testsuite" + " has multiple (e.g. generic|oml|hopping for bts)") + group.add_argument("-i", "--io-uring", + action="store_true", + help="set LIBOSMO_IO_BACKEND=IO_URING") + + group = sub_run.add_argument_group("source/binary options", + "All components are built from source" + " by default.") + group = group.add_mutually_exclusive_group() + group.add_argument("-b", "--binary-repo", metavar="OBS_PROJECT", + help="use binary packages from this Osmocom OBS project" + " instead (e.g. osmocom:nightly)") + group = sub_run.add_argument_group("config file options", + "Testsuite and test component configs" + " for nightly/master versions of test" + " components are used, unless a binary" + " repository ending in :latest is set" + " or --latest is used.") + group.add_argument("--latest", action="store_true", + help="use latest configs") + + group = sub_run.add_argument_group("podman options", + "All components are run directly on the" + " host by default.") + group.add_argument("-p", "--podman", action="store_true", + help="run all components inside podman") + group.add_argument("-d", "--distro", default=distro_default, + help="distribution for podman (default:" + f" {distro_default})") + group.add_argument("-s", "--shell", action="store_true", + help="run an interactive shell before stopping daemons/" + "container") + + group = sub_run.add_argument_group("output options") + group.add_argument("-l", "--log-dir", + help="log here instead of a random dir in /tmp") + group.add_argument("-n", "--no-tee", dest="tee", action="store_false", + help="don't send test component's output to stdout") + + group = sub_run.add_argument_group("cache options") + group.add_argument("--cache", + help=f"cache path (default: {cache_dir_default})", + default=cache_dir_default) + group.add_argument("--ccache", + help=f"ccache path (default: {ccache_dir_default})", + default=ccache_dir_default) + + + sub_clean = sub.add_parser("clean", help="clean previous build artifacts") + + args = parser.parse_args() + + if args.action == "run": + args.testsuite = resolve_testsuite_name_alias(args.testsuite) + if args.binary_repo and args.binary_repo.endswith(":latest"): + logging.debug("Binary repository ends in :latest, using latest" + " configs") + args.latest = True + + +def verify_args_run(): + if args.action != "run": + return + + if args.binary_repo and not args.podman: + raise NoTraceException("--binary-repo requires --podman") + + ttcn3_hacks_dir_src = os.path.realpath(f"{__file__}/../../..") + testsuite_dir = os.path.join(ttcn3_hacks_dir_src, args.testsuite) + if not os.path.exists(testsuite_dir): + raise NoTraceException(f"testsuite dir not found: {testsuite_dir}") + + +def init_args(): + parse_args() + verify_args_run() + + +class ColorFormatter(logging.Formatter): + colors = { + "debug": "\033[37m", # light gray + "info": "\033[94m", # blue + "warning": "\033[93m", # yellow + "error": "\033[91m", # red + "critical": "\033[91m", # red + "reset": "\033[0m", + } + + def __init__(self): + for color in self.colors.keys(): + env_var = f"TESTENV_COLOR_{color.upper()}" + if env_var in os.environ: + self.colors[color] = os.environ.get(env_var) + + super().__init__() + + def format(self, record): + if record.levelno == logging.DEBUG: + color = "debug" + elif record.levelno == logging.INFO: + color = "info" + elif record.levelno == logging.WARNING: + color = "warning" + elif record.levelno == logging.ERROR: + color = "error" + elif record.levelno == logging.CRITICAL: + color = "critical" + + self._style._fmt = f"{self.colors[color]}{log_prefix} %(msg)s{self.colors['reset']}" + result = logging.Formatter.format(self, record) + + return result + + +def init_logging(): + formatter = ColorFormatter() + root_logger = logging.getLogger() + root_logger.setLevel(logging.DEBUG) + root_logger.handlers = [] + + handler = logging.StreamHandler() + handler.setFormatter(formatter) + root_logger.addHandler(handler) + + +def set_log_prefix(new): + global log_prefix + + log_prefix = new diff --git a/_testenv/testenv/cmd.py b/_testenv/testenv/cmd.py new file mode 100644 index 0000000..678062e --- /dev/null +++ b/_testenv/testenv/cmd.py @@ -0,0 +1,102 @@ +import logging +import os +import os.path +import subprocess +import sys +import tempfile +import testenv +import testenv.testsuite + +env_extra = {} +usr_dir = None + + +def init_env(): + global env_extra + global usr_dir + + if testenv.args.podman: + if not testenv.args.binary_repo: + usr_dir = os.path.join(testenv.args.cache, "podman", "usr") + else: + usr_dir = os.path.join(testenv.args.cache, "host", "usr") + + if usr_dir: + pkg_config_path = os.path.join(usr_dir, "lib/pkgconfig") + if "PKG_CONFIG_PATH" in os.environ: + pkg_config_path += f":{os.environ.get('PKG_CONFIG_PATH')}" + pkg_config_path += f":/usr/lib/pkgconfig" + + ld_library_path = os.path.join(usr_dir, "lib") + if "LD_LIBRARY_PATH" in os.environ: + ld_library_path += f":{os.environ.get('LD_LIBRARY_PATH')}" + ld_library_path += f":/usr/lib" + + env_extra["PKG_CONFIG_PATH"] = pkg_config_path + env_extra["LD_LIBRARY_PATH"] = ld_library_path + + env_extra["CCACHE_DIR"] = testenv.args.ccache + env_extra["TESTENV_CACHE_DIR"] = testenv.args.cache + env_extra["TESTENV_SRC_DIR"] = testenv.src_dir + + env_extra["TERM"] = os.environ.get("TERM", "dumb") + + if testenv.args.binary_repo: + env_extra["TESTENV_GIT_DIR"] = testenv.podman_install.git_dir + else: + if testenv.args.podman: + env_extra["OSMO_DEV_MAKE_DIR"] = os.path.join( + testenv.args.cache, "podman", "make") + else: + env_extra["OSMO_DEV_MAKE_DIR"] = os.path.join( + testenv.args.cache, "host", "make") + + +def exit_error_cmd(completed, error_msg): + """ :param completed: return from run_cmd() below """ + + logging.error(error_msg) + logging.debug(f"Command: {completed.args}") + logging.debug(f"Returncode: {completed.returncode}") + raise RuntimeError("shell command related error, find details right above" + " this python trace") + + +def generate_env(env={}, podman=False): + ret = dict(env_extra) + path = os.path.join(testenv.data_dir, 'scripts') + if testenv.testsuite.ttcn3_hacks_dir: + path += f":{os.path.join(testenv.testsuite.ttcn3_hacks_dir, testenv.args.testsuite)}" + + if usr_dir: + path += f":{os.path.join(usr_dir, 'bin')}" + + if podman: + path += f":/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" + else: + path += f":{os.environ.get('PATH')}" + + ret["PATH"] = path + ret["HOME"] = os.environ.get("HOME") + + for var in env: + ret[var] = env[var] + return ret + + +def run(cmd, check=True, env={}, no_podman=False, stdin=subprocess.DEVNULL, + *args, **kwargs): + + if not no_podman and testenv.podman.is_running(): + return testenv.podman.exec_cmd(cmd, check=check, env=env, *args, **kwargs) + + logging.debug(f"+ {cmd}") + + # Set stdin to /dev/null by default so we can still capture ^C with testenv + p = subprocess.run(cmd, env=generate_env(env), shell=isinstance(cmd, str), + stdin=stdin,*args, **kwargs) + + if p.returncode == 0 or not check: + return p + + exit_error_cmd(p, "Command failed unexpectedly") diff --git a/_testenv/testenv/daemons.py b/_testenv/testenv/daemons.py new file mode 100644 index 0000000..4c11901 --- /dev/null +++ b/_testenv/testenv/daemons.py @@ -0,0 +1,117 @@ +import atexit +import logging +import os +import os.path +import shlex +import subprocess +import testenv +import testenv.testdir +import time + +daemons = {} +run_shell_on_stop = False + + +def init(): + global run_shell_on_stop + + if not testenv.args.podman: + atexit.register(stop) + if testenv.args.shell: + run_shell_on_stop = True + + +def start(cfg): + global daemons + + for section in cfg: + if section in ["testenv", "DEFAULT"]: + continue + + section_data = cfg[section] + if "program" not in section_data: + continue + if section == "testsuite": + # Runs in the foreground with testenv.testsuite.run() + continue + + program = section_data["program"] + logging.info(f"Running {section}") + + cwd = os.path.join(testenv.testdir.testdir, section) + os.makedirs(cwd, exist_ok=True) + + log = os.path.join(testenv.testdir.testdir, section, f"{section}.log") + + if testenv.args.tee: + pipe = f"2>&1 | tee {shlex.quote(log)}" + else: + pipe = f">{shlex.quote(log)} 2>&1" + cmd = ["sh", "-c", f"{program} 2>&1 {pipe}"] + + env = {} + if testenv.args.io_uring: + env["LIBOSMO_IO_BACKEND"] = "IO_URING" + + if testenv.podman.is_running(): + daemons[section] = testenv.podman.exec_cmd_background(cmd, cwd=cwd, env=env) + else: + logging.debug(f"+ {cmd}") + daemons[section] = subprocess.Popen(cmd, cwd=cwd, + env=testenv.cmd.generate_env(env)) + + # Wait 200ms and check if it is still running + time.sleep(0.2) + if daemons[section].poll() is not None: + raise testenv.NoTraceException(f"program failed to start: {program}") + + # Run setup script + if "setup" in section_data: + setup = section_data["setup"] + logging.info(f"Running {section} setup script") + testenv.cmd.run(setup, cwd=cwd) + + +def kill_process_tree(pid, ppids): + subprocess.run(["kill", "-9", str(pid)], + check=False, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL) + + for (child_pid, child_ppid) in ppids: + if child_ppid == str(pid): + kill_process_tree(child_pid, ppids) + + +def kill(pid): + cmd = ["ps", "-e", "-o", "pid,ppid"] + ret = subprocess.run(cmd, check=True, stdout=subprocess.PIPE) + ppids = [] + proc_entries = ret.stdout.decode("utf-8").rstrip().split('\n')[1:] + for row in proc_entries: + items = row.split() + if len(items) != 2: + raise RuntimeError("Unexpected ps output: " + row) + ppids.append(items) + + kill_process_tree(pid, ppids) + + +def stop(): + global daemons + global run_shell_on_stop + + if run_shell_on_stop: + logging.info("Running interactive shell before stopping daemons (--shell)") + + # stdin=None: override stdin=/dev/null, so we can type into the shell + testenv.cmd.run(["bash"], cwd=testenv.testdir.testdir, stdin=None, check=False) + + run_shell_on_stop = False + + for daemon in daemons: + pid = daemons[daemon].pid + logging.info(f"Stopping {daemon} ({pid})") + kill(pid) + + daemons = {} diff --git a/_testenv/testenv/osmo_dev.py b/_testenv/testenv/osmo_dev.py new file mode 100644 index 0000000..75bf6df --- /dev/null +++ b/_testenv/testenv/osmo_dev.py @@ -0,0 +1,134 @@ +import logging +import os +import shlex +import sys +import testenv +import testenv.cmd + +make_dir = None +init_done = False + + +def get_osmo_dev_dir(): + # Users may have used osmo-dev to clone osmo-ttcn3-hacks: + # osmo-dev/src/osmo-ttcn3-hacks + alt_path = os.path.realpath(os.path.join(testenv.src_dir, "../")) + if os.path.exists(os.path.join(alt_path, "gen_makefile.py")): + return alt_path + + # Assume osmo-dev is next to osmo-ttcn3-hacks: + # src_dir + # ├── osmo-dev + # └── osmo-ttcn3-hacks + return os.path.join(testenv.src_dir, "osmo-dev") + + +def init_clone(): + osmo_dev_dir = get_osmo_dev_dir() + + if os.path.exists(osmo_dev_dir): + logging.debug(f"osmo-dev found, nothing to do: {osmo_dev_dir}") + return + + testenv.cmd.run(["git", "clone", "https://gerrit.osmocom.org/osmo-dev"], + cwd=testenv.src_dir) + + +def check_init_needed(): + osmo_dev_dir = get_osmo_dev_dir() + + if os.path.exists(osmo_dev_dir): + logging.debug(f"osmo-dev dir: {osmo_dev_dir}") + return + + logging.error("Missing osmo-dev for building test components from source.") + logging.error("Run 'testenv.py init osmo-dev' first.") + logging.error("") + logging.error("osmo-dev and other Osmocom repositories (if they don't" + " already exist) will be cloned to:") + logging.error(testenv.src_dir) + logging.error("") + logging.error("Set the environment variable TESTENV_SRC_DIR to use a" + " different path.") + sys.exit(1) + + +def init(): + global init_done + global make_dir + + if init_done: + return + + extra_opts = [] + if testenv.args.podman: + make_dir = os.path.join(testenv.args.cache, "podman", "make") + extra_opts = [ + "--install-prefix", os.path.join(testenv.args.cache, "podman/usr"), + ] + else: + make_dir = os.path.join(testenv.args.cache, "host", "make") + extra_opts = [ + "--install-prefix", os.path.join(testenv.args.cache, "host/usr"), + ] + + cmd = [ + "./gen_makefile.py", + "--build-debug", + "--make-dir", make_dir, + "--no-ldconfig", + "--src-dir", testenv.src_dir, + "default.opts", + "ccache.opts", + "iu.opts", + "no_dahdi.opts", + "no_doxygen.opts", + "no_systemd.opts", + "werror.opts", + os.path.join(testenv.data_dir, "osmo-dev/osmo-bts-trx.opts"), + ] + extra_opts + + testenv.cmd.run(cmd, cwd=get_osmo_dev_dir()) + init_done = True + + +def make(cfg, limit_section=None): + targets = [] + + for section in cfg: + section_data = cfg[section] + if section == "testsuite": + # Gets built with testenv.testsuite.build() + continue + if limit_section and limit_section != section: + # When called from testenv.podman.install_packages as fallback to + # not having a package available, then we only want to run make + # for the target of one specific config section + continue + + if "make" in section_data and section_data["make"] != "no" \ + and section_data["make"] not in targets: + targets += [section_data["make"]] + + if not targets: + logging.debug("No osmo-dev make targets found in testenv.cfg") + return + + makefile_path = os.path.join(make_dir, "Makefile") + with open(makefile_path) as f: + makefile = f.read() + + for target in targets: + if f"\n{target}:" in makefile: + continue + logging.error(f"Could not find make target: {target}") + logging.error("Add it to osmo-dev by adjusting:") + logging.error("* all.deps") + logging.error("* all.buildsystems (if buildsystem != autotools)") + logging.error("* all.urls (if the project is not on gerrit.osmocom.org)") + logging.error("Location of your osmo-dev.git clone:") + logging.error(os.path.join(testenv.src_dir, "osmo-dev")) + sys.exit(1) + + logging.info(f"Building test components") + testenv.cmd.run(["make"] + targets, cwd=make_dir) diff --git a/_testenv/testenv/podman.py b/_testenv/testenv/podman.py new file mode 100644 index 0000000..ac9fc7c --- /dev/null +++ b/_testenv/testenv/podman.py @@ -0,0 +1,272 @@ +import atexit +import datetime +import json +import logging +import multiprocessing +import os +import shlex +import subprocess +import testenv.cmd +import testenv.testdir +import time + +image_name = None +distro = None +container_name = None # instance of image +apt_dir_var_cache = None +apt_dir_var_lib = None +feed_watchdog_process = None +run_shell_on_stop = False + + +def image_exists(): + return testenv.cmd.run(["podman", "image", "exists", image_name], + check=False).returncode == 0 + + +def image_up_to_date(): + history = testenv.cmd.run(["podman", "history", image_name, "--format", "json"], + capture_output=True, text=True) + created = json.loads(history.stdout)[0]["created"].split(".",1)[0] + created = datetime.datetime.strptime(created, "%Y-%m-%dT%H:%M:%S") + logging.debug(f"Image creation date: {created}") + + # On a local development system, we can say that the podman image is + # outdated if the Dockerfile is newer than the creation date. But this does + # not work for jenkins where Dockerfile may just be from a new git + # checkout. So allow to skip this check. + if os.environ.get("TESTENV_NO_IMAGE_UP_TO_DATE_CHECK"): + logging.debug("Assuming the podman image is up-to-date") + return True + + dockerfile = os.path.join(testenv.data_dir, "podman/Dockerfile") + mtime = os.stat(dockerfile).st_mtime + mtime = datetime.datetime.utcfromtimestamp(mtime) + logging.debug(f"Dockerfile last modified: {str(mtime).split('.')[0]}") + + return mtime < created + + +def image_build(): + if image_exists() and image_up_to_date(): + logging.debug(f"Podman image is up-to-date: {image_name}") + if testenv.args.force: + logging.debug("Building anyway since --force was used") + else: + return + + logging.info(f"Building podman image: {image_name}") + testenv.cmd.run(["buildah", "build", + "--build-arg", f"DISTRO={distro}", + "-t", image_name, + os.path.join(testenv.data_dir, "podman")]) + + +def generate_env_podman(env={}): + ret = [] + + for key, val in testenv.cmd.generate_env(env, True).items(): + ret += ["-e", f"{key}={val}"] + + return ret + + +def init_image_name_distro(): + global image_name + global distro + + distro = getattr(testenv.args, "distro", testenv.distro_default) + image_name = f"{distro}-osmo-ttcn3-testenv" + image_name = image_name.replace(":","-").replace("_","-") + + +def init(): + global apt_dir_var_cache + global apt_dir_var_lib + global run_shell_on_stop + + apt_dir_var_cache = os.path.join(testenv.args.cache, "podman", "var-cache-apt") + apt_dir_var_lib = os.path.join(testenv.args.cache, "podman", "var-lib-apt") + + os.makedirs(apt_dir_var_cache, exist_ok=True) + os.makedirs(apt_dir_var_lib, exist_ok=True) + os.makedirs(testenv.args.ccache, exist_ok=True) + + init_image_name_distro() + + if not image_exists(): + raise testenv.NoTraceException("Missing podman image, run 'testenv.py" + " init podman' first to build it") + if not image_up_to_date(): + logging.warning("The podman image might be outdated, consider running" + " 'testenv.py init podman' to rebuild it") + + atexit.register(stop) + + if testenv.args.shell: + run_shell_on_stop = True + + +def exec_cmd(cmd, podman_opts=[], cwd=None, env={}, *args, **kwargs): + podman_opts = list(podman_opts) + podman_opts += generate_env_podman(env) + # Attach a fake tty (eclipse-titan won't print colored output otherwise) + podman_opts += ["-t"] + + if cwd: + podman_opts += ["-w", cwd] + + if isinstance(cmd, str): + cmd = ["sh", "-c", cmd] + + testenv.cmd.run(["podman", "exec"] + + podman_opts + + [container_name] + + cmd, + no_podman=True, + *args, **kwargs) + + +def exec_cmd_background(cmd, podman_opts=[], cwd=None, env={}): + podman_opts = list(podman_opts) + generate_env_podman(env) + + if cwd: + podman_opts += ["-w", cwd] + + if isinstance(cmd, str): + cmd = ["sh", "-c", cmd] + + cmd = ["podman", "exec"] + podman_opts + [container_name] + cmd + logging.debug(f"+ {cmd}") + + return subprocess.Popen(cmd) + + +def feed_watchdog_loop(): + # The script testenv-podman-main.sh checks every 10s for /tmp/watchdog and + # deletes the file. Create it here every 5s so the container keeps running. + # This ensures that if we run in jenkins and the job gets aborted, the + # container will terminate after a few seconds. + try: + while True: + time.sleep(5) + p = subprocess.run(["podman", "exec", container_name, "touch", + "/tmp/watchdog"]) + if p.returncode: + logging.error("podman container crashed!") + return + except KeyboardInterrupt: + pass + + +def start(): + global container_name + global feed_watchdog_process + + testdir_topdir = testenv.testdir.testdir_topdir + osmo_dev_dir = testenv.osmo_dev.get_osmo_dev_dir() + container_name = testenv.testdir.prefix + # Custom seccomp profile that allows io_uring + seccomp_profile = os.path.join(testenv.data_dir, "podman/seccomp_profile.json") + + cmd = [ + "podman", "run", + "--rm", + "--name", container_name, + "--detach", + f"--security-opt=seccomp={seccomp_profile}", + "--cap-add=NET_ADMIN", # for dumpcap, tun devices, osmo-pcap-client + "--cap-add=NET_RAW", # for dumpcap, osmo-pcap-client + "--device=/dev/net/tun", # for e.g. ggsn_tests + "--volume", f"{apt_dir_var_cache}:/var/cache/apt", + "--volume", f"{apt_dir_var_lib}:/var/lib/apt", + ] + + if not testenv.args.binary_repo: + cmd += [ + "--volume", f"{osmo_dev_dir}:{osmo_dev_dir}", + ] + + cmd += [ + "--volume", f"{testdir_topdir}:{testdir_topdir}", + "--volume", f"{testenv.args.cache}:{testenv.args.cache}", + "--volume", f"{testenv.args.ccache}:{testenv.args.ccache}", + "--volume", f"{testenv.src_dir}:{testenv.src_dir}", + image_name, + os.path.join(testenv.data_dir, "scripts/testenv-podman-main.sh") + ] + + testenv.cmd.run(cmd, no_podman=True) + + feed_watchdog_process = multiprocessing.Process(target=feed_watchdog_loop) + feed_watchdog_process.start() + + exec_cmd(["rm", "/etc/apt/apt.conf.d/docker-clean"]) + + pkgcache = os.path.join(apt_dir_var_cache, "pkgcache.bin") + if not os.path.exists(pkgcache): + exec_cmd(["apt-get", "-q", "update"]) + + +def distro_to_repo_dir(distro): + if distro == "debian:bookworm": + return "Debian_12" + raise RuntimeError(f"Can't translate distro {distro} to repo_dir!") + + +def enable_binary_repo(): + config = "deb [signed-by=/obs.key]" + config += " https://downloads.osmocom.org/packages/" + config += testenv.args.binary_repo.replace(":", ":/") + config += "/" + config += distro_to_repo_dir(distro) + config += "/ ./" + + path = "/etc/apt/sources.list.d/osmocom.list" + + exec_cmd(["sh", "-c", f"echo {shlex.quote(config)} > {path}"]) + exec_cmd(["apt-get", "-q", "update"]) + + +def is_running(): + if container_name is None: + return False + + cmd = ["podman", "ps", "-q", "--filter", f"name={container_name}"] + if not subprocess.run(cmd, capture_output=True, text=True).stdout: + return False + + return True + + +def stop(restart=False): + global container_name + global run_shell_on_stop + + if not is_running(): + return + + if not restart and run_shell_on_stop: + logging.info("Running interactive shell before stopping container (--shell)") + + # stdin=None: override stdin=/dev/null, so we can type into the shell + exec_cmd(["bash"], ["-i"], cwd=testenv.testdir.testdir, stdin=None, + check=False) + + run_shell_on_stop = False + + restart_msg = " (restart)" if restart else "" + logging.info(f"Stopping podman container{restart_msg}") + testenv.cmd.run(["podman", "kill", container_name], no_podman=True) + + if feed_watchdog_process: + feed_watchdog_process.terminate() + + if restart: + testenv.cmd.run(["podman", "wait", container_name], no_podman=True, + check=False) + + container_name = None + + if restart: + start() diff --git a/_testenv/testenv/podman_install.py b/_testenv/testenv/podman_install.py new file mode 100644 index 0000000..e1ca7c2 --- /dev/null +++ b/_testenv/testenv/podman_install.py @@ -0,0 +1,138 @@ +import logging +import multiprocessing +import os +import sys +import testenv.cmd +import testenv.podman + +git_dir = None +bb_dir = None +trxcon_dir = None +sccp_dir = None +jobs = None + + +def init(): + global git_dir + global bb_dir + global trxcon_dir + global sccp_dir + global jobs + + git_dir = os.path.join(testenv.args.cache, "git") + bb_dir = os.path.join(git_dir, "osmocom-bb") + trxcon_dir = os.path.join(bb_dir, "src/host/trxcon") + sccp_dir = os.path.join(git_dir, "libosmo-sccp") + jobs = multiprocessing.cpu_count() + 1 + + os.makedirs(git_dir, exist_ok=True) + + +def apt_install(pkgs): + if not pkgs: + return + + # Remove duplicates + pkgs = list(set(pkgs)) + + logging.info(f"Installing packages: {', '.join(pkgs)}") + testenv.cmd.run(["apt-get", + "-q", + "install", + "-y", + "--no-install-recommends"] + pkgs) + + +def clone_osmocom_bb(): + if os.path.exists(bb_dir): + logging.debug("osmocom-bb: already cloned") + return + + testenv.cmd.run(["git", + "-C", git_dir, + "clone", + "--depth", "1", + "https://gerrit.osmocom.org/osmocom-bb"]) + + +def clone_libosmo_sccp(): + if os.path.exists(sccp_dir): + logging.debug("libosmo-sccp: already cloned") + return + + testenv.cmd.run(["git", + "-C", git_dir, + "clone", + "--depth", "1", + "https://gerrit.osmocom.org/libosmo-sccp"]) + + +def from_source_trxcon(): + trxcon_in_srcdir = os.path.join(trxcon_dir, "src/trxcon") + + if not os.path.exists(trxcon_in_srcdir): + clone_osmocom_bb() + apt_install(["libosmocore-dev"]) + logging.info("Building trxcon") + testenv.cmd.run(["autoreconf", "-fi"], cwd=trxcon_dir) + testenv.cmd.run(["./configure"], cwd=trxcon_dir) + testenv.cmd.run(["make", "-j", f"{jobs}"], cwd=trxcon_dir) + + testenv.cmd.run(["ln", "-s", trxcon_in_srcdir, "/usr/local/bin/trxcon"]) + + +def from_source_sccp_demo_user(): + sccp_demo_user_path = os.path.join(sccp_dir, "examples/sccp_demo_user") + + # Install libraries even if not building sccp_demo_user, because it gets + # linked dynamically against them. + apt_install([ + "libosmo-netif-dev", + "libosmocore-dev", + ]) + + if not os.path.exists(sccp_demo_user_path): + clone_libosmo_sccp() + logging.info("Building sccp_demo_user") + testenv.cmd.run(["autoreconf", "-fi"], cwd=sccp_dir) + testenv.cmd.run(["./configure"], cwd=sccp_dir) + testenv.cmd.run(["make", "-j", f"{jobs}", "libosmo-sigtran.la"], + cwd=os.path.join(sccp_dir, "src")) + testenv.cmd.run(["make", "-j", f"{jobs}", "sccp_demo_user"], + cwd=os.path.join(sccp_dir, "examples")) + + testenv.cmd.run(["ln", "-s", sccp_demo_user_path, + "/usr/local/bin/sccp_demo_user"]) + + +def from_source(cfg, cfg_name, section): + program = cfg[section]["program"] + if program == "trxcon": + return from_source_trxcon() + if program == "run_fake_trx.sh": + return clone_osmocom_bb() + if program == "run_sccp_demo_user.sh": + return from_source_sccp_demo_user() + + logging.error(f"Can't install {section}! Fix this by either:") + logging.error(f"* Adding package= to [{section}] in {cfg_name}") + logging.error(" (if it can be installed from binary packages)") + logging.error("* Editing from_source() in testenv/podman_install.py") + sys.exit(1) + + +def packages(cfg, cfg_name): + packages = [] + + for section in cfg: + if section in ["DEFAULT", "testsuite"]: + continue + section_data = cfg[section] + if "package" in section_data: + if section_data["package"] == "no": + continue + packages += section_data["package"].split(" ") + else: + from_source(cfg, cfg_name, section) + + apt_install(packages) diff --git a/_testenv/testenv/requirements.py b/_testenv/testenv/requirements.py new file mode 100644 index 0000000..763b972 --- /dev/null +++ b/_testenv/testenv/requirements.py @@ -0,0 +1,73 @@ +import logging +import os.path +import shutil +import sys +import testenv +import testenv.cmd +import testenv.testsuite + + +def check_programs(): + programs = [ + "git", + ] + + if testenv.args.podman: + programs += [ + "buildah", + "podman", + "rsync", + ] + else: + programs += [ + "autoconf", + "automake", + "ccache", + "dumpcap", + "g++", + "gcc", + "make", + "pkg-config", + "setcap", + "ttcn3_compiler", + "wget", + ] + + abort = False + for program in programs: + if not shutil.which(program): + logging.error(f"Missing program: {program}") + + if program == "ttcn3_compiler": + logging.error(" Install eclipse-titan, e.g. from osmocom:latest:") + logging.error(" https://osmocom.org/projects/cellular-infrastructure/wiki/Latest_Builds&quo…) + abort = True + + if abort: + sys.exit(1) + + +def check_fftranscode(): + cmd = ["grep", "-q", "fftranscode", + os.path.join(testenv.testsuite.ttcn3_hacks_dir_src, + testenv.args.testsuite, + "regen_makefile.sh")] + if testenv.cmd.run(cmd, check=False).returncode == 1: + return + + cmd = ["pkg-config", "--modversion", "libfftranscode"] + if testenv.cmd.run(cmd, check=False).returncode == 0: + return + + logging.error("Missing library: libfftranscode") + logging.error(" https://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Tests…) + logging.error(" Consider installing it from here:") + logging.error(" https://ftp.osmocom.org/binaries/libfftranscode/") + sys.exit(1) + + +def check(): + check_programs() + + if not testenv.args.podman: + check_fftranscode() diff --git a/_testenv/testenv/testdir.py b/_testenv/testenv/testdir.py new file mode 100644 index 0000000..ffc6fd0 --- /dev/null +++ b/_testenv/testenv/testdir.py @@ -0,0 +1,170 @@ +import atexit +import datetime +import glob +import logging +import os +import os.path +import tempfile +import testenv +import testenv.cmd +import testenv.testsuite +import uuid + + +# Some testsuites have multiple configurations (like bts: generic, hopping, …). +# For each configuration, prepare() gets called. testdir is the one for the +# current configuration. +testdir = None +testdir_topdir = None +prefix = None +clean_scripts = {} + + +def init(): + global testdir_topdir + global prefix + + prefix = f"testenv-{testenv.args.testsuite}-" + if testenv.args.config: + prefix += f"{'-'.join(testenv.args.config)}-" + if testenv.args.binary_repo: + prefix += f"{testenv.args.binary_repo.replace(':','-')}-" + prefix += datetime.datetime.now().strftime("%Y%m%d-%H%M") + prefix += f"-{str(uuid.uuid4()).split('-', 1)[0]}" + + if testenv.args.log_dir: + testdir_topdir = testenv.args.log_dir + os.makedirs(testdir_topdir) + else: + testdir_topdir = tempfile.mkdtemp(prefix=f"{prefix}-") + + atexit.register(clean) + + logging.info(f"Logging to: {testdir_topdir}") + + # Add a convenience symlink + if not testenv.args.log_dir: + if os.path.exists("/tmp/logs"): + testenv.cmd.run(["rm", "/tmp/logs"], no_podman=True) + testenv.cmd.run(["ln", "-sf", testdir_topdir, "/tmp/logs"], + no_podman=True) + + +def prepare(cfg_name, cfg): + global testdir + global clean_scripts + + if len(testenv.testenv_cfg.cfgs) == 1: + testdir = testdir_topdir + else: + testdir = os.path.join(testdir_topdir, + cfg_name.replace("testenv_", "").replace(".cfg", "")) + + logging.info(f"Preparing testdir: {testdir}") + testsuite_dir = os.path.join(testenv.testsuite.ttcn3_hacks_dir, + testenv.args.testsuite) + + atexit.register(clean_run_scripts) + + for section in cfg: + if section in ["DEFAULT"]: + continue + + section_data = cfg[section] + section_dir = os.path.join(testdir, section) + os.makedirs(section_dir) + + if "config" in section_data and section == "testsuite": + file = section_data["config"] + path = os.path.join(testsuite_dir, file) + path_dest = os.path.join(section_dir, file) + testenv.cmd.run(["install", "-Dm644", path, path_dest]) + + if "copy" in section_data: + for file in section_data["copy"].split(" "): + path = os.path.join(testsuite_dir, file) + path_dest = os.path.join(section_dir, file) + mode = 755 if os.access(path, os.X_OK) else 644 + testenv.cmd.run(["install", f"-Dm{mode}", path, path_dest]) + + if "clean" in section_data: + logging.info(f"Running {section} clean script (reason: prepare)") + clean_scripts[section] = section_data["clean"] + env={"TESTENV_CLEAN_REASON": "prepare"} + testenv.cmd.run(section_data["clean"], cwd=section_dir, env=env) + + if "prepare" in section_data: + logging.info(f"Running {section} prepare script") + testenv.cmd.run(section_data["prepare"], cwd=section_dir) + + + # Referenced in testsuite cfgs: *.default + pattern = os.path.join(testsuite_dir, "*.default") + for path in glob.glob(pattern): + path_dest = os.path.join(testdir, "testsuite", + os.path.basename(path)) + testenv.cmd.run(["install", "-Dm644", path, path_dest]) + + # Referenced in testsuite cfgs: Common.cfg + common_cfg = os.path.join(testdir, "testsuite", "Common.cfg") + path = os.path.join(testenv.testsuite.ttcn3_hacks_dir, "Common.cfg") + testenv.cmd.run(["install", "-Dm644", path, common_cfg]) + testenv.cmd.run(["sed", "-i", + f"s#TTCN3_HACKS_PATH := .*#TTCN3_HACKS_PATH := \"{testenv.testsuite.ttcn3_hacks_dir}\"#", + common_cfg]) + + # Adjust testsuite config: set mp_osmo_repo, set Common.cfg path in the + # testsuite's config and in all configs it may include + mp_osmo_repo = "latest" if testenv.args.latest else "nightly" + line = f"Misc_Helpers.mp_osmo_repo := \"{mp_osmo_repo}\"" + + patterns = [ + os.path.join(testdir, "testsuite/**/*.cfg"), + os.path.join(testdir, "testsuite/**/*.default"), + ] + for pattern in patterns: + for cfg_file in glob.glob(pattern, recursive=True): + logging.debug(f"Adjusting testsuite config: {cfg_file}") + testenv.cmd.run(["sed", "-i", + "-e", f"s/\\[MODULE_PARAMETERS\\]/\\[MODULE_PARAMETERS\\]\\n{line}/g", + "-e", f"s#../Common.cfg#Common.cfg#", + cfg_file]) + + +def clean(): + """Don't leave behind an empty testdir_topdir, e.g. if testenv.py aborted + during build of components.""" + # Show log dir path/link if it isn't empty + if os.listdir(testdir_topdir): + msg = "Logs saved to:" + + url = os.environ.get("BUILD_URL") # Jenkins sets this + if url: + # Add a space at the end, so jenkins can transform this into a link + # without adding the color reset escape code to it + msg += f" {url}artifact/logs/ " + else: + msg += f" {testdir_topdir}" + if not testenv.args.log_dir: + msg += " (symlink: /tmp/logs)" + logging.info(msg) + return + + logging.debug("Remving empty log dir") + testenv.cmd.run(["rm", "-d", testdir_topdir], no_podman=True) + + # Remove broken symlink + if not testenv.args.log_dir and os.path.lexists("/tmp/logs") \ + and not os.path.exists("/tmp/logs"): + testenv.cmd.run(["rm", "/tmp/logs"], no_podman=True) + + +def clean_run_scripts(reason="crashed"): + global clean_scripts + + for section, script in clean_scripts.items(): + logging.info(f"Running {section} clean script (reason: {reason})") + env={"TESTENV_CLEAN_REASON": reason} + testenv.cmd.run(script, cwd=os.path.join(testdir, section), + env=env) + clean_scripts = {} diff --git a/_testenv/testenv/testenv_cfg.py b/_testenv/testenv/testenv_cfg.py new file mode 100644 index 0000000..948bb80 --- /dev/null +++ b/_testenv/testenv/testenv_cfg.py @@ -0,0 +1,193 @@ +import configparser +import glob +import logging +import os.path +import sys +import testenv +import testenv.testsuite + +cfgs = {} +current = None + + +def set_current(cfg_name): + global current + current = cfg_name + + if cfg_name == "testenv.cfg": + testenv.set_log_prefix("[testenv]") + else: + cfg_name = cfg_name.replace("testenv_", "") + cfg_name = cfg_name.replace(".cfg", "") + testenv.set_log_prefix(f"[testenv][{cfg_name}]") + + +def exit_error_readme(): + readme = os.path.join(testenv.testsuite.ttcn3_hacks_dir_src, + "_testenv/README.md") + logging.error(f"More information: {readme}") + sys.exit(1) + + +def handle_latest(cfg, path): + """Remove _latest keys from cfg or use them instead of the regular keys, + if --latest is set.""" + + for section in cfg: + for key in cfg[section]: + if not key.endswith("_latest"): + continue + + if testenv.args.latest: + key_regular = key.replace("_latest", "") + logging.debug(f"{path}: [{section}]: using {key} instead of" + f" {key_regular} (--latest is set)") + cfg[section][key_regular] = cfg[section][key] + else: + logging.debug(f"{path}: [{section}]: ignoring {key} (--latest" + " is not set)") + + del cfg[section][key] + + +def verify(cfg, path): + keys_valid_testsuite = [ + "clean", + "config", + "copy", + "program", + ] + keys_valid_component = [ + "clean", + "copy", + "make", + "package", + "prepare", + "program", + "setup", + ] + keys_invalid = { + "configs": "config", + "packages": "package", + "programs": "program", + } + + if "testsuite" not in cfg: + logging.error(f"{path}: missing [testsuite] section") + exit_error_readme() + if "program" not in cfg["testsuite"]: + logging.error(f"{path}: missing program= in [testsuite]") + exit_error_readme() + if " " in cfg["testsuite"]["program"]: + logging.error(f"{path}: program= in [testsuite] must not have arguments") + exit_error_readme() + if " " in cfg["testsuite"]["config"]: + logging.error(f"{path}: config= in [testsuite] must not have spaces") + exit_error_readme() + if "config" not in cfg["testsuite"]: + logging.error(f"{path}: missing config= in [testsuite]") + exit_error_readme() + + for section in cfg: + for key in cfg[section].keys(): + valid = keys_valid_component + if section == "testsuite": + valid = keys_valid_testsuite + + if key in valid: + continue + + msg = f"{path}: [{section}]: {key}= is invalid" + if key in keys_invalid and keys_invalid[key] in valid: + msg += f", did you mean {keys_invalid[key]}=?" + + logging.error(msg) + exit_error_readme() + + if section not in ["DEFAULT", "testsuite"] and "make" not in cfg[section]: + logging.error(f"{path}: missing make= in section [{section}].") + logging.error("If this is on purpose, set make=no.") + exit_error_readme() + + +def raise_error_config_arg(glob_result): + valid = [] + for path in glob_result: + basename = os.path.basename(path) + if basename != "testenv.cfg": + valid += [basename.split("_", 1)[1].split(".", -1)[0]] + + msg = f"Invalid parameter for --config: {testenv.args.config}" + + if valid: + msg += f" (valid: all, {', '.join(valid)})" + else: + msg += (f" (the {testenv.args.testsuite} testsuite only has one" + " testenv.cfg file, therefore just omit --config)") + + raise testenv.NoTraceException(msg) + + +def find_configs(): + dir_testsuite = os.path.join(testenv.testsuite.ttcn3_hacks_dir_src, + testenv.args.testsuite) + pattern = os.path.join(dir_testsuite, "testenv*.cfg") + ret = glob.glob(pattern) + + if not ret: + logging.error(f"Missing testenv.cfg in: {dir_testsuite}") + exit_error_readme() + sys.exit(1) + + if len(ret) > 1 and os.path.exists(os.path.join(dir_testsuite, "testenv.cfg")): + logging.error("Found multiple testenv*.cfg, and a testenv.cfg.") + logging.error("The testenv.cfg file must be renamed, consider naming" + " it testenv_generic.cfg.") + sys.exit(1) + + if len(ret) == 1 and not os.path.exists(os.path.join(dir_testsuite, "testenv.cfg")): + logging.error("There is only one testenv*.cfg file, so please rename it:") + logging.error(f"$ mv {os.path.basename(ret[0])} testenv.cfg") + sys.exit(1) + + if len(ret) > 1 and not testenv.args.config: + logging.error("Found multiple testenv.cfg files:") + for path in ret: + logging.error(f" * {os.path.basename(path)}") + example = os.path.basename(ret[0]).split("_", 1)[1].split(".cfg", 1)[0] + logging.error(f"Select a specific config (e.g. '-c {example}')" + " or all ('-c all')") + sys.exit(1) + + return ret + + +def init(): + global cfgs + + config_paths = find_configs() + + for path in config_paths: + basename = os.path.basename(path) + if basename != "testenv.cfg" and not basename.startswith("testenv_"): + raise testenv.NoTraceException("Invalid filename, expected either" + f" testenv.cfg or testenv_*.cfg: {cfg}") + if basename == "testenv_all.cfg": + raise testenv.NoTraceException(f"Invalid filename: {cfg}") + + cfg = configparser.ConfigParser() + cfg.read(path) + handle_latest(cfg, path) + verify(cfg, path) + + if not testenv.args.config: + cfgs[basename] = cfg + continue + + for config_arg in testenv.args.config: + if config_arg == "all" or f"testenv_{config_arg}.cfg" == basename: + cfgs[basename] = cfg + break + + if not cfgs: + raise_error_config_arg(config_paths) diff --git a/_testenv/testenv/testsuite.py b/_testenv/testenv/testsuite.py new file mode 100644 index 0000000..daca11d --- /dev/null +++ b/_testenv/testenv/testsuite.py @@ -0,0 +1,215 @@ +import atexit +import glob +import logging +import multiprocessing +import os +import os.path +import shlex +import shutil +import subprocess +import sys +import testenv +import testenv.cmd +import time + +ttcn3_hacks_dir = None +ttcn3_hacks_dir_src = os.path.realpath(f"{__file__}/../../..") +testsuite_proc = None + + +def update_deps(): + deps_marker = os.path.join(testenv.args.cache, "ttcn3-deps-updated") + if os.path.exists(deps_marker): + return + + logging.info("Updating osmo-ttcn3-hacks/deps") + deps_dir = os.path.join(ttcn3_hacks_dir_src, "deps") + testenv.cmd.run(["make", "-C", deps_dir]) + testenv.cmd.run(["touch", deps_marker]) + + +def copy_ttcn3_hacks_dir(): + """Copy source files of osmo-ttcn3-hacks.git to the cache dir, so we don't + mix binary objects from host and inside podman that are very likely to + be incompatible""" + global ttcn3_hacks_dir + + ttcn3_hacks_dir = os.path.join(testenv.args.cache, + "podman", + "osmo-ttcn3-hacks") + + logging.info(f"Copying osmo-ttcn3-hacks sources to: {ttcn3_hacks_dir}") + + # Rsync can't directly parse the .gitignore with ! rules, so create a list + # of files to be copied with git + copy_list = os.path.join(testenv.args.cache, "podman", "ttcn3-copy-list") + testenv.cmd.run("git" + " ls-files" + " -o" + " -c" + " --exclude-standard" + f" > {shlex.quote(copy_list)}", + cwd=ttcn3_hacks_dir_src, + no_podman=True) + + # Copy source files, excluding binary objects + testenv.cmd.run(["rsync", + "--links", + "--recursive", + f"--files-from={copy_list}", + f"{ttcn3_hacks_dir_src}/", + f"{ttcn3_hacks_dir}/"], + no_podman=True) + + # The "deps" dir is in gitignore, copy it separately + testenv.cmd.run(["rsync", + "--links", + "--recursive", + "--exclude", "/.git", + f"{ttcn3_hacks_dir_src}/deps/", + f"{ttcn3_hacks_dir}/deps/"], + no_podman=True) + + +def prepare_testsuite_dir(): + testsuite_dir = f"{ttcn3_hacks_dir}/{testenv.args.testsuite}" + logging.info(f"Generating links and Makefile for {testenv.args.testsuite}") + testenv.cmd.run(["./gen_links.sh"], cwd=testsuite_dir) + testenv.cmd.run("USE_CCACHE=1 ./regen_makefile.sh", cwd=testsuite_dir) + + +def init(): + global ttcn3_hacks_dir + + atexit.register(stop) + + update_deps() + + if testenv.args.podman: + copy_ttcn3_hacks_dir() + else: + ttcn3_hacks_dir = ttcn3_hacks_dir_src + + prepare_testsuite_dir() + + +def build(): + logging.info("Building testsuite") + testsuite_dir = f"{ttcn3_hacks_dir}/{testenv.args.testsuite}" + testenv.cmd.run(["make", "compile"], cwd=testsuite_dir) + + jobs = multiprocessing.cpu_count() + 1 + testenv.cmd.run(["make", "-j", f"{jobs}"], cwd=testsuite_dir) + + +def is_running(pid): + # Check if a process is still running, or if it is dead / a zombie. We + # can't just use proc.poll() because this gets called from another thread. + cmdline = f"/proc/{pid}/cmdline" + if not os.path.exists(cmdline): + return False + + # The cmdline file is empty if it is a zombie + with open(cmdline) as f: + return f.read() != "" + + +def merge_log_files(cfg): + section_data = cfg["testsuite"] + cwd = os.path.join(testenv.testdir.testdir, "testsuite") + + logging.info("Merging log files") + log_merge = os.path.join(ttcn3_hacks_dir, "log_merge.sh") + # stdout of this script is very verbose, making it harder to see the output + # that matters (tests failed or not), so redirect it to /dev/null + cmd = f"{shlex.quote(log_merge)} {shlex.quote(section_data['program'])} --rm >/dev/null" + testenv.cmd.run(cmd, cwd=cwd) + + +def format_log_files(cfg): + section_data = cfg["testsuite"] + cwd = os.path.join(testenv.testdir.testdir, "testsuite") + + logging.info("Formatting log files") + cmd = os.path.join(testenv.data_dir, "scripts/log_format.sh") + testenv.cmd.run(cmd, cwd=cwd) + + +def cat_junit_logs(): + tool = "cat" + + if testenv.args.podman or shutil.which("source-highlight"): + colors = os.environ.get("TESTENV_SOURCE_HIGHLIGHT_COLORS", "esc256") + tool = f"source-highlight -f {shlex.quote(colors)} -s xml -i" + + pattern = os.path.join(testenv.testdir.testdir, "testsuite", "junit-*.log") + for path in glob.glob(pattern): + cmd = f"echo && {tool} {shlex.quote(path)} && echo" + logging.info(f"Showing {os.path.basename(path)}") + testenv.cmd.run(cmd) + + +def run(cfg): + global testsuite_proc + + section_data = cfg["testsuite"] + + cwd = os.path.join(testenv.testdir.testdir, "testsuite") + start_testsuite = os.path.join(ttcn3_hacks_dir, "start-testsuite.sh") + suite = os.path.join(ttcn3_hacks_dir, testenv.args.testsuite, + section_data["program"]) + + env = { + "TTCN3_PCAP_PATH": os.path.join(testenv.testdir.testdir, "testsuite"), + } + + cmd = [start_testsuite, suite, section_data["config"]] + + test_arg = testenv.args.test + if test_arg: + if "." in test_arg: + cmd += [test_arg] + else: + cmd += [f"{section_data['program']}.{test_arg}"] + + logging.info("Running testsuite") + + if testenv.podman.is_running(): + testsuite_proc = testenv.podman.exec_cmd_background(cmd, cwd=cwd, env=env) + else: + logging.debug(f"+ {cmd}") + testsuite_proc = subprocess.Popen(cmd, cwd=cwd, env=env) + + # Ensure all daemons run until the testsuite stops + while True: + time.sleep(1) + + if not is_running(testsuite_proc.pid): + logging.debug("Testsuite is done") + stop() + break + + for daemon_name, daemon_proc in testenv.daemons.daemons.items(): + if not is_running(daemon_proc.pid): + raise testenv.NoTraceException(f"{daemon_name} crashed!") + + merge_log_files(cfg) + format_log_files(cfg) + + +def run_prepare_script(cfg): + section_data = cfg["testsuite"] + if "prepare" not in section_data: + return + + logging.info("Running testsuite prepare script") + testenv.cmd.run(section_data["prepare"]) + + +def stop(): + global testsuite_proc + + if testsuite_proc: + logging.info(f"Stopping testsuite ({testsuite_proc.pid})") + testenv.daemons.kill(testsuite_proc.pid) + testsuite_proc = None diff --git a/testenv.py b/testenv.py new file mode 120000 index 0000000..36c795e --- /dev/null +++ b/testenv.py @@ -0,0 +1 @@ +_testenv/testenv.py \ No newline at end of file -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/37694?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: If9f8b79dd6e5b4f06be4e5ff73db97759c3acfb2 Gerrit-Change-Number: 37694 Gerrit-PatchSet: 1 Gerrit-Owner: osmith &lt;osmith(a)sysmocom.de&gt; Gerrit-MessageType: newchange