[L] Change in osmo-ttcn3-hacks[master]: testenv: support running SUT in QEMU

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/38343?usp=email ) Change subject: testenv: support running SUT in QEMU ...................................................................... testenv: support running SUT in QEMU Add two new arguments -C|--custom-kernel and -D|--debian-kernel. If any of these is set, pass an environment variable TESTENV_QEMU_KERNEL with the path to the kernel when running commands from testenv.cfg. These commands can then source the new qemu_functions.sh and use it to build an initramfs with the SUT and depending libraries on the fly, and start up QEMU to boot right to starting the SUT. All of that takes about ~1s on my system with kvm. Without kvm ~5s. A follow-up patch will adjust the ggsn testenv configs to optionally run osmo-ggsn in QEMU for testing kernel GTP-U. These scripts are based on scripts/kernel-tests from docker-playground. Related: osmo-ci Id64a1a778fa38eec20498c36b390332f75d7d3f5 Change-Id: Ic9cb7092fd029b7ba530fc755b5d4d73a9d86350 --- M .gitignore M _testenv/README.md M _testenv/data/podman/Dockerfile A _testenv/data/scripts/qemu/qemu_functions.sh A _testenv/data/scripts/qemu/qemu_ifup.sh A _testenv/data/scripts/qemu/qemu_init.sh A _testenv/data/scripts/qemu/qemu_wait.sh M _testenv/testenv/__init__.py M _testenv/testenv/cmd.py M _testenv/testenv/podman.py M _testenv/testenv/requirements.py M _testenv/testenv/testenv_cfg.py 12 files changed, 400 insertions(+), 1 deletion(-) Approvals: pespin: Looks good to me, but someone else must approve laforge: Looks good to me, but someone else must approve fixeria: Looks good to me, approved Jenkins Builder: Verified diff --git a/.gitignore b/.gitignore index 241a1d2..94061e3 100644 --- a/.gitignore +++ b/.gitignore @@ -25,3 +25,4 @@ sms.db-wal __pycache__ tags +.linux diff --git a/_testenv/README.md b/_testenv/README.md index 7db3e24..3fed9c9 100644 --- a/_testenv/README.md +++ b/_testenv/README.md @@ -20,6 +20,24 @@ Additional packages get installed after starting the container, with a package cache mounted to avoid unnecessary downloads. +### QEMU + +For SUTs that interact with kernel drivers, it is desirable to run them with a +separate kernel in QEMU. This can be enabled by running `./testenv.py run` with +the `-D` (Debian kernel) or `-C` (custom kernel) arguments. See the `ggsn` +testsuite for reference. + +#### Custom kernel + +When using `-C`, testenv uses a `.linux` file in the `osmo-ttcn3-hacks` dir as +kernel. You can download a pre-built kernel from jenkins: + +$ wget -O .linux "https://jenkins.osmocom.org/jenkins/job/build-kernel-torvalds/lastSuccessfulBuild/artifact/output/linux" +$ wget -O .linux "https://jenkins.osmocom.org/jenkins/job/build-kernel-net-next/lastSuccessfulBuild/artifact/output/linux" + +Or build your own kernel, see: +https://gitea.osmocom.org/osmocom/osmo-ci/src/branch/master/scripts/kernel + ## testenv.cfg The `testenv.cfg` file has one `[testsuite]` section, typically with one or @@ -98,6 +116,11 @@ * `vty_host=`: optionally set the VTY host for the SUT component to be used when obtaining a talloc report. If this is not set, `127.0.0.1` is used. +* `qemu=`: set to `optional` to allow running this test component in QEMU. + Additional logic must be added to build an initrd with the test component and + actually run it in QEMU, this is done by sourcing `qemu_functions.sh` and + using functions from there. See the `ggsn` testsuite for reference. + ### Executables #### $PATH @@ -108,6 +131,7 @@ * The directory of the testsuite * The directory for binaries built from source * The directory `_testenv/data/scripts` (which has e.g. `respawn.sh`) +* The directory `_testenv/data/scripts/qemu` #### $PWD (current working dir) @@ -180,6 +204,11 @@ files. Jenkins sets this to `esc` instead of `esc256` for better contrast on white background. +* `TESTENV_NO_KVM`: + Do not mount /dev/kvm in podman. This is used in jenkins where /dev/kvm is + available but doesn't work in podman. QEMU runs a bit slower when this is + set. + ## Troubleshooting ### Timeout waiting for RESET-ACK after sending RESET diff --git a/_testenv/data/podman/Dockerfile b/_testenv/data/podman/Dockerfile index 13412a2..6f6c739 100644 --- a/_testenv/data/podman/Dockerfile +++ b/_testenv/data/podman/Dockerfile @@ -12,20 +12,29 @@ # Install packages from Debian repositories (alphabetic order) ENV DEBIAN_FRONTEND=noninteractive RUN set -x && \ + mkdir -p /etc/kernel/postinst.d && \ + touch /etc/kernel/postinst.d/initramfs-tools && \ apt-get update && \ - apt-get install -y --no-install-recommends \ + apt-get install \ + -y \ + --no-install-recommends \ + -o Dpkg::Options::="--force-confold" \ autoconf \ automake \ + bc \ bison \ bridge-utils \ build-essential \ + busybox-static \ ca-certificates \ ccache \ cmake \ + cpio \ erlang-nox \ flex \ gdb \ git \ + gzip \ iproute2 \ iputils-ping \ libbson-dev \ @@ -34,6 +43,8 @@ libcurl4-gnutls-dev \ libdbd-sqlite3 \ libdbi-dev \ + libelf-dev \ + libgcc-11-dev \ libgcrypt-dev \ libgnutls28-dev \ libidn11-dev \ @@ -57,13 +68,16 @@ libusb-1.0-0-dev \ libyaml-dev \ libzmq3-dev \ + linux-image-amd64 \ meson \ netcat-openbsd \ + pax-utils \ pcscd \ pkg-config \ procps \ psmisc \ python3-pip \ + qemu-system-x86 \ rsync \ source-highlight \ sqlite3 \ diff --git a/_testenv/data/scripts/qemu/qemu_functions.sh b/_testenv/data/scripts/qemu/qemu_functions.sh new file mode 100755 index 0000000..c0dd8cc --- /dev/null +++ b/_testenv/data/scripts/qemu/qemu_functions.sh @@ -0,0 +1,191 @@ +#!/bin/sh -ex +INITRD_DIR="$PWD/_initrd" + +# Add one or more files to the initramfs, with parent directories. +# usr-merge: resolve symlinks for /lib -> /usr/lib etc. so "cp --parents" does +# not fail with "cp: cannot make directory '/tmp/initrd/lib': File exists" +# $@: path to files +qemu_initrd_add_file() { + local i + + for i in "$@"; do + case "$i" in + /bin/*|/sbin/*|/lib/*|/lib64/*) + cp -a --parents "$i" "$INITRD_DIR"/usr + ;; + *) + cp -a --parents "$i" "$INITRD_DIR" + ;; + esac + done +} + +# Add kernel module files with dependencies +# $@: kernel module names +qemu_initrd_add_mod() { + if [ "$TESTENV_QEMU_KERNEL" != "debian" ]; then + # Assume all drivers are statically built into the kernel, when + # using a custom kernel + return + fi + + local kernel="$(basename /lib/modules/*)" + local files="$(modprobe \ + -a \ + --dry-run \ + --show-depends \ + --set-version="$kernel" \ + "$@" \ + | sort -u \ + | cut -d ' ' -f 2)" + + qemu_initrd_add_file $files + + # Save the list of modules + for i in $@; do + echo "$i" >> "$INITRD_DIR"/modules + done +} + +# Add binaries with depending libraries +# $@: paths to binaries +qemu_initrd_add_bin() { + local bin + local bin_path + local file + + for bin in "$@"; do + local bin_path="$(which "$bin")" + if [ -z "$bin_path" ]; then + echo "ERROR: file not found: $bin" + exit 1 + fi + + lddtree_out="$(lddtree -l "$bin_path")" + if [ -z "$lddtree_out" ]; then + echo "ERROR: lddtree failed on '$bin_path'" + exit 1 + fi + + for file in $lddtree_out; do + qemu_initrd_add_file "$file" + + # Copy resolved symlink + if [ -L "$file" ]; then + qemu_initrd_add_file "$(realpath "$file")" + fi + done + done +} + +# Add command to run inside the initramfs +# $@: commands +qemu_initrd_add_cmd() { + local i + + if ! [ -e "$INITRD_DIR"/cmd.sh ]; then + echo "#!/bin/sh -ex" > "$INITRD_DIR"/cmd.sh + chmod +x "$INITRD_DIR"/cmd.sh + fi + + for i in "$@"; do + echo "$i" >> "$INITRD_DIR"/cmd.sh + done +} + +qemu_initrd_init() { + mkdir "$INITRD_DIR" + + for dir in bin sbin lib lib64; do + ln -s usr/"$dir" "$INITRD_DIR"/"$dir" + done + + mkdir -p \ + "$INITRD_DIR"/dev/net \ + "$INITRD_DIR"/proc \ + "$INITRD_DIR"/sys \ + "$INITRD_DIR"/tmp \ + "$INITRD_DIR"/usr/bin \ + "$INITRD_DIR"/usr/sbin + + qemu_initrd_add_bin \ + busybox + + qemu_initrd_add_cmd \ + "export PATH='$PATH:\$PATH'" + + if [ "$TESTENV_QEMU_KERNEL" = "debian" ]; then + qemu_initrd_add_mod \ + virtio_net \ + virtio_pci + + qemu_initrd_add_file \ + /lib/modules/*/modules.dep + fi +} + +qemu_initrd_finish() { + cp "$TESTENV_QEMU_SCRIPTS"/qemu_init.sh "$INITRD_DIR"/init + + tree -a "$INITRD_DIR" + + ( cd "$INITRD_DIR"; find . -print0 \ + | cpio --quiet -o -0 -H newc \ + | gzip -1 > "$INITRD_DIR".gz ) +} + +qemu_initrd_exit_error() { + # Building the initrd is quite verbose, therefore put it in a log file + # and only output its contents on error (see e.g. osmo-ggsn/run.sh) + cat "$1" + set +x + echo + echo "ERROR: failed to build the initrd!" + echo + exit 1 +} + +qemu_random_mac() { + printf "52:54:" + date "+%c %N" | sha1sum | sed 's/\(.\{2\}\)/\1:/g' | cut -d: -f 1-4 +} + +qemu_run() { + local machine_arg + local kernel="$TESTENV_QEMU_KERNEL" + local kernel_cmdline=" + root=/dev/ram0 + console=ttyS0 + panic=-1 + init=/init + loglevel=8 + " + + if [ "$kernel" = "debian" ]; then + kernel="$(ls -1 /boot/vmlinuz*)" + fi + + if [ -e /dev/kvm ]; then + machine_arg="-machine pc,accel=kvm" + else + machine_arg="-machine pc" + fi + + # sudo is required to set up networking in qemu_ifup.sh + # </dev/null to deatch stdin, so qemu doesn't capture ^C + sudo sh -c " + qemu-system-x86_64 \ + $machine_arg \ + -smp 1 \ + -m 512M \ + -no-user-config -nodefaults -display none \ + -no-reboot \ + -kernel '$kernel' \ + -initrd '$INITRD_DIR.gz' \ + -append '$kernel_cmdline' \ + -serial stdio \ + -netdev 'tap,id=nettest,script=$TESTENV_QEMU_SCRIPTS/qemu_ifup.sh' \ + -device 'virtio-net-pci,netdev=nettest,mac=$(qemu_random_mac)' \ + </dev/null + " +} diff --git a/_testenv/data/scripts/qemu/qemu_ifup.sh b/_testenv/data/scripts/qemu/qemu_ifup.sh new file mode 100755 index 0000000..370bea5 --- /dev/null +++ b/_testenv/data/scripts/qemu/qemu_ifup.sh @@ -0,0 +1,10 @@ +#!/bin/sh -e +br="testenv0" +qemu_if="$1" + +echo "qemu_ifup.sh: $br, $qemu_if" +set -x + +ip link set "$qemu_if" up + +brctl addif "$br" "$qemu_if" diff --git a/_testenv/data/scripts/qemu/qemu_init.sh b/_testenv/data/scripts/qemu/qemu_init.sh new file mode 100755 index 0000000..1c8b39b --- /dev/null +++ b/_testenv/data/scripts/qemu/qemu_init.sh @@ -0,0 +1,36 @@ +#!/bin/busybox sh +echo "Running initrd-init.sh" +set -ex + +export HOME=/root +export LD_LIBRARY_PATH=/usr/local/lib +export PATH=/usr/local/bin:/usr/bin:/bin:/sbin:/usr/local/sbin:/usr/sbin +export TERM=screen + +/bin/busybox --install -s + +mknod /dev/null c 1 3 + +# Required for osmo-ggsn +mknod /dev/net/tun c 10 200 + +hostname qemu + +mount -t proc proc /proc +mount -t sysfs sys /sys + +# Load modules from qemu_initrd_add_mod() +if [ -e /modules ]; then + cat /modules | xargs -t -n1 modprobe +fi + +ip link set lo up +ip link set eth0 up + +echo "KERNEL_TEST_VM_IS_READY" + +# Use '|| true' to avoid "attempting to kill init" kernel panic on failure +/cmd.sh || true + +# Avoid kernel panic when init exits +poweroff -f diff --git a/_testenv/data/scripts/qemu/qemu_wait.sh b/_testenv/data/scripts/qemu/qemu_wait.sh new file mode 100755 index 0000000..fb45c7e --- /dev/null +++ b/_testenv/data/scripts/qemu/qemu_wait.sh @@ -0,0 +1,22 @@ +#!/bin/sh -e +if [ -z "$TESTENV_QEMU_KERNEL" ]; then + exit 0 +fi + +LOGFILE="$(basename "$PWD").log" + +i=0 +for i in $(seq 1 600); do + sleep 0.1 + if grep -q KERNEL_TEST_VM_IS_READY "$LOGFILE"; then + # Wait some more for SUT to become ready + sleep 1 + exit 0 + fi +done + +echo +echo "ERROR: Timeout while waiting for QEMU to become ready" +echo + +exit 1 diff --git a/_testenv/testenv/__init__.py b/_testenv/testenv/__init__.py index 2af83ed..eb687f3 100644 --- a/_testenv/testenv/__init__.py +++ b/_testenv/testenv/__init__.py @@ -13,6 +13,7 @@ src_dir = os.environ.get("TESTENV_SRC_DIR", os.path.realpath(f"{__file__}/../../../..")) data_dir = os.path.join(os.path.realpath(f"{__file__}/../.."), "data") +custom_kernel_path = os.path.join(os.path.realpath(f"{__file__}/../../.."), ".linux") distro_default = "debian:bookworm" cache_dir_default = os.path.join(os.path.expanduser("~/.cache"), "osmo-ttcn3-testenv") ccache_dir_default = os.path.join(cache_dir_default, "ccache") @@ -91,6 +92,29 @@ metavar="OBS_PROJECT", help="use binary packages from this Osmocom OBS project instead (e.g. osmocom:nightly)", ) + + group = sub_run.add_argument_group( + "QEMU options", + "For some tests, the SUT can or must run in QEMU, typically to use kernel GTP-U.", + ) + group = group.add_mutually_exclusive_group() + group.add_argument( + "-D", + "--debian-kernel", + action="store_const", + dest="kernel", + const="debian", + help="run SUT in QEMU with debian kernel", + ) + group.add_argument( + "-C", + "--custom-kernel", + action="store_const", + dest="kernel", + const="custom", + help=f"run SUT in QEMU with custom kernel ({custom_kernel_path})", + ) + group = sub_run.add_argument_group( "config file options", "Testsuite and test component configs" @@ -159,6 +183,15 @@ if args.binary_repo and not args.podman: raise NoTraceException("--binary-repo requires --podman") + if args.kernel == "debian" and not args.podman: + raise NoTraceException("--kernel-debian requires --podman") + + if args.kernel == "custom" and not os.path.exists(custom_kernel_path): + logging.critical( + "See _testenv/README.md for more information on downloading a pre-built kernel or building your own kernel." + ) + raise NoTraceException(f"For --kernel-custom, put a symlink or copy of your kernel to: {custom_kernel_path}") + ttcn3_hacks_dir_src = os.path.realpath(f"{__file__}/../../..") testsuite_dir = os.path.join(ttcn3_hacks_dir_src, args.testsuite) if not os.path.exists(testsuite_dir): diff --git a/_testenv/testenv/cmd.py b/_testenv/testenv/cmd.py index 6f502fc..5e651bd 100644 --- a/_testenv/testenv/cmd.py +++ b/_testenv/testenv/cmd.py @@ -49,6 +49,13 @@ else: env_extra["OSMO_DEV_MAKE_DIR"] = os.path.join(testenv.args.cache, "host", "make") + if testenv.args.kernel == "debian": + env_extra["TESTENV_QEMU_KERNEL"] = "debian" + elif testenv.args.kernel == "custom": + env_extra["TESTENV_QEMU_KERNEL"] = testenv.custom_kernel_path + if testenv.args.kernel: + env_extra["TESTENV_QEMU_SCRIPTS"] = os.path.join(testenv.data_dir, "scripts/qemu") + def exit_error_cmd(completed, error_msg): """:param completed: return from run_cmd() below""" @@ -62,6 +69,7 @@ def generate_env(env={}, podman=False): ret = dict(env_extra) path = os.path.join(testenv.data_dir, "scripts") + path += f":{os.path.join(testenv.data_dir, 'scripts/qemu')}" if testenv.testsuite.ttcn3_hacks_dir: path += f":{os.path.join(testenv.testsuite.ttcn3_hacks_dir, testenv.args.testsuite)}" diff --git a/_testenv/testenv/podman.py b/_testenv/testenv/podman.py index 60aecca..bd1e8d8 100644 --- a/_testenv/testenv/podman.py +++ b/_testenv/testenv/podman.py @@ -221,6 +221,13 @@ f"{osmo_dev_dir}:{osmo_dev_dir}", ] + if testenv.args.kernel: + if not os.environ.get("TESTENV_NO_KVM") and os.path.exists("/dev/kvm"): + cmd += ["--volume", "/dev/kvm:/dev/kvm"] + if os.path.islink(testenv.custom_kernel_path): + dest = os.readlink(testenv.custom_kernel_path) + cmd += ["--volume", "{dest}:{dest}:ro"] + cmd += [ "--volume", f"{testdir_topdir}:{testdir_topdir}", diff --git a/_testenv/testenv/requirements.py b/_testenv/testenv/requirements.py index 7c39a7c..4973f6e 100644 --- a/_testenv/testenv/requirements.py +++ b/_testenv/testenv/requirements.py @@ -36,6 +36,15 @@ "wget", ] + if testenv.args.kernel: + programs += [ + "busybox", + "cpio", + "gzip", + "lddtree", + "qemu-system-x86_64", + ] + abort = False for program in programs: if not shutil.which(program): diff --git a/_testenv/testenv/testenv_cfg.py b/_testenv/testenv/testenv_cfg.py index ba830a1..8bb4119 100644 --- a/_testenv/testenv/testenv_cfg.py +++ b/_testenv/testenv/testenv_cfg.py @@ -71,6 +71,39 @@ return host, port +def verify_qemu_cfgs(): + """Check if -C or -K is set, but any of the selected configs can't run with + QEMU.""" + if not testenv.args.kernel: + return + + for basename, cfg in cfgs.items(): + missing = True + + for section in cfg.keys(): + if "qemu" in cfg[section]: + missing = False + break + + if missing: + testsuite = testenv.args.testsuite + logging.critical(f"{testsuite}/{basename}: doesn't support running in QEMU") + exit_error_readme() + + +def verify_qemu_section(path, cfg, section): + """Verify that qemu= has proper values.""" + if "qemu" not in cfg[section]: + return + + valid = ["optional"] + value = cfg[section]["qemu"] + + if value not in valid: + logging.error(f"{path}: [{section}]: qemu={value} is invalid, must be one of: {valid}") + exit_error_readme() + + def verify(cfg, path): keys_valid_testsuite = [ "clean", @@ -85,6 +118,7 @@ "package", "prepare", "program", + "qemu", "setup", "vty_host", "vty_port", @@ -127,6 +161,8 @@ logging.error(msg) exit_error_readme() + verify_qemu_section(path, cfg, section) + if section not in ["DEFAULT", "testsuite"] and "make" not in cfg[section]: logging.error(f"{path}: missing make= in section [{section}].") logging.error("If this is on purpose, set make=no.") @@ -207,6 +243,7 @@ # No --config argument given, and there is only one testenv.cfg if not testenv.args.config: cfgs[basename] = cfg + verify_qemu_cfgs() return cfgs_all[basename] = cfg @@ -228,3 +265,5 @@ if not matched: raise_error_config_arg(config_paths, config_arg) + + verify_qemu_cfgs() -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/38343?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: Ic9cb7092fd029b7ba530fc755b5d4d73a9d86350 Gerrit-Change-Number: 38343 Gerrit-PatchSet: 2 Gerrit-Owner: osmith &lt;osmith(a)sysmocom.de&gt; Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria &lt;vyanitskiy(a)sysmocom.de&gt; Gerrit-Reviewer: laforge &lt;laforge(a)osmocom.org&gt; Gerrit-Reviewer: osmith &lt;osmith(a)sysmocom.de&gt; Gerrit-Reviewer: pespin &lt;pespin(a)sysmocom.de&gt;