osmith has uploaded this change for review.
testenv: support running SUT in QEMU
Add two new arguments -C|--custom-kernel and -D|--debian-kernel. If any
of these is set, pass an environment variable TESTENV_QEMU_KERNEL with
the path to the kernel when running commands from testenv.cfg.
These commands can then source the new qemu_functions.sh and use it to
build an initramfs with the SUT and depending libraries on the fly, and
start up QEMU to boot right to starting the SUT. All of that takes about
~1s on my system with kvm. Without kvm ~5s.
A follow-up patch will adjust the ggsn testenv configs to optionally run
osmo-ggsn in QEMU for testing kernel GTP-U.
These scripts are based on scripts/kernel-tests from docker-playground.
Related: osmo-ci Id64a1a778fa38eec20498c36b390332f75d7d3f5
Change-Id: Ic9cb7092fd029b7ba530fc755b5d4d73a9d86350
---
M .gitignore
M _testenv/README.md
M _testenv/data/podman/Dockerfile
A _testenv/data/scripts/qemu/qemu_functions.sh
A _testenv/data/scripts/qemu/qemu_ifup.sh
A _testenv/data/scripts/qemu/qemu_init.sh
A _testenv/data/scripts/qemu/qemu_wait.sh
M _testenv/testenv/__init__.py
M _testenv/testenv/cmd.py
M _testenv/testenv/podman.py
M _testenv/testenv/requirements.py
M _testenv/testenv/testenv_cfg.py
12 files changed, 400 insertions(+), 1 deletion(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/43/38343/1
diff --git a/.gitignore b/.gitignore
index 241a1d2..94061e3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -25,3 +25,4 @@
sms.db-wal
__pycache__
tags
+.linux
diff --git a/_testenv/README.md b/_testenv/README.md
index 7db3e24..3fed9c9 100644
--- a/_testenv/README.md
+++ b/_testenv/README.md
@@ -20,6 +20,24 @@
Additional packages get installed after starting the container, with a package
cache mounted to avoid unnecessary downloads.
+### QEMU
+
+For SUTs that interact with kernel drivers, it is desirable to run them with a
+separate kernel in QEMU. This can be enabled by running `./testenv.py run` with
+the `-D` (Debian kernel) or `-C` (custom kernel) arguments. See the `ggsn`
+testsuite for reference.
+
+#### Custom kernel
+
+When using `-C`, testenv uses a `.linux` file in the `osmo-ttcn3-hacks` dir as
+kernel. You can download a pre-built kernel from jenkins:
+
+$ wget -O .linux "https://jenkins.osmocom.org/jenkins/job/build-kernel-torvalds/lastSuccessfulBuild/artifact/output/linux"
+$ wget -O .linux "https://jenkins.osmocom.org/jenkins/job/build-kernel-net-next/lastSuccessfulBuild/artifact/output/linux"
+
+Or build your own kernel, see:
+https://gitea.osmocom.org/osmocom/osmo-ci/src/branch/master/scripts/kernel
+
## testenv.cfg
The `testenv.cfg` file has one `[testsuite]` section, typically with one or
@@ -98,6 +116,11 @@
* `vty_host=`: optionally set the VTY host for the SUT component to be used
when obtaining a talloc report. If this is not set, `127.0.0.1` is used.
+* `qemu=`: set to `optional` to allow running this test component in QEMU.
+ Additional logic must be added to build an initrd with the test component and
+ actually run it in QEMU, this is done by sourcing `qemu_functions.sh` and
+ using functions from there. See the `ggsn` testsuite for reference.
+
### Executables
#### $PATH
@@ -108,6 +131,7 @@
* The directory of the testsuite
* The directory for binaries built from source
* The directory `_testenv/data/scripts` (which has e.g. `respawn.sh`)
+* The directory `_testenv/data/scripts/qemu`
#### $PWD (current working dir)
@@ -180,6 +204,11 @@
files. Jenkins sets this to `esc` instead of `esc256` for better contrast on
white background.
+* `TESTENV_NO_KVM`:
+ Do not mount /dev/kvm in podman. This is used in jenkins where /dev/kvm is
+ available but doesn't work in podman. QEMU runs a bit slower when this is
+ set.
+
## Troubleshooting
### Timeout waiting for RESET-ACK after sending RESET
diff --git a/_testenv/data/podman/Dockerfile b/_testenv/data/podman/Dockerfile
index 13412a2..6f6c739 100644
--- a/_testenv/data/podman/Dockerfile
+++ b/_testenv/data/podman/Dockerfile
@@ -12,20 +12,29 @@
# Install packages from Debian repositories (alphabetic order)
ENV DEBIAN_FRONTEND=noninteractive
RUN set -x && \
+ mkdir -p /etc/kernel/postinst.d && \
+ touch /etc/kernel/postinst.d/initramfs-tools && \
apt-get update && \
- apt-get install -y --no-install-recommends \
+ apt-get install \
+ -y \
+ --no-install-recommends \
+ -o Dpkg::Options::="--force-confold" \
autoconf \
automake \
+ bc \
bison \
bridge-utils \
build-essential \
+ busybox-static \
ca-certificates \
ccache \
cmake \
+ cpio \
erlang-nox \
flex \
gdb \
git \
+ gzip \
iproute2 \
iputils-ping \
libbson-dev \
@@ -34,6 +43,8 @@
libcurl4-gnutls-dev \
libdbd-sqlite3 \
libdbi-dev \
+ libelf-dev \
+ libgcc-11-dev \
libgcrypt-dev \
libgnutls28-dev \
libidn11-dev \
@@ -57,13 +68,16 @@
libusb-1.0-0-dev \
libyaml-dev \
libzmq3-dev \
+ linux-image-amd64 \
meson \
netcat-openbsd \
+ pax-utils \
pcscd \
pkg-config \
procps \
psmisc \
python3-pip \
+ qemu-system-x86 \
rsync \
source-highlight \
sqlite3 \
diff --git a/_testenv/data/scripts/qemu/qemu_functions.sh b/_testenv/data/scripts/qemu/qemu_functions.sh
new file mode 100755
index 0000000..c0dd8cc
--- /dev/null
+++ b/_testenv/data/scripts/qemu/qemu_functions.sh
@@ -0,0 +1,191 @@
+#!/bin/sh -ex
+INITRD_DIR="$PWD/_initrd"
+
+# Add one or more files to the initramfs, with parent directories.
+# usr-merge: resolve symlinks for /lib -> /usr/lib etc. so "cp --parents" does
+# not fail with "cp: cannot make directory '/tmp/initrd/lib': File exists"
+# $@: path to files
+qemu_initrd_add_file() {
+ local i
+
+ for i in "$@"; do
+ case "$i" in
+ /bin/*|/sbin/*|/lib/*|/lib64/*)
+ cp -a --parents "$i" "$INITRD_DIR"/usr
+ ;;
+ *)
+ cp -a --parents "$i" "$INITRD_DIR"
+ ;;
+ esac
+ done
+}
+
+# Add kernel module files with dependencies
+# $@: kernel module names
+qemu_initrd_add_mod() {
+ if [ "$TESTENV_QEMU_KERNEL" != "debian" ]; then
+ # Assume all drivers are statically built into the kernel, when
+ # using a custom kernel
+ return
+ fi
+
+ local kernel="$(basename /lib/modules/*)"
+ local files="$(modprobe \
+ -a \
+ --dry-run \
+ --show-depends \
+ --set-version="$kernel" \
+ "$@" \
+ | sort -u \
+ | cut -d ' ' -f 2)"
+
+ qemu_initrd_add_file $files
+
+ # Save the list of modules
+ for i in $@; do
+ echo "$i" >> "$INITRD_DIR"/modules
+ done
+}
+
+# Add binaries with depending libraries
+# $@: paths to binaries
+qemu_initrd_add_bin() {
+ local bin
+ local bin_path
+ local file
+
+ for bin in "$@"; do
+ local bin_path="$(which "$bin")"
+ if [ -z "$bin_path" ]; then
+ echo "ERROR: file not found: $bin"
+ exit 1
+ fi
+
+ lddtree_out="$(lddtree -l "$bin_path")"
+ if [ -z "$lddtree_out" ]; then
+ echo "ERROR: lddtree failed on '$bin_path'"
+ exit 1
+ fi
+
+ for file in $lddtree_out; do
+ qemu_initrd_add_file "$file"
+
+ # Copy resolved symlink
+ if [ -L "$file" ]; then
+ qemu_initrd_add_file "$(realpath "$file")"
+ fi
+ done
+ done
+}
+
+# Add command to run inside the initramfs
+# $@: commands
+qemu_initrd_add_cmd() {
+ local i
+
+ if ! [ -e "$INITRD_DIR"/cmd.sh ]; then
+ echo "#!/bin/sh -ex" > "$INITRD_DIR"/cmd.sh
+ chmod +x "$INITRD_DIR"/cmd.sh
+ fi
+
+ for i in "$@"; do
+ echo "$i" >> "$INITRD_DIR"/cmd.sh
+ done
+}
+
+qemu_initrd_init() {
+ mkdir "$INITRD_DIR"
+
+ for dir in bin sbin lib lib64; do
+ ln -s usr/"$dir" "$INITRD_DIR"/"$dir"
+ done
+
+ mkdir -p \
+ "$INITRD_DIR"/dev/net \
+ "$INITRD_DIR"/proc \
+ "$INITRD_DIR"/sys \
+ "$INITRD_DIR"/tmp \
+ "$INITRD_DIR"/usr/bin \
+ "$INITRD_DIR"/usr/sbin
+
+ qemu_initrd_add_bin \
+ busybox
+
+ qemu_initrd_add_cmd \
+ "export PATH='$PATH:\$PATH'"
+
+ if [ "$TESTENV_QEMU_KERNEL" = "debian" ]; then
+ qemu_initrd_add_mod \
+ virtio_net \
+ virtio_pci
+
+ qemu_initrd_add_file \
+ /lib/modules/*/modules.dep
+ fi
+}
+
+qemu_initrd_finish() {
+ cp "$TESTENV_QEMU_SCRIPTS"/qemu_init.sh "$INITRD_DIR"/init
+
+ tree -a "$INITRD_DIR"
+
+ ( cd "$INITRD_DIR"; find . -print0 \
+ | cpio --quiet -o -0 -H newc \
+ | gzip -1 > "$INITRD_DIR".gz )
+}
+
+qemu_initrd_exit_error() {
+ # Building the initrd is quite verbose, therefore put it in a log file
+ # and only output its contents on error (see e.g. osmo-ggsn/run.sh)
+ cat "$1"
+ set +x
+ echo
+ echo "ERROR: failed to build the initrd!"
+ echo
+ exit 1
+}
+
+qemu_random_mac() {
+ printf "52:54:"
+ date "+%c %N" | sha1sum | sed 's/\(.\{2\}\)/\1:/g' | cut -d: -f 1-4
+}
+
+qemu_run() {
+ local machine_arg
+ local kernel="$TESTENV_QEMU_KERNEL"
+ local kernel_cmdline="
+ root=/dev/ram0
+ console=ttyS0
+ panic=-1
+ init=/init
+ loglevel=8
+ "
+
+ if [ "$kernel" = "debian" ]; then
+ kernel="$(ls -1 /boot/vmlinuz*)"
+ fi
+
+ if [ -e /dev/kvm ]; then
+ machine_arg="-machine pc,accel=kvm"
+ else
+ machine_arg="-machine pc"
+ fi
+
+ # sudo is required to set up networking in qemu_ifup.sh
+ # </dev/null to deatch stdin, so qemu doesn't capture ^C
+ sudo sh -c "
+ qemu-system-x86_64 \
+ $machine_arg \
+ -smp 1 \
+ -m 512M \
+ -no-user-config -nodefaults -display none \
+ -no-reboot \
+ -kernel '$kernel' \
+ -initrd '$INITRD_DIR.gz' \
+ -append '$kernel_cmdline' \
+ -serial stdio \
+ -netdev 'tap,id=nettest,script=$TESTENV_QEMU_SCRIPTS/qemu_ifup.sh' \
+ -device 'virtio-net-pci,netdev=nettest,mac=$(qemu_random_mac)' \
+ </dev/null
+ "
+}
diff --git a/_testenv/data/scripts/qemu/qemu_ifup.sh b/_testenv/data/scripts/qemu/qemu_ifup.sh
new file mode 100755
index 0000000..370bea5
--- /dev/null
+++ b/_testenv/data/scripts/qemu/qemu_ifup.sh
@@ -0,0 +1,10 @@
+#!/bin/sh -e
+br="testenv0"
+qemu_if="$1"
+
+echo "qemu_ifup.sh: $br, $qemu_if"
+set -x
+
+ip link set "$qemu_if" up
+
+brctl addif "$br" "$qemu_if"
diff --git a/_testenv/data/scripts/qemu/qemu_init.sh b/_testenv/data/scripts/qemu/qemu_init.sh
new file mode 100755
index 0000000..1c8b39b
--- /dev/null
+++ b/_testenv/data/scripts/qemu/qemu_init.sh
@@ -0,0 +1,36 @@
+#!/bin/busybox sh
+echo "Running initrd-init.sh"
+set -ex
+
+export HOME=/root
+export LD_LIBRARY_PATH=/usr/local/lib
+export PATH=/usr/local/bin:/usr/bin:/bin:/sbin:/usr/local/sbin:/usr/sbin
+export TERM=screen
+
+/bin/busybox --install -s
+
+mknod /dev/null c 1 3
+
+# Required for osmo-ggsn
+mknod /dev/net/tun c 10 200
+
+hostname qemu
+
+mount -t proc proc /proc
+mount -t sysfs sys /sys
+
+# Load modules from qemu_initrd_add_mod()
+if [ -e /modules ]; then
+ cat /modules | xargs -t -n1 modprobe
+fi
+
+ip link set lo up
+ip link set eth0 up
+
+echo "KERNEL_TEST_VM_IS_READY"
+
+# Use '|| true' to avoid "attempting to kill init" kernel panic on failure
+/cmd.sh || true
+
+# Avoid kernel panic when init exits
+poweroff -f
diff --git a/_testenv/data/scripts/qemu/qemu_wait.sh b/_testenv/data/scripts/qemu/qemu_wait.sh
new file mode 100755
index 0000000..fb45c7e
--- /dev/null
+++ b/_testenv/data/scripts/qemu/qemu_wait.sh
@@ -0,0 +1,22 @@
+#!/bin/sh -e
+if [ -z "$TESTENV_QEMU_KERNEL" ]; then
+ exit 0
+fi
+
+LOGFILE="$(basename "$PWD").log"
+
+i=0
+for i in $(seq 1 600); do
+ sleep 0.1
+ if grep -q KERNEL_TEST_VM_IS_READY "$LOGFILE"; then
+ # Wait some more for SUT to become ready
+ sleep 1
+ exit 0
+ fi
+done
+
+echo
+echo "ERROR: Timeout while waiting for QEMU to become ready"
+echo
+
+exit 1
diff --git a/_testenv/testenv/__init__.py b/_testenv/testenv/__init__.py
index 2af83ed..eb687f3 100644
--- a/_testenv/testenv/__init__.py
+++ b/_testenv/testenv/__init__.py
@@ -13,6 +13,7 @@
src_dir = os.environ.get("TESTENV_SRC_DIR", os.path.realpath(f"{__file__}/../../../.."))
data_dir = os.path.join(os.path.realpath(f"{__file__}/../.."), "data")
+custom_kernel_path = os.path.join(os.path.realpath(f"{__file__}/../../.."), ".linux")
distro_default = "debian:bookworm"
cache_dir_default = os.path.join(os.path.expanduser("~/.cache"), "osmo-ttcn3-testenv")
ccache_dir_default = os.path.join(cache_dir_default, "ccache")
@@ -91,6 +92,29 @@
metavar="OBS_PROJECT",
help="use binary packages from this Osmocom OBS project instead (e.g. osmocom:nightly)",
)
+
+ group = sub_run.add_argument_group(
+ "QEMU options",
+ "For some tests, the SUT can or must run in QEMU, typically to use kernel GTP-U.",
+ )
+ group = group.add_mutually_exclusive_group()
+ group.add_argument(
+ "-D",
+ "--debian-kernel",
+ action="store_const",
+ dest="kernel",
+ const="debian",
+ help="run SUT in QEMU with debian kernel",
+ )
+ group.add_argument(
+ "-C",
+ "--custom-kernel",
+ action="store_const",
+ dest="kernel",
+ const="custom",
+ help=f"run SUT in QEMU with custom kernel ({custom_kernel_path})",
+ )
+
group = sub_run.add_argument_group(
"config file options",
"Testsuite and test component configs"
@@ -159,6 +183,15 @@
if args.binary_repo and not args.podman:
raise NoTraceException("--binary-repo requires --podman")
+ if args.kernel == "debian" and not args.podman:
+ raise NoTraceException("--kernel-debian requires --podman")
+
+ if args.kernel == "custom" and not os.path.exists(custom_kernel_path):
+ logging.critical(
+ "See _testenv/README.md for more information on downloading a pre-built kernel or building your own kernel."
+ )
+ raise NoTraceException(f"For --kernel-custom, put a symlink or copy of your kernel to: {custom_kernel_path}")
+
ttcn3_hacks_dir_src = os.path.realpath(f"{__file__}/../../..")
testsuite_dir = os.path.join(ttcn3_hacks_dir_src, args.testsuite)
if not os.path.exists(testsuite_dir):
diff --git a/_testenv/testenv/cmd.py b/_testenv/testenv/cmd.py
index 6f502fc..5e651bd 100644
--- a/_testenv/testenv/cmd.py
+++ b/_testenv/testenv/cmd.py
@@ -49,6 +49,13 @@
else:
env_extra["OSMO_DEV_MAKE_DIR"] = os.path.join(testenv.args.cache, "host", "make")
+ if testenv.args.kernel == "debian":
+ env_extra["TESTENV_QEMU_KERNEL"] = "debian"
+ elif testenv.args.kernel == "custom":
+ env_extra["TESTENV_QEMU_KERNEL"] = testenv.custom_kernel_path
+ if testenv.args.kernel:
+ env_extra["TESTENV_QEMU_SCRIPTS"] = os.path.join(testenv.data_dir, "scripts/qemu")
+
def exit_error_cmd(completed, error_msg):
""":param completed: return from run_cmd() below"""
@@ -62,6 +69,7 @@
def generate_env(env={}, podman=False):
ret = dict(env_extra)
path = os.path.join(testenv.data_dir, "scripts")
+ path += f":{os.path.join(testenv.data_dir, 'scripts/qemu')}"
if testenv.testsuite.ttcn3_hacks_dir:
path += f":{os.path.join(testenv.testsuite.ttcn3_hacks_dir, testenv.args.testsuite)}"
diff --git a/_testenv/testenv/podman.py b/_testenv/testenv/podman.py
index 60aecca..bd1e8d8 100644
--- a/_testenv/testenv/podman.py
+++ b/_testenv/testenv/podman.py
@@ -221,6 +221,13 @@
f"{osmo_dev_dir}:{osmo_dev_dir}",
]
+ if testenv.args.kernel:
+ if not os.environ.get("TESTENV_NO_KVM") and os.path.exists("/dev/kvm"):
+ cmd += ["--volume", "/dev/kvm:/dev/kvm"]
+ if os.path.islink(testenv.custom_kernel_path):
+ dest = os.readlink(testenv.custom_kernel_path)
+ cmd += ["--volume", "{dest}:{dest}:ro"]
+
cmd += [
"--volume",
f"{testdir_topdir}:{testdir_topdir}",
diff --git a/_testenv/testenv/requirements.py b/_testenv/testenv/requirements.py
index 7c39a7c..4973f6e 100644
--- a/_testenv/testenv/requirements.py
+++ b/_testenv/testenv/requirements.py
@@ -36,6 +36,15 @@
"wget",
]
+ if testenv.args.kernel:
+ programs += [
+ "busybox",
+ "cpio",
+ "gzip",
+ "lddtree",
+ "qemu-system-x86_64",
+ ]
+
abort = False
for program in programs:
if not shutil.which(program):
diff --git a/_testenv/testenv/testenv_cfg.py b/_testenv/testenv/testenv_cfg.py
index ba830a1..8bb4119 100644
--- a/_testenv/testenv/testenv_cfg.py
+++ b/_testenv/testenv/testenv_cfg.py
@@ -71,6 +71,39 @@
return host, port
+def verify_qemu_cfgs():
+ """Check if -C or -K is set, but any of the selected configs can't run with
+ QEMU."""
+ if not testenv.args.kernel:
+ return
+
+ for basename, cfg in cfgs.items():
+ missing = True
+
+ for section in cfg.keys():
+ if "qemu" in cfg[section]:
+ missing = False
+ break
+
+ if missing:
+ testsuite = testenv.args.testsuite
+ logging.critical(f"{testsuite}/{basename}: doesn't support running in QEMU")
+ exit_error_readme()
+
+
+def verify_qemu_section(path, cfg, section):
+ """Verify that qemu= has proper values."""
+ if "qemu" not in cfg[section]:
+ return
+
+ valid = ["optional"]
+ value = cfg[section]["qemu"]
+
+ if value not in valid:
+ logging.error(f"{path}: [{section}]: qemu={value} is invalid, must be one of: {valid}")
+ exit_error_readme()
+
+
def verify(cfg, path):
keys_valid_testsuite = [
"clean",
@@ -85,6 +118,7 @@
"package",
"prepare",
"program",
+ "qemu",
"setup",
"vty_host",
"vty_port",
@@ -127,6 +161,8 @@
logging.error(msg)
exit_error_readme()
+ verify_qemu_section(path, cfg, section)
+
if section not in ["DEFAULT", "testsuite"] and "make" not in cfg[section]:
logging.error(f"{path}: missing make= in section [{section}].")
logging.error("If this is on purpose, set make=no.")
@@ -207,6 +243,7 @@
# No --config argument given, and there is only one testenv.cfg
if not testenv.args.config:
cfgs[basename] = cfg
+ verify_qemu_cfgs()
return
cfgs_all[basename] = cfg
@@ -228,3 +265,5 @@
if not matched:
raise_error_config_arg(config_paths, config_arg)
+
+ verify_qemu_cfgs()
To view, visit change 38343. To unsubscribe, or for help writing mail filters, visit settings.