osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-dev/+/40356?usp=email ) Change subject: Remove old ttcn3.sh + related files ...................................................................... Remove old ttcn3.sh + related files Remove the precursor to testenv in osmo-ttcn3-hacks. This script worked by building docker containers from docker-playground, and then building Osmocom components and osmo-ttcn3-hacks testsuites again from source and mounting them inside the containers. With testenv there is only one container, it is possible to switch between building from source with osmo-dev and using binary packages, etc. Removing this allows removing --docker-cmd from gen_makefile.py which were only used by ttcn3.sh. This will make maintaining gen_makefile.py easier. Related: OS#6494 Change-Id: I5c53dadd24786240eaaca6d487bac02538556ebe --- M README D ttcn3/scripts/docker_configure_make.sh D ttcn3/ttcn3.opts D ttcn3/ttcn3.sh 4 files changed, 0 insertions(+), 657 deletions(-) Approvals: pespin: Looks good to me, but someone else must approve Jenkins Builder: Verified fixeria: Looks good to me, approved diff --git a/README b/README index d20499c..e425c8e 100644 --- a/README +++ b/README @@ -141,37 +141,6 @@ Find other useful scripts in src/, see src/README. - -=== ttcn3/ttcn3.sh - -Run a given testsuite with docker, so it behaves exactly the same as when -jenkins runs it [1], but use osmo-dev to build all related Osmocom components -and the testsuite, and mount them inside the docker containers. After building -the docker containers for the first time (which takes rather long), this gives -short test cycles: just modify something, run the script again and the relevant -objects will be built directly, already built objects do not need to be -rebuilt. (This is a big speed improvement over just using docker containers as -you don't need to push your sources to a git server first, have the docker -containers fetch the git sources and build the modified component from scratch -again.) - -The docker containers are used, so we don't have to maintain a second set of -configs which works without docker (that was tried before, and found not to be -feasible). - -Example usage: - - ./ttcn3/ttcn3.sh mgw - IMAGE_SUFFIX=latest ./ttcn3/ttcn3.sh mgw - -More about the testsuites: -https://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Testsuites - -Video explaining this script, related approaches and osmo-dev in general: -https://media.ccc.de/v/osmodevcall-20211210-osmith-osmo-dev-ttcn3 - -[1] https://jenkins.osmocom.org/jenkins/view/TTCN3/ - === ttcn3/tmux It's possible to run TTCN-3 test cases without using Docker. This directory diff --git a/ttcn3/scripts/docker_configure_make.sh b/ttcn3/scripts/docker_configure_make.sh deleted file mode 100755 index e6857fd..0000000 --- a/ttcn3/scripts/docker_configure_make.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/bin/sh -ex -# Passed by ttcn3.sh to gen_makefile.py to run './configure', 'make' and -# 'make install' inside docker. The osmo-dev directory is mounted at the same -# location inside the docker container. A usr_local dir is mounted to -# /usr/local, so 'make install' can put all files there and following builds -# have the files available. -# env vars: OSMODEV_PROJECT -DIR_OSMODEV="$(readlink -f "$(dirname $0)/../..")" -DIR_MAKE="$DIR_OSMODEV/ttcn3/make" -DIR_USR_LOCAL="$DIR_OSMODEV/ttcn3/usr_local" -DIR_VAR_LOCAL="$DIR_OSMODEV/ttcn3/var_local" -DIR_CCACHE="$DIR_OSMODEV/ttcn3/ccache/osmocom-programs" -RUN_SCRIPT="$DIR_OSMODEV/ttcn3/.run.sh" -UID="$(id -u)" - -# Osmocom libraries and programs relevant for the current testsuite will be -# built in this container. It must have all build dependencies available and -# be based on the same distribution that master-* containers are based on, so -# there are no incompatibilities with shared libraries. -DOCKER_IMG_BUILD="debian-bookworm-build" -DOCKER_IMG_BUILD_OGS="open5gs-master" - -docker_image_exists() { - test -n "$(docker images -q "$1")" -} - -build_docker_img() { - local img="$1" - local dp="${DIR_OSMODEV}/src/docker-playground" - - if ! docker_image_exists "$USER/$img"; then - echo "Building docker image: $USER/$img" - make -C "$dp/$img" - fi -} - -build_docker_imgs() { - build_docker_img "$DOCKER_IMG_BUILD" - - if [ "$OSMODEV_PROJECT" = "open5gs" ]; then - build_docker_img "$DOCKER_IMG_BUILD_OGS" - fi -} - -set_docker_img_var() { - case "$OSMODEV_PROJECT" in - open5gs) - DOCKER_IMG="$USER/$DOCKER_IMG_BUILD_OGS" - ;; - *) - DOCKER_IMG="$USER/$DOCKER_IMG_BUILD" - ;; - esac -} - -mkdir -p \ - "$DIR_MAKE" \ - "$DIR_CCACHE" \ - "$DIR_USR_LOCAL" \ - "$DIR_VAR_LOCAL" - -# Script running as user inside docker -echo "#!/bin/sh -ex" > "$RUN_SCRIPT" -echo "cd \"$PWD\"" >> "$RUN_SCRIPT" -for i in "$@"; do - echo -n "'$i' " >> "$RUN_SCRIPT" -done -echo >> "$RUN_SCRIPT" -chmod +x "$RUN_SCRIPT" - -build_docker_imgs -set_docker_img_var - -docker run \ - --rm \ - -t \ - -e "LD_LIBRARY_PATH=/usr/local/lib" \ - -v "$DIR_OSMODEV:$DIR_OSMODEV" \ - -v "$DIR_USR_LOCAL:/usr/local" \ - -v "$DIR_VAR_LOCAL:/var/local" \ - -v "$RUN_SCRIPT:/tmp/run.sh:ro" \ - -v "$DIR_CCACHE:/home/build/.ccache" \ - "$DOCKER_IMG" \ - sh -ex -c " - if ! id -u $UID 2>/dev/null; then - useradd -u $UID user - fi - su \$(id -un $UID) -c /tmp/run.sh - " diff --git a/ttcn3/ttcn3.opts b/ttcn3/ttcn3.opts deleted file mode 100644 index 7949d16..0000000 --- a/ttcn3/ttcn3.opts +++ /dev/null @@ -1,4 +0,0 @@ -osmo-bts --enable-trx - -# Find the ccache wrapper in /usr/lib/ccache and /usr/lib/ccache/bin -ALL CC="$(find /usr/lib/ccache -name gcc | head -n1)" CXX="$(find /usr/lib/ccache -name g++ | head -n1)" diff --git a/ttcn3/ttcn3.sh b/ttcn3/ttcn3.sh deleted file mode 100755 index 4b6392b..0000000 --- a/ttcn3/ttcn3.sh +++ /dev/null @@ -1,533 +0,0 @@ -#!/bin/sh -e -PROJECT="" -PROJECT_UPPER="" -DIR_OSMODEV="$(readlink -f "$(dirname $0)/..")" -DIR_MAKE="${DIR_MAKE:-${DIR_OSMODEV}/ttcn3/make}" -DIR_OUTPUT="${DIR_OUTPUT:-${DIR_OSMODEV}/ttcn3/out}" -DIR_USR_LOCAL="$DIR_OSMODEV/ttcn3/usr_local" -DIR_VAR_LOCAL="$DIR_OSMODEV/ttcn3/var_local" -JOBS="$(nproc)" -KERNEL_DIR="" -KERNEL_SKIP_MARKER="$DIR_MAKE/.kernel_built_from_source" -ARG_TEST_NAME="" -ARG_TEST_CONFIGS="" -DOCKER_IMG_TITAN="debian-bookworm-titan" - -clean() { - if [ "$1" != "clean" ]; then - return - fi - if [ "$#" != 1 ]; then - parse_args -h - fi - - set -x - rm -rf \ - "$DIR_MAKE" \ - "$DIR_OUTPUT" \ - "$DIR_USR_LOCAL" \ - "$DIR_VAR_LOCAL" - - if [ -d "$DIR_OSMODEV/src/osmo-ttcn3-hacks" ]; then - sudo chown -R "$(id -u):$(id -g)" \ - "$DIR_OSMODEV/src/osmo-ttcn3-hacks" - make -C "$DIR_OSMODEV/src/osmo-ttcn3-hacks" clean - fi - - exit 0 -} - -set_project() { - PROJECT="$1" - PROJECT_UPPER="$(echo "$PROJECT" | tr '[:lower:]' '[:upper:]')" - PROJECT_NO_OGS="$(echo "$PROJECT" | sed 's/-ogs$//')" -} - -parse_args() { - while getopts 'ht:c:dkf' OPTION; do - case "$OPTION" in - t) - ARG_TEST_NAME="$OPTARG" - ;; - c) - ARG_TEST_CONFIGS="$OPTARG" - ;; - d) - if [ -n "$KERNEL_TEST" ]; then - echo "ERROR: use either -d or -k" - exit 1 - fi - export KERNEL_TEST=1 - ;; - k) - if [ -n "$KERNEL_TEST" ]; then - echo "ERROR: use either -d or -k" - exit 1 - fi - if [ -z "$KERNEL_DIR" ]; then - KERNEL_DIR="$(realpath "$DIR_OSMODEV/../linux")" - fi - if ! [ -e "$KERNEL_DIR/Kbuild" ]; then - echo "ERROR: KERNEL_DIR is invalid: $KERNEL_DIR" - exit 1 - fi - - export KERNEL_TEST=1 - export KERNEL_BUILD=1 - export KERNEL_SKIP_REBUILD=1 - ;; - f) - if [ -z "$KERNEL_BUILD" ]; then - echo "ERROR: don't use -f without -k" - exit 1 - fi - rm -f "$KERNEL_SKIP_MARKER" - ;; - h|*) - local name="$(basename $0)" - echo "usage: $name [-h] [-t TESTNAME] [-c CONFIG] [-d|-k [-f]] PROJECT" - echo " or: $name clean" - echo - echo "arguments:" - echo " -h show help" - echo " -t only run the test with this name" - echo " -c only run test configuration with this name" - echo - echo "arguments for kernel tests:" - echo " -d run kernel tests with debian kernel" - echo " -k run kernel tests with kernel built from source" - echo " -f (use with -k) don't skip build if kernel already exists" - echo - echo " PROJECT the testsuite project to run" - echo - echo "examples:" - echo " $name bsc" - echo " $name bsc-sccplite" - echo " $name -t TC_gsup_check_imei hlr" - echo " $name -d ggsn" - echo " $name -k ggsn" - echo " $name -k -f ggsn" - echo " $name ggsn-ogs" - echo " $name hss" - echo " $name mme" - echo " $name pgw" - echo " $name -t TC_est_dchan -c generic bts" - exit 1 - ;; - esac - done - shift "$(($OPTIND - 1))" - - if [ "$#" != "1" ]; then - parse_args -h - fi - - set_project "$1" - if ! [ -d "$(get_testsuite_dir_docker)" ]; then - set_project "$1-ogs" - if ! [ -d "$(get_testsuite_dir_docker)" ]; then - echo "ERROR: can't find testsuite_dir, adjust" \ - "get_testsuite_dir_docker" - fi - fi - - echo "PROJECT: $PROJECT" -} - -check_usage() { - if [ -z "$PROJECT" ]; then - parse_args -h - fi -} - -get_testsuite_dir() { - local hacks="${DIR_OSMODEV}/src/osmo-ttcn3-hacks" - - case "$PROJECT" in - bsc-*) echo "$hacks/bsc" ;; - bts-*) echo "$hacks/bts" ;; - ggsn|ggsn-ogs) echo "$hacks/ggsn_tests" ;; - pcu-sns) echo "$hacks/pcu" ;; - *) echo "$hacks/$PROJECT_NO_OGS" ;; - esac -} - -get_testsuite_dir_docker() { - local dp="${DIR_OSMODEV}/src/docker-playground" - - case "$PROJECT" in - bsc-*) - echo "$dp/ttcn3-bsc-test-$(echo "$PROJECT" | cut -d - -f 2-)" - ;; - *-ogs) - echo "$dp/ttcn3-$PROJECT_NO_OGS-test-ogs" - ;; - *) - echo "$dp/ttcn3-$PROJECT-test" - ;; - esac -} - -get_testsuite_image() { - case "$PROJECT" in - bsc-*) - echo "$USER/ttcn3-bsc-test" - ;; - ggsn-ogs) - echo "$USER/ttcn3-ggsn-test" - ;; - *-ogs) - echo "$USER/ttcn3-$PROJECT_NO_OGS-test-ogs" - ;; - *) - echo "$USER/ttcn3-$PROJECT-test" - ;; - esac -} - -# Dependencies not mentioned in all.deps -get_extra_libraries() { - case "$PROJECT" in - ggsn) echo "libgtpnl" ;; # needed for --enable-gtp-linux - esac -} - -# Programs that need to be built -get_programs() { - case "$PROJECT" in - # Open5gs - *-ogs) echo "open5gs" ;; - pgw) echo "open5gs osmo-uecups-daemon" ;; - - # Osmocom - bsc|bsc-*) echo "osmo-stp osmo-bsc osmo-bts-omldummy" ;; - bts) echo "osmo-bsc osmo-bts-trx trxcon" ;; - msc) echo "osmo-stp osmo-msc" ;; - pcu-sns) echo "osmo-pcu" ;; - pcu) echo "osmo-pcu osmo-bsc osmo-bts-virtual" ;; - sgsn) echo "osmo-stp osmo-sgsn" ;; - sip) echo "osmo-sip-connector" ;; - *) echo "osmo-$PROJECT" ;; - esac -} - -# Return the git repository name, which has the source for a specific program. -# $1: program name -get_program_repo() { - case "$1" in - osmo-bts-*) echo "osmo-bts" ;; - osmo-pcap-*) echo "osmo-pcap" ;; - osmo-stp) echo "libosmo-sigtran" ;; - osmo-uecups-*) echo "osmo-uecups" ;; - trxcon) echo "osmocom-bb_trxcon" ;; - *) echo "$1" ;; - esac -} - -check_ttcn3_install() { - if ! command -v ttcn3_compiler > /dev/null; then - echo "ERROR: ttcn3_compiler is not installed." - echo "Install eclipse-titan from the Osmocom latest repository." - echo "Details: https://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Tests… - exit 1 - fi -} - -setup_dir_make() { - cd "$DIR_OSMODEV" - - local docker_cmd="$DIR_OSMODEV/ttcn3/scripts/docker_configure_make.sh" - - ./gen_makefile.py \ - default.opts \ - gtp_linux.opts \ - iu.opts \ - no_dahdi.opts \ - no_doxygen.opts \ - no_optimization.opts \ - no_systemd.opts \ - ttcn3/ttcn3.opts \ - werror.opts \ - --docker-cmd "$docker_cmd" \ - --make-dir "$DIR_MAKE" \ - --no-ldconfig \ - --no-make-check \ - --auto-distclean -} - -# $1: name of repository (e.g. osmo-ttcn3-hacks) -clone_repo() { - if ! [ -e "$DIR_OSMODEV/ttcn3/make/.make.${1}.clone" ]; then - make -C "$DIR_MAKE" ".make.${1}.clone" - fi -} - -# Require testsuite dir and docker-playground dir -check_dir_testsuite() { - local program - local config_testsuite - local dir_testsuite="$(get_testsuite_dir)" - local dir_testsuite_docker="$(get_testsuite_dir_docker)" - - if ! [ -d "$dir_testsuite" ]; then - echo "ERROR: project '$PROJECT' is invalid, resulting path not found: $dir_testsuite" - exit 1 - fi - - if ! [ -d "$dir_testsuite_docker" ]; then - echo "ERROR: could not find docker dir for project: $PROJECT: $dir_testsuite_docker" - echo "Adjust get_testsuite_dir_docker?" - exit 1 - fi - - # Sanity check for jenkins.sh - if ! grep -q DOCKER_ARGS $dir_testsuite_docker/jenkins.sh; then - echo "ERROR: DOCKER_ARGS not found in $dir_testsuite_docker/jenkins.sh!" - exit 1 - fi -} - -# Copy scripts from docker-playground to /usr/local/bin, so we don't miss them when mounting the outside /usr/local/bin -# inside the docker container -prepare_local_bin() { - local scripts=" - ${DIR_OSMODEV}/src/docker-playground/common/respawn.sh - ${DIR_OSMODEV}/src/docker-playground/common/ttcn3-docker-run.sh - " - - for script in $scripts; do - local script_path_localbin - local name_install="$(basename "$script")" - - case "$name_install" in - ttcn3-docker-run.sh) name_install="ttcn3-docker-run" ;; - esac - - script_path_localbin="$DIR_USR_LOCAL/bin/$name_install" - - install -v -Dm755 "$script" "$script_path_localbin" - done -} - -prepare_docker_testsuite_container() { - local testsuite_image="$(get_testsuite_image)" - - if docker_image_exists "$testsuite_image"; then - return - fi - - if ! docker_image_exists "$USER/$DOCKER_IMG_TITAN"; then - echo "Building docker image: $USER/$DOCKER_IMG_TITAN" - local dp="${DIR_OSMODEV}/src/docker-playground" - make -C "$dp/$DOCKER_IMG_TITAN" - fi - - echo "Building docker image: $testsuite_image" - local testsuite_dir="$(get_testsuite_dir_docker)" - make -C "$testsuite_dir" -} - -# Use osmo-dev to build libraries not mentioned in all.deps, for example the -# libgtpnl dependency of osmo-ggsn that is only needed with --enable-gtp-linux. -build_extra_libraries() { - local library - local libraries="$(get_extra_libraries)" - - for library in $libraries; do - set -x - make -C "$DIR_MAKE" "$library" - set +x - done -} - -# Use osmo-dev to build one Osmocom program and its dependencies -build_osmo_programs() { - local program - local programs="$(get_programs)" - local make_args="-C $DIR_MAKE" - - for program in $programs; do - local repo="$(get_program_repo "$program")" - make_args="$make_args $repo" - done - - set -x - make $make_args - set +x - - for program in $programs; do - if [ "$program" = "open5gs" ]; then - # open5gs binaries have different names (open5gs-mmed, - # etc.) - continue - fi - - local repo="$(get_program_repo "$program")" - local usr_local_bin="$DIR_USR_LOCAL/bin" - - if [ -z "$(find "$usr_local_bin" -name "$program")" ]; then - echo "ERROR: program was not installed properly: $program" - echo "Expected it to be in path: $PATH_dest" - exit 1 - fi - - local reference="$DIR_MAKE/.make.$repo.build" - if [ -z "$(find "$usr_local_bin" -name "$program" -newer "$reference")" ]; then - echo "ERROR: $path is outdated!" - echo "Maybe you need to pass a configure argument to $repo.git, so it builds and installs" \ - "$program?" - exit 1 - fi - done -} - -docker_image_exists() { - test -n "$(docker images -q "$1")" -} - -build_testsuite() { - cd "$(get_testsuite_dir)" - - local deps_marker="${DIR_OSMODEV}/ttcn3/make/.ttcn3-deps" - if ! [ -e "$deps_marker" ]; then - make -C "${DIR_OSMODEV}/src/osmo-ttcn3-hacks/deps" -j"$JOBS" - touch "$deps_marker" - fi - - # Build inside docker, so the resulting binary links against the - # libraries available in docker and not the ones from the host system, - # since we will run it inside docker later too. - local hacks="${DIR_OSMODEV}/src/osmo-ttcn3-hacks" - - local testsuite_image="$(get_testsuite_image)" - echo "testsuite_image: $testsuite_image" - - mkdir -p "$"$DIR_OSMODEV/ttcn3/ccache/ttcn3 - - # -t: add a tty, so we get color output from the compiler - docker run \ - --rm \ - -t \ - -v "$hacks:/osmo-ttcn3-hacks" \ - -v "$DIR_OSMODEV/ttcn3/ccache/ttcn3:/root/.ccache" \ - "$testsuite_image" \ - sh -exc " - cd /osmo-ttcn3-hacks/$(basename "$(get_testsuite_dir)"); - ./gen_links.sh; - USE_CCACHE=1 ./regen_makefile.sh; - make compile; - make -j"$JOBS" - " -} - -# Similar to update_kernel_config in docker-playground -# scripts/kernel-test/kernel-build.sh -update_kernel_config() { - local previous="$DIR_MAKE/.kernel.previous.config" - local fragment="$(get_testsuite_dir_docker)/fragment.config" - - set -x - make defconfig - scripts/kconfig/merge_config.sh -m .config "$fragment" - make olddefconfig - set +x - - if [ -e "$previous" ] && ! diff -q "$previous" .config; then - # Remove everything built with previous config - echo "Kernel config changed, running 'make clean'" - make clean - fi - - cp .config "$previous" -} - -build_kernel() { - local image="arch/x86/boot/bzImage" - - if [ -z "$KERNEL_DIR" ]; then - return - fi - - cd "$KERNEL_DIR" - if ! [ -e "$image" ]; then - rm -f "$KERNEL_SKIP_MARKER" - fi - if ! [ -e "$KERNEL_SKIP_MARKER" ]; then - update_kernel_config - make -j"$JOBS" - fi - - # Copy the kernel to the path expected by - # docker-playground scripts/kernel-test/kernel-build.sh - local dest="$DIR_OSMODEV/src/docker-playground/_cache/linux/$image" - mkdir -p "$(dirname "$dest")" - cp "$image" "$dest" - - touch "$KERNEL_SKIP_MARKER" -} - -run_docker() { - local hacks="${DIR_OSMODEV}/src/osmo-ttcn3-hacks" - local docker_dir="$(get_testsuite_dir_docker)" - local docker_name="$(basename "$docker_dir")" - local marker="${DIR_OSMODEV}/ttcn3/make/.docker.$docker_name.$IMAGE_SUFFIX" - - # Skip building docker containers if this already ran - if [ -e "$marker" ]; then - echo "NOTE: skipping docker container build, because marker exists: $marker" - export NO_DOCKER_IMAGE_BUILD=1 - fi - - cd "$(get_testsuite_dir_docker)" - DOCKER_ARGS="\ - -e LD_LIBRARY_PATH=/usr/local/lib \ - -v "$DIR_USR_LOCAL":/usr/local \ - -v "$DIR_VAR_LOCAL":/var/local \ - -v $hacks:/osmo-ttcn3-hacks:ro \ - " - if [ -n "$ARG_TEST_NAME" ]; then - DOCKER_ARGS="$DOCKER_ARGS -e TEST_NAME=$ARG_TEST_NAME" - fi - - export DOCKER_ARGS - export NO_LIST_OSMO_PACKAGES=1 - export TEST_CONFIGS="$ARG_TEST_CONFIGS" - ./jenkins.sh - - touch "$marker" -} - -remove_old_logs() { - sudo rm -rf /tmp/tmp.* 2>/dev/null -} - -collect_logs() { - # Format logs - for log in /tmp/logs/*-tester*/*.merged; do - ttcn3_logformat -o "${log}.log" "$log" - sudo rm "$log" - done - - # Print log path - echo "---" - echo "Logs: /tmp/logs" - echo "---" -} - -clean "$@" -parse_args "$@" -check_usage -check_ttcn3_install -setup_dir_make -clone_repo "osmo-ttcn3-hacks" -clone_repo "docker-playground" -check_dir_testsuite -prepare_local_bin -build_extra_libraries -build_osmo_programs -build_kernel -prepare_docker_testsuite_container -build_testsuite -remove_old_logs -run_docker -collect_logs -- To view, visit https://gerrit.osmocom.org/c/osmo-dev/+/40356?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: osmo-dev Gerrit-Branch: master Gerrit-Change-Id: I5c53dadd24786240eaaca6d487bac02538556ebe Gerrit-Change-Number: 40356 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>