<p>osmith has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.osmocom.org/c/osmo-dev/+/25190">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">ttcn3.sh: build everything inside docker<br><br>Replace the previous approach of building outside of docker and mounting<br>everything inside docker while running tests, with also building<br>everything inside docker. This prevents incompatibilities between host<br>system and docker (e.g. different glibc). As nice side-effect,<br>/usr/local is not filled up anymore.<br><br>Change-Id: Ib6db8ffd916c788c9de0b3d51c82e1d7bb3f6828<br>---<br>M .gitignore<br>A ttcn3/scripts/docker_configure_make.sh<br>M ttcn3/ttcn3.sh<br>3 files changed, 73 insertions(+), 10 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.osmocom.org:29418/osmo-dev refs/changes/90/25190/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/.gitignore b/.gitignore</span><br><span>index 6326775..61d0fcf 100644</span><br><span>--- a/.gitignore</span><br><span>+++ b/.gitignore</span><br><span>@@ -17,3 +17,5 @@</span><br><span> ttcn3/3G+2G_ttcn3.deps</span><br><span> ttcn3/out/</span><br><span> ttcn3/make/</span><br><span style="color: hsl(120, 100%, 40%);">+ttcn3/.run.sh</span><br><span style="color: hsl(120, 100%, 40%);">+ttcn3/usr_local</span><br><span>diff --git a/ttcn3/scripts/docker_configure_make.sh b/ttcn3/scripts/docker_configure_make.sh</span><br><span>new file mode 100755</span><br><span>index 0000000..25cedb4</span><br><span>--- /dev/null</span><br><span>+++ b/ttcn3/scripts/docker_configure_make.sh</span><br><span>@@ -0,0 +1,38 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/bin/sh -ex</span><br><span style="color: hsl(120, 100%, 40%);">+# Passed by ttcn3.sh to gen_makefile.py to run './configure', 'make' and</span><br><span style="color: hsl(120, 100%, 40%);">+# 'make install' inside docker. The osmo-dev directory is mounted at the same</span><br><span style="color: hsl(120, 100%, 40%);">+# location inside the docker container. A usr_local dir is mounted to</span><br><span style="color: hsl(120, 100%, 40%);">+# /usr/local, so 'make install' can put all files there and following builds</span><br><span style="color: hsl(120, 100%, 40%);">+# have the files available.</span><br><span style="color: hsl(120, 100%, 40%);">+DIR_OSMODEV="$(readlink -f "$(dirname $0)/../..")"</span><br><span style="color: hsl(120, 100%, 40%);">+DIR_MAKE="$DIR_OSMODEV/ttcn3/make"</span><br><span style="color: hsl(120, 100%, 40%);">+DIR_USR_LOCAL="$DIR_OSMODEV/ttcn3/usr_local"</span><br><span style="color: hsl(120, 100%, 40%);">+RUN_SCRIPT="$DIR_OSMODEV/ttcn3/.run.sh"</span><br><span style="color: hsl(120, 100%, 40%);">+DOCKER_IMG="$1"</span><br><span style="color: hsl(120, 100%, 40%);">+UID="$(id -u)"</span><br><span style="color: hsl(120, 100%, 40%);">+shift</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+mkdir -p "$DIR_MAKE"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Script running as user inside docker</span><br><span style="color: hsl(120, 100%, 40%);">+echo "#!/bin/sh -ex" > "$RUN_SCRIPT"</span><br><span style="color: hsl(120, 100%, 40%);">+echo "cd \"$PWD\"" >> "$RUN_SCRIPT"</span><br><span style="color: hsl(120, 100%, 40%);">+for i in "$@"; do</span><br><span style="color: hsl(120, 100%, 40%);">+      echo -n "'$i' " >> "$RUN_SCRIPT"</span><br><span style="color: hsl(120, 100%, 40%);">+done</span><br><span style="color: hsl(120, 100%, 40%);">+echo >> "$RUN_SCRIPT"</span><br><span style="color: hsl(120, 100%, 40%);">+chmod +x "$RUN_SCRIPT"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+docker run \</span><br><span style="color: hsl(120, 100%, 40%);">+     --rm \</span><br><span style="color: hsl(120, 100%, 40%);">+        -e "LD_LIBRARY_PATH=/usr/local/lib" \</span><br><span style="color: hsl(120, 100%, 40%);">+       -v "$DIR_OSMODEV:$DIR_OSMODEV" \</span><br><span style="color: hsl(120, 100%, 40%);">+    -v "$DIR_USR_LOCAL:/usr/local" \</span><br><span style="color: hsl(120, 100%, 40%);">+    -v "$RUN_SCRIPT:/tmp/run.sh:ro" \</span><br><span style="color: hsl(120, 100%, 40%);">+   "$DOCKER_IMG" \</span><br><span style="color: hsl(120, 100%, 40%);">+     sh -ex -c "</span><br><span style="color: hsl(120, 100%, 40%);">+              if ! id -u $UID 2>/dev/null; then</span><br><span style="color: hsl(120, 100%, 40%);">+                  useradd -u $UID user</span><br><span style="color: hsl(120, 100%, 40%);">+          fi</span><br><span style="color: hsl(120, 100%, 40%);">+            su \$(id -un $UID) -c /tmp/run.sh</span><br><span style="color: hsl(120, 100%, 40%);">+     "</span><br><span>diff --git a/ttcn3/ttcn3.sh b/ttcn3/ttcn3.sh</span><br><span>index 9f8c1f3..6300ff7 100755</span><br><span>--- a/ttcn3/ttcn3.sh</span><br><span>+++ b/ttcn3/ttcn3.sh</span><br><span>@@ -4,8 +4,15 @@</span><br><span> DIR_OSMODEV="$(readlink -f "$(dirname $0)/..")"</span><br><span> DIR_MAKE="${DIR_MAKE:-${DIR_OSMODEV}/ttcn3/make}"</span><br><span> DIR_OUTPUT="${DIR_OUTPUT:-${DIR_OSMODEV}/ttcn3/out}"</span><br><span style="color: hsl(120, 100%, 40%);">+DIR_USR_LOCAL="$DIR_OSMODEV/ttcn3/usr_local"</span><br><span> JOBS="${JOBS:-9}"</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+# Osmocom libraries and programs relevant for the current testsuite will be</span><br><span style="color: hsl(120, 100%, 40%);">+# built in this container. It must have all build dependencies available and</span><br><span style="color: hsl(120, 100%, 40%);">+# be based on the same distribution that master-* containers are based on, so</span><br><span style="color: hsl(120, 100%, 40%);">+# there are no incompatibilities with shared libraries.</span><br><span style="color: hsl(120, 100%, 40%);">+DOCKER_IMG_BUILD="debian-stretch-jenkins"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> check_usage() {</span><br><span>       if [ -z "$PROJECT" ]; then</span><br><span>                 echo "usage: $(basename $0) PROJECT"</span><br><span>@@ -101,6 +108,9 @@</span><br><span>           echo "osmo-ttcn3-hacks"</span><br><span>    echo "osmocom-bb") > ttcn3/3G+2G_ttcn3.deps</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+        local docker_cmd="$DIR_OSMODEV/ttcn3/scripts/docker_configure_make.sh"</span><br><span style="color: hsl(120, 100%, 40%);">+      docker_cmd="$docker_cmd $USER/$DOCKER_IMG_BUILD"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span>         ./gen_makefile.py \</span><br><span>          ttcn3/3G+2G_ttcn3.deps \</span><br><span>             default.opts \</span><br><span>@@ -109,7 +119,10 @@</span><br><span>                no_doxygen.opts \</span><br><span>            no_dahdi.opts \</span><br><span>              no_optimization.opts \</span><br><span style="color: hsl(0, 100%, 40%);">-          ttcn3/ttcn3.opts -I -m "$DIR_MAKE"</span><br><span style="color: hsl(120, 100%, 40%);">+          ttcn3/ttcn3.opts \</span><br><span style="color: hsl(120, 100%, 40%);">+            --docker-cmd "$docker_cmd" \</span><br><span style="color: hsl(120, 100%, 40%);">+                --make-dir "$DIR_MAKE" \</span><br><span style="color: hsl(120, 100%, 40%);">+            --no-ldconfig</span><br><span> }</span><br><span> </span><br><span> # $1: name of repository (e.g. osmo-ttcn3-hacks)</span><br><span>@@ -157,7 +170,7 @@</span><br><span>                     ttcn3-docker-run.sh) name_install="ttcn3-docker-run" ;;</span><br><span>            esac</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-                script_path_localbin="/usr/local/bin/$name_install"</span><br><span style="color: hsl(120, 100%, 40%);">+         script_path_localbin="$DIR_USR_LOCAL/bin/$name_install"</span><br><span>            if [ -x "$script_path_localbin" ]; then</span><br><span>                    continue</span><br><span>             fi</span><br><span>@@ -185,25 +198,34 @@</span><br><span> # $1 program</span><br><span> build_osmo_program_osmodev() {</span><br><span>         local repo="$(get_program_repo "$program")"</span><br><span style="color: hsl(120, 100%, 40%);">+       local usr_local_bin="$DIR_USR_LOCAL/bin"</span><br><span>   make -C "$DIR_MAKE" "$repo"</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-     local path="$(command -v "$program")"</span><br><span style="color: hsl(0, 100%, 40%);">-       if [ -z "$path" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">-               echo "ERROR: program was not installed to PATH: $program"</span><br><span style="color: hsl(0, 100%, 40%);">-             echo "Maybe you need to add /usr/local/bin to PATH?"</span><br><span style="color: hsl(120, 100%, 40%);">+        if [ -z "$(find "$usr_local_bin" -name "$program")" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+             echo "ERROR: program was not installed properly: $program"</span><br><span style="color: hsl(120, 100%, 40%);">+          echo "Expected it to be in path: $PATH_dest"</span><br><span>               exit 1</span><br><span>       fi</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-  local pathdir="$(dirname "$path")"</span><br><span>       local reference="$DIR_MAKE/.make.$repo.build"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ -z "$(find "$pathdir" -name "$program" -newer "$reference")" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+     if [ -z "$(find "$usr_local_bin" -name "$program" -newer "$reference")" ]; then</span><br><span>              echo "ERROR: $path is outdated!"</span><br><span>           echo "Maybe you need to pass a configure argument to $repo.git, so it builds and installs $program?"</span><br><span style="color: hsl(0, 100%, 40%);">-          echo "Or the order in PATH is wrong?"</span><br><span>              exit 1</span><br><span>       fi</span><br><span> }</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+prepare_docker_build_container() {</span><br><span style="color: hsl(120, 100%, 40%);">+       local marker="$DIR_OSMODEV/ttcn3/make/.ttcn3-docker-build"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+        if [ -e "$marker" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+           return</span><br><span style="color: hsl(120, 100%, 40%);">+        fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+  make -C "$DIR_OSMODEV/src/docker-playground/$DOCKER_IMG_BUILD"</span><br><span style="color: hsl(120, 100%, 40%);">+      touch "$marker"</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> # Use osmo-dev to build one Osmocom program and its dependencies</span><br><span> build_osmo_programs() {</span><br><span>        local program</span><br><span>@@ -247,7 +269,7 @@</span><br><span>  cd "$(get_testsuite_dir_docker)"</span><br><span>   export DOCKER_ARGS="\</span><br><span>           -e LD_LIBRARY_PATH=/usr/local/lib \</span><br><span style="color: hsl(0, 100%, 40%);">-             -v /usr/local:/usr/local:ro \</span><br><span style="color: hsl(120, 100%, 40%);">+         -v "$DIR_USR_LOCAL":/usr/local:ro \</span><br><span>                -v $hacks:/osmo-ttcn3-hacks:ro \</span><br><span>             "</span><br><span>       export NO_LIST_OSMO_PACKAGES=1</span><br><span>@@ -283,6 +305,7 @@</span><br><span> clone_repo "docker-playground"</span><br><span> check_dir_testsuite</span><br><span> prepare_local_bin</span><br><span style="color: hsl(120, 100%, 40%);">+prepare_docker_build_container</span><br><span> build_osmo_programs</span><br><span> build_testsuite</span><br><span> remove_old_logs</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-dev/+/25190">change 25190</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-dev/+/25190"/><meta itemprop="name" content="View Change"/></div></div>

<div style="display:none"> Gerrit-Project: osmo-dev </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: Ib6db8ffd916c788c9de0b3d51c82e1d7bb3f6828 </div>
<div style="display:none"> Gerrit-Change-Number: 25190 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: osmith <osmith@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>