<p>osmith <strong>merged</strong> this change.</p><p><a href="https://gerrit.osmocom.org/13415">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Max: Looks good to me, but someone else must approve
Harald Welte: Looks good to me, approved
osmith: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">Add osmocom-build-old-tags-against-master.sh<br><br>Build old releases of Osmocom programs and libraries against<br>"master of the day" to detect breakage.<br><br>Redirect each build's output to its own log file, so it is easy to see<br>what is currently getting build, and what failed. On error, print the<br>end of the new failing build logs, along with a note to find the full<br>logs in the jenkins artifacts.<br><br>This initial configuration builds the last three release tags of the<br>Osmocom repositories against master. The configuration can be changed<br>easily.<br><br>Indicate known failures with "err" instead of "ERR" in the output, do<br>not cause them to fail the build and do not print the beginning of the<br>error log (it is still in the artifacts). This way, new errors stand out<br>and don't get overlooked among the known errors.<br><br>Related: OS#3765<br>Change-Id: I7cb45cc40c9930840a3d4e6a86f39e1400478ed3<br>---<br>M .gitignore<br>A scripts/osmocom-build-old-tags-against-master.sh<br>2 files changed, 234 insertions(+), 0 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/.gitignore b/.gitignore</span><br><span>index 1717928..00fe73d 100644</span><br><span>--- a/.gitignore</span><br><span>+++ b/.gitignore</span><br><span>@@ -6,3 +6,5 @@</span><br><span> jenkins_jobs.ini</span><br><span> jenkins-jobs.ini</span><br><span> __pycache__/</span><br><span style="color: hsl(120, 100%, 40%);">+_temp/</span><br><span style="color: hsl(120, 100%, 40%);">+_deps/</span><br><span>diff --git a/scripts/osmocom-build-old-tags-against-master.sh b/scripts/osmocom-build-old-tags-against-master.sh</span><br><span>new file mode 100755</span><br><span>index 0000000..542785d</span><br><span>--- /dev/null</span><br><span>+++ b/scripts/osmocom-build-old-tags-against-master.sh</span><br><span>@@ -0,0 +1,232 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/bin/sh -e</span><br><span style="color: hsl(120, 100%, 40%);">+# Environment variables:</span><br><span style="color: hsl(120, 100%, 40%);">+# * PARALLEL_MAKE: -jN argument for make (default: -j5).</span><br><span style="color: hsl(120, 100%, 40%);">+# * SKIP_MASTER: don't build REPOS_MASTER (assume that they were just built and keep _temp).</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+# Latest result:</span><br><span style="color: hsl(120, 100%, 40%);">+# https://jenkins.osmocom.org/jenkins/job/Osmocom-build-tags-against-master/lastBuild/console</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+cd "$(dirname "$0")"</span><br><span style="color: hsl(120, 100%, 40%);">+. ./common.sh</span><br><span style="color: hsl(120, 100%, 40%);">+ERROR_LOGS=""</span><br><span style="color: hsl(120, 100%, 40%);">+PARALLEL_MAKE="${PARALLEL_MAKE:--j5}"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+REPOS_MASTER="</span><br><span style="color: hsl(120, 100%, 40%);">+ libosmocore</span><br><span style="color: hsl(120, 100%, 40%);">+ libosmo-abis</span><br><span style="color: hsl(120, 100%, 40%);">+ libosmo-netif</span><br><span style="color: hsl(120, 100%, 40%);">+ libosmo-sccp</span><br><span style="color: hsl(120, 100%, 40%);">+ libsmpp34</span><br><span style="color: hsl(120, 100%, 40%);">+ libasn1c</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-ggsn</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-iuh</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-hlr</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-mgw</span><br><span style="color: hsl(120, 100%, 40%);">+"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+REPOS_TAGS="</span><br><span style="color: hsl(120, 100%, 40%);">+ libosmo-abis</span><br><span style="color: hsl(120, 100%, 40%);">+ libosmocore</span><br><span style="color: hsl(120, 100%, 40%);">+ libosmo-netif</span><br><span style="color: hsl(120, 100%, 40%);">+ libosmo-sccp</span><br><span style="color: hsl(120, 100%, 40%);">+ openbsc</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-bsc</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-bts</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-ggsn</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-hlr</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-iuh</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-mgw</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-msc</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-pcu</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-sgsn</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-sip-connector</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-trx</span><br><span style="color: hsl(120, 100%, 40%);">+"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Print tags, which should be built, but where errors are expected and should be ignored.</span><br><span style="color: hsl(120, 100%, 40%);">+# This function is part of the configuration, do not insert functions above.</span><br><span style="color: hsl(120, 100%, 40%);">+# $1: repository</span><br><span style="color: hsl(120, 100%, 40%);">+# output format: one tag per line</span><br><span style="color: hsl(120, 100%, 40%);">+tags_to_ignore() {</span><br><span style="color: hsl(120, 100%, 40%);">+ case "$1" in</span><br><span style="color: hsl(120, 100%, 40%);">+ openbsc)</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.0.0" # testsuite</span><br><span style="color: hsl(120, 100%, 40%);">+ ;;</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-bsc)</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.2.1" # depends on libosmo-legacy-mgcp</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.4.0" # testsuite</span><br><span style="color: hsl(120, 100%, 40%);">+ ;;</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-mgw)</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.3.0" # testsuite</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.4.0" # testsuite</span><br><span style="color: hsl(120, 100%, 40%);">+ ;;</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-msc)</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.3.0" # -Werror and deprecated gsm0480_create_ussd_release_complete()</span><br><span style="color: hsl(120, 100%, 40%);">+ ;;</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-pcu)</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "0.5.0" # testsuite</span><br><span style="color: hsl(120, 100%, 40%);">+ ;;</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo-sgsn)</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.2.0" # sgsn_test.c: gtp.h: No such file or directory</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.3.0" # testsuite</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "1.4.0" # testsuite</span><br><span style="color: hsl(120, 100%, 40%);">+ ;;</span><br><span style="color: hsl(120, 100%, 40%);">+ esac</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Print which tags should be built for a specific repository.</span><br><span style="color: hsl(120, 100%, 40%);">+# This function is part of the configuration.</span><br><span style="color: hsl(120, 100%, 40%);">+# $1: repository</span><br><span style="color: hsl(120, 100%, 40%);">+# output format: one tag per line</span><br><span style="color: hsl(120, 100%, 40%);">+tags_to_build() {</span><br><span style="color: hsl(120, 100%, 40%);">+ case "$1" in</span><br><span style="color: hsl(120, 100%, 40%);">+ # Add repository specific tag listings here:</span><br><span style="color: hsl(120, 100%, 40%);">+ # libosmocore)</span><br><span style="color: hsl(120, 100%, 40%);">+ # echo "0.1.0"</span><br><span style="color: hsl(120, 100%, 40%);">+ # osmo_git_last_tags "$1" 3</span><br><span style="color: hsl(120, 100%, 40%);">+ # ;;</span><br><span style="color: hsl(120, 100%, 40%);">+ *)</span><br><span style="color: hsl(120, 100%, 40%);">+ osmo_git_last_tags "$1" 3</span><br><span style="color: hsl(120, 100%, 40%);">+ ;;</span><br><span style="color: hsl(120, 100%, 40%);">+ esac</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Delete existing temp dir and create a new one, output the path.</span><br><span style="color: hsl(120, 100%, 40%);">+prepare_temp_dir() {</span><br><span style="color: hsl(120, 100%, 40%);">+ TEMP="$(cd ..; pwd)/_temp"</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -n "$SKIP_MASTER" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ if ! [ -d "$TEMP" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "ERROR: SKIP_MASTER is set, but temp dir not found: $TEMP"</span><br><span style="color: hsl(120, 100%, 40%);">+ exit 1</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -d "$TEMP" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -r "$TEMP"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ mkdir -p "$TEMP/log"</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Temp dir: $TEMP"</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# When builds have failed, print the last lines of each failing log and exit with 1.</span><br><span style="color: hsl(120, 100%, 40%);">+show_errors_exit() {</span><br><span style="color: hsl(120, 100%, 40%);">+ local log</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z "$ERROR_LOGS" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for log in $ERROR_LOGS; do</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "---"</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "BUILD FAILED: $(basename "$log")"</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Showing last lines of build log below (full log in temp dir/jenkins artifacts):"</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "---"</span><br><span style="color: hsl(120, 100%, 40%);">+ tail -n 20 "$log"</span><br><span style="color: hsl(120, 100%, 40%);">+ done</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "---"</span><br><span style="color: hsl(120, 100%, 40%);">+ exit 1</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Build a repository either from master or from a specific tag against master.</span><br><span style="color: hsl(120, 100%, 40%);">+# The build output is redirected to a file, and partially shown on error.</span><br><span style="color: hsl(120, 100%, 40%);">+# $1: installation path (either $TEMP/inst_master or $TEMP/inst)</span><br><span style="color: hsl(120, 100%, 40%);">+# $2: repository</span><br><span style="color: hsl(120, 100%, 40%);">+# $3: branch, tag or commit</span><br><span style="color: hsl(120, 100%, 40%);">+# returns: 0 on sucessful build, 1 on error</span><br><span style="color: hsl(120, 100%, 40%);">+build_repo() {</span><br><span style="color: hsl(120, 100%, 40%);">+ local log="$TEMP/log/$2-$3.txt"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if ! PATH="$PWD:$PATH"\</span><br><span style="color: hsl(120, 100%, 40%);">+ PKG_CONFIG_PATH="$TEMP/inst_master/lib/pkgconfig:$PKG_CONFIG_PATH" \</span><br><span style="color: hsl(120, 100%, 40%);">+ LD_LIBRARY_PATH="$TEMP/inst_master/lib:$LD_LIBRARY_PATH" \</span><br><span style="color: hsl(120, 100%, 40%);">+ MAKE="make" \</span><br><span style="color: hsl(120, 100%, 40%);">+ PARALLEL_MAKE="$PARALLEL_MAKE" \</span><br><span style="color: hsl(120, 100%, 40%);">+ CHECK="1" \</span><br><span style="color: hsl(120, 100%, 40%);">+ deps="../_deps" \</span><br><span style="color: hsl(120, 100%, 40%);">+ inst="$1" \</span><br><span style="color: hsl(120, 100%, 40%);">+ ./osmo-build-dep.sh "$2" "$3" \</span><br><span style="color: hsl(120, 100%, 40%);">+ > "$log" 2>&1</span><br><span style="color: hsl(120, 100%, 40%);">+ then</span><br><span style="color: hsl(120, 100%, 40%);">+ return 1</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Build all configured repositories from master and install to $TEMP/inst_master.</span><br><span style="color: hsl(120, 100%, 40%);">+build_repos_master() {</span><br><span style="color: hsl(120, 100%, 40%);">+ local repo_master</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Building libraries from current master..."</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -n "$SKIP_MASTER" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "=> SKIPPED (SKIP_MASTER is set)"</span><br><span style="color: hsl(120, 100%, 40%);">+ return</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for repo_master in $REPOS_MASTER; do</span><br><span style="color: hsl(120, 100%, 40%);">+ local commit="$(osmo_git_head_commit "$repo_master")"</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z "$commit" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "ERROR: failed to get head commit for repository: $repo_master"</span><br><span style="color: hsl(120, 100%, 40%);">+ exit 1</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ printf "%-21s %s" " * $repo_master" "$commit"</span><br><span style="color: hsl(120, 100%, 40%);">+ if ! build_repo "$TEMP/inst_master" "$repo_master" "$commit"; then</span><br><span style="color: hsl(120, 100%, 40%);">+ printf "\n"</span><br><span style="color: hsl(120, 100%, 40%);">+ ERROR_LOGS="$TEMP/log/$repo_master-$commit.txt"</span><br><span style="color: hsl(120, 100%, 40%);">+ show_errors_exit</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ printf "\n"</span><br><span style="color: hsl(120, 100%, 40%);">+ done</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# $1: repository</span><br><span style="color: hsl(120, 100%, 40%);">+# $2: tag</span><br><span style="color: hsl(120, 100%, 40%);">+# returns: 0 when the error should be ignored, 1 otherwise</span><br><span style="color: hsl(120, 100%, 40%);">+ignore_error() {</span><br><span style="color: hsl(120, 100%, 40%);">+ local tag</span><br><span style="color: hsl(120, 100%, 40%);">+ for tag in $(tags_to_ignore "$1"); do</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ "$2" = "$tag" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ done</span><br><span style="color: hsl(120, 100%, 40%);">+ return 1</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Build all configured repositories on specific tags against master. The result is installed to $TEMP/inst, but deleted</span><br><span style="color: hsl(120, 100%, 40%);">+# after each build.</span><br><span style="color: hsl(120, 100%, 40%);">+build_repos_tags() {</span><br><span style="color: hsl(120, 100%, 40%);">+ local repo</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Building old tags against libraries from current master... (ERR: new error, err: known error)"</span><br><span style="color: hsl(120, 100%, 40%);">+ for repo in $REPOS_TAGS; do</span><br><span style="color: hsl(120, 100%, 40%);">+ local tags="$(tags_to_build "$repo" | sort -V | tr '\n' ' ')"</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -z "$tags" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ printf "%-21s %s\n" " * $repo" "(no tags configured)"</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ printf "%-21s" " * $repo"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ local tag</span><br><span style="color: hsl(120, 100%, 40%);">+ for tag in $tags; do</span><br><span style="color: hsl(120, 100%, 40%);">+ printf "%10s" " $tag"</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ -d "$TEMP/inst" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ rm -r "$TEMP/inst"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ if build_repo "$TEMP/inst" "$repo" "$tag"; then</span><br><span style="color: hsl(120, 100%, 40%);">+ printf " "</span><br><span style="color: hsl(120, 100%, 40%);">+ elif ignore_error "$repo" "$tag"; then</span><br><span style="color: hsl(120, 100%, 40%);">+ printf " (err)"</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ printf " (ERR)"</span><br><span style="color: hsl(120, 100%, 40%);">+ ERROR_LOGS="$ERROR_LOGS $TEMP/log/$repo-$tag.txt"</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+ done</span><br><span style="color: hsl(120, 100%, 40%);">+ printf "\n"</span><br><span style="color: hsl(120, 100%, 40%);">+ done</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+prepare_temp_dir</span><br><span style="color: hsl(120, 100%, 40%);">+build_repos_master</span><br><span style="color: hsl(120, 100%, 40%);">+build_repos_tags</span><br><span style="color: hsl(120, 100%, 40%);">+show_errors_exit</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/13415">change 13415</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/13415"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-ci </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: merged </div>
<div style="display:none"> Gerrit-Change-Id: I7cb45cc40c9930840a3d4e6a86f39e1400478ed3 </div>
<div style="display:none"> Gerrit-Change-Number: 13415 </div>
<div style="display:none"> Gerrit-PatchSet: 7 </div>
<div style="display:none"> Gerrit-Owner: osmith <osmith@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Harald Welte <laforge@gnumonks.org> </div>
<div style="display:none"> Gerrit-Reviewer: Max <msuraev@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Pau Espin Pedrol <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: osmith <osmith@sysmocom.de> </div>