[MERGED] osmo-ci[master]: add osmo-clean-workspace.sh, use in osmo-deps.sh

Neels Hofmeyr gerrit-no-reply at lists.osmocom.org
Fri Oct 27 23:59:53 UTC 2017


Neels Hofmeyr has submitted this change and it was merged.

Change subject: add osmo-clean-workspace.sh, use in osmo-deps.sh
......................................................................


add osmo-clean-workspace.sh, use in osmo-deps.sh

So far, each jenkins job does its own cleanup, more or less well. Also, jenkins
git config offers the 'Clean before checkout' option, which seems to fail when
there are non-writable leftovers from a failed 'make distcheck'.

Furthermore, our jenkins build slaves have unused compiled binaries piling up
by the gigabytes: each matrix build x each parallel build and each compiled
dependency therein builds .o, .a, .so and executables plus installs them to a
local prefix, and just leaves them sitting around to rot until the job runs
again. Instead, we want to clean them out when building is done.

All of this calls for a unified cleanup script that knows how to clean a
workspace properly, to run once before and once after each jenkins build.
Here it is.

Use that function in osmo-deps.sh instead of duplicating cleanup steps.

Change-Id: I2409b2928b4d7ebbd6c005097d4ad7337307dd93
---
M scripts/osmo-build-dep.sh
A scripts/osmo-clean-workspace.sh
M scripts/osmo-deps.sh
3 files changed, 48 insertions(+), 1 deletion(-)

Approvals:
  Neels Hofmeyr: Verified
  Harald Welte: Looks good to me, approved



diff --git a/scripts/osmo-build-dep.sh b/scripts/osmo-build-dep.sh
index 2a107ed..8104ba7 100755
--- a/scripts/osmo-build-dep.sh
+++ b/scripts/osmo-build-dep.sh
@@ -40,7 +40,6 @@
 
 mkdir -p "$deps"
 cd "$deps"
-rm -rf "$project"
 osmo-deps.sh "$project"
 cd "$project"
 if [ -n "$branch" ]; then
diff --git a/scripts/osmo-clean-workspace.sh b/scripts/osmo-clean-workspace.sh
new file mode 100755
index 0000000..11a5b72
--- /dev/null
+++ b/scripts/osmo-clean-workspace.sh
@@ -0,0 +1,42 @@
+#!/bin/sh
+# Clean workspace.
+# This should be the first and last step of every jenkins build:
+# a) to make sure the workspace has no build artifacts from previous runs.
+# b) to reduce disk space lost to unused binaries; parallel and/or matrix
+#    builds create numerous workspaces, blowing up disk usage.
+# Note that if a build fails, the last steps will not run, hence calling this
+# as last step cleans only in case there was no build failure, i.e. where we
+# don't need to keep anything anyway.
+#
+# Assume $PWD is a git clone's root dir. Usually, that's also the jenkins
+# workspace root. Do not wipe subdir 'layer1-headers' as well as all dirs under
+# '$deps'. These are assumed to be git clones that do not need to be re-cloned
+# every time. Do a 'git clean' in each of them individually. If '$deps' is not
+# defined or the mentioned dirs do not exist, nothing special happens, so this
+# script can be safely run in any git clone in deeper subdirs of the workspace.
+
+set -ex
+
+# make sure no write protected cruft is in the way. A failed 'make distcheck'
+# has a habit of leaving a non-writable dir tree behind.
+chmod -R +w .
+
+# wipe all local modifications
+git checkout -f HEAD
+
+# wipe all unversioned leftovers, except deps gits.
+git clean -dxf -e "$deps" -e "layer1-headers"
+
+# leave the deps checkouts around, to not clone entire git history every time,
+# but clean each git of build artifacts.
+if [ -d "$deps" ]; then
+  for dep_dir in "$deps"/* ; do
+    git checkout -f HEAD
+    git -C "$dep_dir" clean -dxf
+  done
+fi
+
+if [ -d "layer1-headers" ]; then
+  git checkout -f HEAD
+  git -C "layer1-headers" clean -dxf
+fi
diff --git a/scripts/osmo-deps.sh b/scripts/osmo-deps.sh
index 86da9a6..c256a6a 100755
--- a/scripts/osmo-deps.sh
+++ b/scripts/osmo-deps.sh
@@ -8,5 +8,11 @@
 
 cd $1
 git fetch origin
+
+# Cleanup should already have happened during a global osmo-clean-workspace.sh,
+# but in case the caller did not (want to) call that, let's also do cleanup in
+# the dep subdir separately:
+osmo-clean-workspace.sh
+
 git reset --hard origin/master
 git rev-parse HEAD

-- 
To view, visit https://gerrit.osmocom.org/4467
To unsubscribe, visit https://gerrit.osmocom.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2409b2928b4d7ebbd6c005097d4ad7337307dd93
Gerrit-PatchSet: 1
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Owner: Neels Hofmeyr <nhofmeyr at sysmocom.de>
Gerrit-Reviewer: Harald Welte <laforge at gnumonks.org>
Gerrit-Reviewer: Neels Hofmeyr <nhofmeyr at sysmocom.de>


More information about the gerrit-log mailing list