[PATCH] osmo-ci[master]: Introduce artifacts holding dependencies to speed up builds.

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/gerrit-log@lists.osmocom.org/.

blobb gerrit-no-reply at lists.osmocom.org
Fri May 5 19:06:01 UTC 2017


Hello Neels Hofmeyr, Harald Welte,

I'd like you to reexamine a change.  Please visit

    https://gerrit.osmocom.org/2465

to look at the new patch set (#7).

Introduce artifacts holding dependencies to speed up builds.

Basically, osmo-build.sh holds logic to check whether the necessary
artifact is available. If so it fetches artifact, unpacks it and
triggers the actual build. In case the necessary artifact is not
available osmo-build.sh simply builds all dependencies from source
by using osmo-build-dep.sh and archives deps to the ARTIFACT_STORE
afterwards.

The necessary functions to determine the artifact name from remote and
local repositories as well as the handling of artifact files live in
osmo-artifacts.sh, which is sourced by osmo-build.sh.

osmo-build.sh will be sourced by the contrib/jenkins.sh build script
inside each git repository and invoked via 'build'.
See jenkins-openBsc.sh [1] for more details.

Artifacts will be stored as follows:

	$ARTIFACT_STORE/$JOB_NAME/<dep_1>.<branch_1>.<rev_1>_...
		..._<dep_n>.<branch_n>.<rev_n>.tar.gz

Furthermore, ARTIFACT_STORE environment variable has to be set on all
jenkins slaves. The JOB_NAME variables is injected to each jenkins job
by jenkins.

[1] https://github.com/blobbsen/diy-artifacts/blob/master/jenkins-openBSC.sh

Change-Id: Ifee0a2f837d23b19aa5326f810234d5452e47484
---
A scripts/osmo-build.sh
1 file changed, 200 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/65/2465/7

diff --git a/scripts/osmo-build.sh b/scripts/osmo-build.sh
new file mode 100644
index 0000000..f58ec15
--- /dev/null
+++ b/scripts/osmo-build.sh
@@ -0,0 +1,200 @@
+#!/bin/bash
+#
+# This script enables artifacts holding dependencies on a jenkins job level to
+# speed up builds. Basically, it holds logic to check whether the necessary artifact
+# is available. If so it fetches artifact, unpacks it and triggers the actual build.
+#
+# Otherwise it simply builds all dependencies from source by using osmo-build-dep.sh
+# and archives deps to the ARTIFACT_STORE afterwards. Revisions of locally built
+# dependencies are detrmined after dependencies are built to ensure catching new
+# changes in dep_n+1 meanwhile dep_n building.
+#
+# Artifacts will be stored as follows:
+#	$ARTIFACT_STORE/$JOB_NAME/<dep_1>.<branch_1>.<rev_1>_...
+#		..._<dep_n>.<branch_n>.<rev_n>.tar.gz
+#
+# In order to make use of osmo-build.sh one needs to source it, e.g. from
+# ./contrib/jenkins.sh. Furthermore you need to declare following functions
+# within a build script that sources osmo-build.sh:
+#
+#	- genericDeps()
+#	- buildProject()
+#
+# More information about those function are given in the following template:
+#
+#	source osmo-build.sh
+#
+#	genericDeps() {
+#		# Specify dependencies in the following way:
+#
+#		x="$($1 libosmocore master ac_cv_path_DOXYGEN=false)"
+#		x="${x}_$(PARALLEL_MAKE=-j1 $1 libosmo-abis)"
+#		x="${x}_$($1 libosmo-netif)"
+#
+#		echo "${x}.tar.gz"
+#
+#		# The following 'parameters' can be adjusted within the genericDeps function:
+#		#
+#		# x="${x}_$(<PARALLEL_MAKE> $1 <DEPENDENCY/REPO> <BRANCH(default:master)> <CFG>"
+#		#
+#		# Furthermore, $1 represents the following script/functions:
+#		#	- osmo-build-dep.sh
+#		#	- getArtifactNameLocally()   (osmo-build.sh)
+#		#	- getArtifactNameRemotely()  (osmo-build.sh)
+#	}
+#
+#	buildProject() {
+#		# Necessary commands to build the project, expecting all dependencies have
+#		# been build or fetched. Commands within buildProject() will be executed
+#		# in jenkins' $WORKSPACE per default.
+#	}
+#
+#	build
+#
+# Furthermore, ARTIFACT_STORE environment variable has to be set on all jenkins slaves.
+# The JOB_NAME variables will be injected to each jenkins' job by jenkins itself.
+
+# BUILD FUNCTIONS
+
+initBuild() {
+
+	if [ -z "$JOB_NAME" ]; then
+		set +x
+		echo
+		echo "[ERROR] JOB_NAME variable is not set, running in Jenkins?"
+		echo
+		set -x
+		exit 1
+	fi
+
+	if [ -z "$ARTIFACT_STORE" ]; then
+		set +x
+		echo
+		echo "[ERROR] ARTIFACT_STORE variable is not set on this build slave"
+		echo
+		set -x
+		exit 1
+	fi
+
+	base="$(pwd)"
+	deps="$base/deps"
+	inst="$deps/install"
+
+	project=$(git config --get --local remote.origin.url \
+		| cut -d '/' -f4 | cut -d '.' -f1)
+
+	job_name="${JOB_NAME//\//#}"
+
+	export base deps inst project job_name
+	export PKG_CONFIG_PATH="$inst/lib/pkgconfig:$PKG_CONFIG_PATH"
+	export LD_LIBRARY_PATH="$inst/lib"
+}
+
+buildDeps() {
+	set +x
+	echo
+	echo "[INFO] Compile $project dependencies from source."
+	echo
+	set -x
+
+	mkdir -p "$deps"
+	rm -rf "$inst"
+
+	genericDeps "osmo-build-dep.sh"
+}
+
+build() {
+
+	initBuild
+
+	needed_artifact="$(getArtifactNameRemotely)"
+	needed_artifact_path="$ARTIFACT_STORE/$job_name/$needed_artifact"
+
+	if [ -f "$needed_artifact_path" ]; then
+		fetchArtifact "$needed_artifact_path"
+	else
+		buildDeps
+		archiveArtifact
+	fi
+
+	set +x
+	echo
+	echo " ============================= $project =========================="
+	echo
+	set -x
+
+	buildProject
+}
+
+# ARTIFACT FUNCTIONS
+
+getArtifactNameLocally() {
+	genericDeps "getBranchAndRevLocally"
+	cd "$base"
+}
+
+getArtifactNameRemotely() {
+	genericDeps "getBranchAndRevRemotely"
+}
+
+getBranchAndRevLocally() {
+	cd "$deps/$1"
+	rev=$(git rev-parse --short HEAD)
+	branch=$(git rev-parse --abbrev-ref HEAD)
+	echo "$1.${branch//\//#}.${rev}"
+}
+
+getBranchAndRevRemotely() {
+	if [ -z "${2+x}" ]; then branch="master"; else branch="$2"; fi
+	rev=$(git ls-remote "https://git.osmocom.org/$1" "refs/heads/$branch")
+	echo "$1.${branch//\//#}.${rev:0:7}"
+}
+
+archiveArtifact() {
+	set +x
+	echo
+	echo "[INFO] Archiving artifact to artifactStore."
+	echo
+	set -x
+
+	cd "$base"
+	artifact="$(getArtifactNameLocally)"
+	# temp_job_store is necessary since WORKSPACE might
+	# be on a different partition or build runs in docker.
+	temp_job_store="$ARTIFACT_STORE/tmp/$job_name/"
+	job_store="$ARTIFACT_STORE/$job_name/"
+
+	if [ ! -f "$temp_job_store/$artifact" ]; then
+		mkdir -p "$job_store" "$temp_job_store"
+		# remove outdated artifact first to avoid temporarily
+		# doubling of artifact storage consumption
+		rm -f "$job_store/*"
+		tar czf "$temp_job_store/$artifact" "deps"
+		mv -n "$temp_job_store/$artifact" "$job_store/$artifact"
+		rm -rf "$temp_job_store"
+
+		generateArtifactHashes "$job_store/$artifact"
+	fi
+}
+
+fetchArtifact() {
+	set +x
+	echo
+	echo "[INFO] Fetching artifact from artifactStore."
+	echo
+	set -x
+
+	generateArtifactHashes "$1"
+	tar xzf "$1"
+}
+
+# checksum is not used by this script itself,
+# but might be handy in logs when debugging.
+generateArtifactHashes() {
+	set +x
+	echo
+	echo "[INFO] name: $1"
+	echo "[INFO] sha256: $(sha256sum "$1" | cut -d ' ' -f1)"
+	echo
+	set -x
+}

-- 
To view, visit https://gerrit.osmocom.org/2465
To unsubscribe, visit https://gerrit.osmocom.org/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifee0a2f837d23b19aa5326f810234d5452e47484
Gerrit-PatchSet: 7
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Owner: blobb <dr.blobb at gmail.com>
Gerrit-Reviewer: Harald Welte <laforge at gnumonks.org>
Gerrit-Reviewer: Holger Freyther <holger at freyther.de>
Gerrit-Reviewer: Neels Hofmeyr <nhofmeyr at sysmocom.de>
Gerrit-Reviewer: blobb <dr.blobb at gmail.com>
Gerrit-Reviewer: neels <nhofmeyr at sysmocom.de>



More information about the gerrit-log mailing list