osmo-ci[master]: Introduce artifacts holding dependencies to speed up builds.

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/gerrit-log@lists.osmocom.org/.

Neels Hofmeyr gerrit-no-reply at lists.osmocom.org
Fri May 19 11:24:25 UTC 2017


Patch Set 7: Code-Review-1

(12 comments)

thanks for working on this! Here is some more review...

https://gerrit.osmocom.org/#/c/2465/7/scripts/osmo-build.sh
File scripts/osmo-build.sh:

Line 23: # More information about those function are given in the following template:
This is an example for building <insert project name> which depends on <a>, <b>, <c>


Line 28: #		# Specify dependencies in the following way:
More and better docs. It is absolutely crucial that anyone who may take over jenkins maintenance can easily understand this. e.g.

  Pass all required dependencies to a command given in $1.
  Collect the output of these calls and concatenate, separated with underscores.
  The result is an artifact file name used to store in/retrieve from the artifact store.

  $1 may be one of these commands:

  - osmo-build-dep.sh: build a source tree
  - getFoo: frobnicate zagnomodes
  - getMoo: instacube labnitudes

  The arguments passed to these are:
  
  - PARALLEL_MAKE as environment variable, passed to 'make'
  - arg 1: the git repository name, gets places in a git.osmocom.org URL.
  - arg 2: optional: ...
  - arg 3: optional: ...


Line 43: #		#	- getArtifactNameRemotely()  (osmo-build.sh)
(rather put this on top like above)


Line 49: #		# in jenkins' $WORKSPACE per default.
like what. cd foo? ./configure && make install?


Line 55: # The JOB_NAME variables will be injected to each jenkins' job by jenkins itself.
let's have what this script needs right at the top


Line 82: 
comment: "jenkins should check out the git tree of the project to be built in the workspace root." (this probably near the top of this file) and here "obtain the project name from the git clone found in the workspace root"


Line 86: 	job_name="${JOB_NAME//\//#}"
what does this do?


Line 130: 
locally? remotely?


Line 163: 	# be on a different partition or build runs in docker.
you're lacking the root reason: we want to atomically move an artifact into place avoiding half-written files.


Line 171: 		rm -f "$job_store/*"
hmm, we might still have a GC problem here. If another job tries to use this artifact and this job is removing it? -- that depends on whether we retrieve the artifact "atomically"...


Line 187: 	generateArtifactHashes "$1"
add comment:

  # extract the artifact directly from the artifact store.
  # If another job run were to remove the artifact while we're doing this,
  # we trust that 'tar' will keep the file open and thus we are allowed 
  # to read to the end before the file is discarded by the OS.

And about the GC problem above: the most waterproof way of avoiding is to probe whether the artifact exists *by* doing a copy/untar. If it succeeds, we have it. If not, rebuild. In case we first query and decide, then copy, some other job run may have removed in the meantime?

How likely is this to happen? What would it take? concurrent jenkins job builds of different versions of the same repos? (so far we have concurrent builds disabled mostly for cosmetic reasons)


Line 193: generateArtifactHashes() {
rename to 'logArtifactHashes'


-- 
To view, visit https://gerrit.osmocom.org/2465
To unsubscribe, visit https://gerrit.osmocom.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifee0a2f837d23b19aa5326f810234d5452e47484
Gerrit-PatchSet: 7
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Owner: blobb <dr.blobb at gmail.com>
Gerrit-Reviewer: Harald Welte <laforge at gnumonks.org>
Gerrit-Reviewer: Holger Freyther <holger at freyther.de>
Gerrit-Reviewer: Neels Hofmeyr <nhofmeyr at sysmocom.de>
Gerrit-Reviewer: blobb <dr.blobb at gmail.com>
Gerrit-Reviewer: neels <nhofmeyr at sysmocom.de>
Gerrit-HasComments: Yes



More information about the gerrit-log mailing list