dependency artifacts for faster builds/gerrit-verifications

historical

On Tue, May 02, 2017 at 07:59:35PM +0200, André Boddenberg wrote:
> Hi all,
> 
> here's a (late) recap on the "dependency-artifacts" topic discussed
> with Holger and Neels at OsmoDevCon.
> 
> 
> The "mv $artifact"-issue, which occurs when run within a docker
> container has been resolved. Now artifacts are created inside a
> temporary directory within the ARTIFACT_STORE and not inside the
> Docker container, so the "mv" command won't take as long as a "cp"
> anymore.
> 
> Moreover, artifacts are now stored per jenkins job and not per git
> project. A JOB_NAME-directory inside the ARTIFACT_STORE is created for
> each jenkins job to distinguish between them. This means artifacts
> aren't shared across jobs which build the same git project. This might
> change in the future, but is safe for now as some jenkins jobs might
> need dependencies built with different configurations.
> 
> Note: the necessary disk space for openBSC artifacts of one
> matrix-configuration job is ~400 MB (~50 MB per axis).
> 
> Furthermore, each jenkins job takes care about garbage collection on
> its own by simply deleting the entire content of its
> JOB_NAME-directory inside the ARTIFACT_STORE.

Which is, for the record, the reason to have it per-job: we always know
which artifacts need to be / we are allowed to clean up. With multiple
jobs accessing the same artifact store, GC is not trivial.

> The two scripts osmo-{build,artifacts}.sh have been tested [1][2],
> shellcheck'ed and pushed to gerrit [3].

nice!

> The following actions are pending to enable the use of artifacts
> inside contrib/jenkins.sh within each git repository, but can not be
> applied by me:
> 
>   - setting ARTIFACT_STORE environment variable for each jenkins slave
>   - copying osmo-artifact.sh and osmo-build.sh to ~/bin of each slave user

Add to osmo-ci. Could you submit a patch for it?
git clone https://gerrit.osmocom.org/osmo-ci

>   - adding a new docker volume for the ARTIFACT_STORE to the image

We're talking about the docker invocation in the OpenBSC and
OpenBSC-gerrit jobs.

> But, let's review first. =)

I notice there are hashes of various dependent builds in the tar names:

libosmocore.master.33e0306_libosmo-abis.master.bf7976c_libosmo-netif.master.56add1e_libosmo-sccp.master.b354652_libsmpp34.master.cc0bcd6_openggsn.master.19e19e3.tar.gz

I remember that we said to instead have each on its own -- but not saying
we have to, just curious.

So the semantics are: figure out all of the hashes of all dependencies, as
soon as one of them mismatches rebuild all and make a new single tar with
all included. Right?

Also curious about how a matrix build would do it: would every matrix
rebuild the same dependency? (We said that that's fine IIRC)

~N
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20170504/6bc7da71/attachment.bin>