On Tue, May 02, 2017 at 07:59:35PM +0200, André Boddenberg wrote:
Hi all,
here's a (late) recap on the "dependency-artifacts" topic discussed with Holger and Neels at OsmoDevCon.
The "mv $artifact"-issue, which occurs when run within a docker container has been resolved. Now artifacts are created inside a temporary directory within the ARTIFACT_STORE and not inside the Docker container, so the "mv" command won't take as long as a "cp" anymore.
Moreover, artifacts are now stored per jenkins job and not per git project. A JOB_NAME-directory inside the ARTIFACT_STORE is created for each jenkins job to distinguish between them. This means artifacts aren't shared across jobs which build the same git project. This might change in the future, but is safe for now as some jenkins jobs might need dependencies built with different configurations.
Note: the necessary disk space for openBSC artifacts of one matrix-configuration job is ~400 MB (~50 MB per axis).
Furthermore, each jenkins job takes care about garbage collection on its own by simply deleting the entire content of its JOB_NAME-directory inside the ARTIFACT_STORE.
Which is, for the record, the reason to have it per-job: we always know which artifacts need to be / we are allowed to clean up. With multiple jobs accessing the same artifact store, GC is not trivial.
The two scripts osmo-{build,artifacts}.sh have been tested [1][2], shellcheck'ed and pushed to gerrit [3].
nice!
The following actions are pending to enable the use of artifacts inside contrib/jenkins.sh within each git repository, but can not be applied by me:
- setting ARTIFACT_STORE environment variable for each jenkins slave
- copying osmo-artifact.sh and osmo-build.sh to ~/bin of each slave user
Add to osmo-ci. Could you submit a patch for it? git clone https://gerrit.osmocom.org/osmo-ci
- adding a new docker volume for the ARTIFACT_STORE to the image
We're talking about the docker invocation in the OpenBSC and OpenBSC-gerrit jobs.
But, let's review first. =)
I notice there are hashes of various dependent builds in the tar names:
libosmocore.master.33e0306_libosmo-abis.master.bf7976c_libosmo-netif.master.56add1e_libosmo-sccp.master.b354652_libsmpp34.master.cc0bcd6_openggsn.master.19e19e3.tar.gz
I remember that we said to instead have each on its own -- but not saying we have to, just curious.
So the semantics are: figure out all of the hashes of all dependencies, as soon as one of them mismatches rebuild all and make a new single tar with all included. Right?
Also curious about how a matrix build would do it: would every matrix rebuild the same dependency? (We said that that's fine IIRC)
~N