dependency artifacts for faster builds/gerrit-verifications
Hi all, here's a (late) recap on the "dependency-artifacts" topic discussed with Holger and Neels at OsmoDevCon. The "mv $artifact"-issue, which occurs when run within a docker container has been resolved. Now artifacts are created inside a temporary directory within the ARTIFACT_STORE and not inside the Docker container, so the "mv" command won't take as long as a "cp" anymore. Moreover, artifacts are now stored per jenkins job and not per git project. A JOB_NAME-directory inside the ARTIFACT_STORE is created for each jenkins job to distinguish between them. This means artifacts aren't shared across jobs which build the same git project. This might change in the future, but is safe for now as some jenkins jobs might need dependencies built with different configurations. Note: the necessary disk space for openBSC artifacts of one matrix-configuration job is ~400 MB (~50 MB per axis). Furthermore, each jenkins job takes care about garbage collection on its own by simply deleting the entire content of its JOB_NAME-directory inside the ARTIFACT_STORE. The two scripts osmo-{build,artifacts}.sh have been tested [1][2], shellcheck'ed and pushed to gerrit [3]. The following actions are pending to enable the use of artifacts inside contrib/jenkins.sh within each git repository, but can not be applied by me: - setting ARTIFACT_STORE environment variable for each jenkins slave - copying osmo-artifact.sh and osmo-build.sh to ~/bin of each slave user - adding a new docker volume for the ARTIFACT_STORE to the image But, let's review first. =) Cheers, André [1] https://jenkins.blobb.me/job/openBSC_multi-configuration_withArtifacts_testR... [2] https://jenkins.blobb.me/job/openBSC_multi-configuration_withArtifacts_testR... [3] https://gerrit.osmocom.org/#/c/2465/
On Tue, May 02, 2017 at 07:59:35PM +0200, André Boddenberg wrote:
Hi all,
here's a (late) recap on the "dependency-artifacts" topic discussed with Holger and Neels at OsmoDevCon.
The "mv $artifact"-issue, which occurs when run within a docker container has been resolved. Now artifacts are created inside a temporary directory within the ARTIFACT_STORE and not inside the Docker container, so the "mv" command won't take as long as a "cp" anymore.
Moreover, artifacts are now stored per jenkins job and not per git project. A JOB_NAME-directory inside the ARTIFACT_STORE is created for each jenkins job to distinguish between them. This means artifacts aren't shared across jobs which build the same git project. This might change in the future, but is safe for now as some jenkins jobs might need dependencies built with different configurations.
Note: the necessary disk space for openBSC artifacts of one matrix-configuration job is ~400 MB (~50 MB per axis).
Furthermore, each jenkins job takes care about garbage collection on its own by simply deleting the entire content of its JOB_NAME-directory inside the ARTIFACT_STORE.
Which is, for the record, the reason to have it per-job: we always know which artifacts need to be / we are allowed to clean up. With multiple jobs accessing the same artifact store, GC is not trivial.
The two scripts osmo-{build,artifacts}.sh have been tested [1][2], shellcheck'ed and pushed to gerrit [3].
nice!
The following actions are pending to enable the use of artifacts inside contrib/jenkins.sh within each git repository, but can not be applied by me:
- setting ARTIFACT_STORE environment variable for each jenkins slave - copying osmo-artifact.sh and osmo-build.sh to ~/bin of each slave user
Add to osmo-ci. Could you submit a patch for it? git clone https://gerrit.osmocom.org/osmo-ci
- adding a new docker volume for the ARTIFACT_STORE to the image
We're talking about the docker invocation in the OpenBSC and OpenBSC-gerrit jobs.
But, let's review first. =)
I notice there are hashes of various dependent builds in the tar names: libosmocore.master.33e0306_libosmo-abis.master.bf7976c_libosmo-netif.master.56add1e_libosmo-sccp.master.b354652_libsmpp34.master.cc0bcd6_openggsn.master.19e19e3.tar.gz I remember that we said to instead have each on its own -- but not saying we have to, just curious. So the semantics are: figure out all of the hashes of all dependencies, as soon as one of them mismatches rebuild all and make a new single tar with all included. Right? Also curious about how a matrix build would do it: would every matrix rebuild the same dependency? (We said that that's fine IIRC) ~N
On Thu, May 04, 2017 at 05:52:28PM +0200, Neels Hofmeyr wrote:
On Tue, May 02, 2017 at 07:59:35PM +0200, André Boddenberg wrote:
The following actions are pending to enable the use of artifacts inside contrib/jenkins.sh within each git repository, but can not be applied by me:
- setting ARTIFACT_STORE environment variable for each jenkins slave - copying osmo-artifact.sh and osmo-build.sh to ~/bin of each slave user
Add to osmo-ci. Could you submit a patch for it? git clone https://gerrit.osmocom.org/osmo-ci
Of course that's https://gerrit.osmocom.org/2465 ~N
Hi Neels, thanks for adding missing details for those who haven't participated in our f2f conversation!
I notice there are hashes of various dependent builds in the tar names:
libosmocore.master.33e0306_libosmo-abis.master.bf7976c_libosmo-netif.master.56add1e_libosmo-sccp.master.b354652_libsmpp34.master.cc0bcd6_openggsn.master.19e19e3.tar.gz
I remember that we said to instead have each on its own -- but not saying we have to, just curious.
That would be great, but I don't know whether we could simply fetch and inject e.g. "libosmo-netif.master.b4aed1e.tar.gz" to update an artifact which is up-to-date except libosmo-netif, because libosmo-netif depends on libosmo-abis, which depends itself on libosmocore, correct? If correct an artifact for libosmo-netif would need to hold its dependencies in its name as well, because libosmo-netif may didn't change, but its dependencies may did!? Ive created a diagram [1] showing openBSC dependencies and the dependencies of them to illustrate above-stated. As I see, libsmpp34 would be the only openBSC dependency that could be simply injected? Additionally, I created a second diagram [2] that shows a bit of brainstorming. In this thought experiment openBSC depends on the following three "sub-artifacts": - libosmocore.master.33e0306_libosmo-abis.master.bf7976c_libosmo-netif.master.56add1e_libosmo-sccp.master.b354652 - libosmocore.master.33e0306_openggsn.master.19e19e3.tar.gz - libsmpp34.master.cc0bcd6 , which then can be extracted before triggering actual openBSC build and/or merging all "sub-artifacts" archives to: - libosmocore.master.33e0306_libosmo-abis.master.bf7976c_libosmo-netif.master.56add1e_libosmo-sccp.master.b354652_libsmpp34.master.cc0bcd6_openggsn.master.19e19e3.tar.gz In general, I'd say such improvement is definitely worth to investigate. Furthermore, it might be a good approach to start with libosmocore as a first project that exposes artifact(s) for depending projects e.g. libosmo-abis and openggsn. Such mechanism would be much more elegant as the current, in fact right now all projects are only carrying about their own necessary dependencies and that's probably a bit too selfish ;) Note: *.dot files of diagrams can be accessed by simply replacing 'png' with 'dot' in URLs.
So the semantics are: figure out all of the hashes of all dependencies, as soon as one of them mismatches rebuild all and make a new single tar with all included. Right?
Yes.
Also curious about how a matrix build would do it: would every matrix rebuild the same dependency? (We said that that's fine IIRC)
Yes, every matrix/axes-combination will build its own artifact for the sake of a simple GC. But at the same time it's probably a good place for further improvements à la axes-combination share same ARTIFACT_STORE folder, which could be easily achieved by changing line 86 in osmo-build.sh [3] to: $ jobName=$(echo "$JOB_NAME" | cut -d '/' -f1 ...just brainstorming. =) [1] https://blobb.me/osmocom/openBSC_allDepsAndTheirDeps.png [2] https://blobb.me/osmocom/openBSC_smartDeps.png [3] https://gerrit.osmocom.org/#/c/2465/6/scripts/osmo-build.sh
Out of curiosity: On 05.05.2017 17:19, André Boddenberg wrote:
Ive created a diagram [1] showing openBSC dependencies and the dependencies of them to illustrate above-stated.
Is this manual work or you're extracting dependency info from configure.ac or debian/? -- Max Suraev <msuraev@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
Out of curiosity:
On 05.05.2017 17:19, André Boddenberg wrote:
Ive created a diagram [1] showing openBSC dependencies and the dependencies of them to illustrate above-stated.
Is this manual work or you're extracting dependency info from configure.ac or debian/?
It's manually work and actually my first diagram made with dotty (nice tool!). I simply went through openBsc' and its dependencies' contrib/jenkins.sh files to create the diagrams. How would you extract information about dependencies from configure.ac? I would probably curl http://git.osmocom.org/${project}/plain/contrib/jenkins.sh and grep+cut all strings occurring after osmo-build-dep.sh and recursively do the same for all found dependencies. Do you in general think an automated way of creating dotty dependency graph would be nice to have?
On 05.05.2017 17:48, André Boddenberg wrote:
>
> How would you extract information about dependencies from configure.ac?
PKG_CHECK_MODULES() entries should be good enough. There's probably some autotools
trick I don't know which would allow to get those automatically without grepping
through. Alternatively one can look in debian/control for Build-Depends entries. Not
sure if Debian got any dh_* which could extract this in a nice way.
> I would probably curl
> http://git.osmocom.org/${project}/plain/contrib/jenkins.sh and
> grep+cut all strings occurring after osmo-build-dep.sh and recursively
> do the same for all found dependencies.
>
> Do you in general think an automated way of creating dotty dependency
> graph would be nice to have?
I wouldn't bother just for dotty. Having dependencies part of jenkins jobs to be
autogenerated would be kinda cool but:
- not sure if it's not overengineering: dependencies do not change that often
- not sure how many false positives/negatives the approaches above give
--
Max Suraev <msuraev@sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
Hi Neels, I'd like to reanimate this thread. The gerrit review is still pending [1]. Once it's through, we can start enabling the "(re)builds using artifacts"-feature for OpenBSC and other build jobs. Regards, André [1] https://gerrit.osmocom.org/#/c/2465/
participants (3)
-
André Boddenberg -
Max -
Neels Hofmeyr