Hi,
some weeks a go the topic about using dependency-artifacts to speed up gerrit verifications was mentioned by Neels in the "RFC: jenkins pipeline" thread [1].
I was curious and wrote a bash script (osmo-artifacts.sh) [2], which enables build jobs to archive and fetch dependencies. Instead of Docker it simply uses archives to store them. An artifact's name has the following form:
( n(${project}.${branch}.${shortRev}_) ).tar.gz
Openbsc builds are 50 % faster when using artifacts. [3]
The jenkins.sh [2] script had to be slightly modified i.e. basically split into two functions:
1) buildProjectDeps() 2) buildProject()
A third function to generate the artifact name has been added. The template.sh [2] script illustrates this description.
What do you think about the result/artifactStore/approach in general?
A cleanUpArtifactStore.sh script running as a cron job is already on my mind, but imo it's a good point to discuss the intermediate result.
blobb
[1] http://lists.osmocom.org/pipermail/openbsc/2017-March/010400.html [2] git clone https://github.com/blobbsen/diy-artifacts.git [3] https://jenkins.blobb.me/job/openBSC_multi-configuration_withArtifacts_testR...
Hi Andre,
just a heads-up about my complete absence from jenkins/CI related discussions: That doesn't mean I'm not interested, it just means that I'm happy to defer the decisions on this to those who understand anything about it (which I don't claim to), i.e. currently most likely Neels and Holger.
On 30. Mar 2017, at 10:17, Harald Welte laforge@gnumonks.org wrote:
Hi Andre,
Hey!
just a heads-up about my complete absence from jenkins/CI related discussions: That doesn't mean I'm not interested, it just means that I'm happy to defer the decisions on this to those who understand anything about it (which I don't claim to), i.e. currently most likely Neels and Holger.
I was very happy to see it. One nice aspect of running in a namespace is that all jobs can install to /somedir without interfering with each other. This makes caching intermediate results easy and as shown by Andre a nice speed-up!
I try to look at it over the next couple of days. My uneducated questions right now would be. Why not use the archive artifact feature of jenkins? Why do we need to keep more than one? In general we build all software against master of libraries (e.g. even for the 3G support).
But this feature is something we want, no need to rebuild software that has not changed. :)
holger
Why not use the archive artifact feature of jenkins?
True, archiving artifacts comes ootb, but the functionality to copy/fetch artifacts from another buildjob/project as a build step has to be installed via plugin [1].
Would you consider installing the copyArtifact plugin?
In case installing such plugin is not an option one can simply use:
"wget $URL_OF_LATEST_SUCCESSFUL_ARTIFACT" [2]
to fetch artifacts. Such command can be put in an additional "execute shell" build step, so it wouldn't touch any build script...
But a matrixConfigurationJob [3] archives artifacts per axis, so jenkins would store 8 artifacts [4]. Unfortunately the "archive artifacts" post-build-step cannot be triggered by a specific axis configuration (afaik).
A quick brainstorming how to change this behavior ended in:
- additional "execute shell" step evaluating its axes configuration and delete artifact accordingly to avoid redundant archiving.
- change "archive artifact" "post build step" to not fail build if no artifact could be archived. Unfortunately this configuration applies for all axes, so when archiving of the "chosen" artifact fails no one will complain.
Furthermore the "latest successful artifact" might already be outdated when fetching it. This happens every time when a openbsc [5] build is triggered by an "upstream project" until its artifact has been archived. So a mechanism is needed to verify whether fetched artifact is still valid and compiling all dependencies elsewise.
At this point I just got the feeling that writing a custom script and using a directory as an artifact store may will be much more straight forward. I agrre, that this directory approach might be to prototyp'ish.
But let's discuss it. We can change everything. :)
Why do we need to keep more than one? In general we build all software against master of libraries (e.g. even for the 3G support).
Good question! If one artifact is sufficient, let's archive only one. (sure thing)
But according to contrib/jenkins.sh in openbsc repo, some non-master branches (of libosmo-(netif && sccp)) are used in all axes where "IU" is enabled:
20) if [ "x$IU" = "x--enable-iu" ]; then 21) netif_branch="sysmocom/sctp" 22) sccp_branch="sysmocom/iu" 23) fi -- 26) osmo-build-dep.sh libosmo-netif $netif_branch 27) osmo-build-dep.sh libosmo-sccp $sccp_branch
To be honest I'm a bit confused or am I just missing something?
blobb
[1] https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin [2] wget https://jenkins.blobb.me/job/matrixJobArtifactHandlingExample/HELLO=Salut,WO... [3] https://jenkins.blobb.me/job/matrixJobArtifactHandlingExample [4] https://jenkins.blobb.me/job/matrixJobArtifactHandlingExample/HELLO=Salut,WO... [5] http://jenkins.osmocom.org/jenkins/view/OpenBSC/job/OpenBSC/
2017-03-30 17:20 GMT+02:00 Holger Freyther holger@freyther.de:
On 30. Mar 2017, at 10:17, Harald Welte laforge@gnumonks.org wrote:
Hi Andre,
Hey!
just a heads-up about my complete absence from jenkins/CI related discussions: That doesn't mean I'm not interested, it just means that I'm happy to defer the decisions on this to those who understand anything about it (which I don't claim to), i.e. currently most likely Neels and Holger.
I was very happy to see it. One nice aspect of running in a namespace is that all jobs can install to /somedir without interfering with each other. This makes caching intermediate results easy and as shown by Andre a nice speed-up!
I try to look at it over the next couple of days. My uneducated questions right now would be. Why not use the archive artifact feature of jenkins? Why do we need to keep more than one? In general we build all software against master of libraries (e.g. even for the 3G support).
But this feature is something we want, no need to rebuild software that has not changed. :)
holger
On Fri, Mar 31, 2017 at 01:39:44AM +0200, André Boddenberg wrote:
Why not use the archive artifact feature of jenkins?
But a matrixConfigurationJob [3] archives artifacts per axis, so jenkins would store 8 artifacts [4]. Unfortunately the "archive artifacts" post-build-step cannot be triggered by a specific axis configuration (afaik).
[...]
At this point I just got the feeling that writing a custom script and using a directory as an artifact store may will be much more straight forward. I agrre, that this directory approach might be to prototyp'ish.
To me it sounds that an own approach is indeed simpler. If we need to add a plugin, resolve URLs and go crazy, might as well use the file system that's already there and working. Maybe one could benefit from distributed builds, i.e. when one build slave's artifact can be used across other build slaves, but that doesn't apply to our situation. Would anyway make sense to build dependencies once per build slave in a project where each build slave is a different OS or config instead of sharing load.
But according to contrib/jenkins.sh in openbsc repo, some non-master branches (of libosmo-(netif && sccp)) are used in all axes where "IU" is enabled:
- if [ "x$IU" = "x--enable-iu" ]; then
netif_branch="sysmocom/sctp"sccp_branch="sysmocom/iu"- fi
-- 26) osmo-build-dep.sh libosmo-netif $netif_branch 27) osmo-build-dep.sh libosmo-sccp $sccp_branch
To be honest I'm a bit confused or am I just missing something?
The good news is that the dep branches have become unnecessary just two days ago. I will remove them from the build scripts now.
The IU prepares for the vlr_3G branch of openbsc, which needed these branched dependencies. The ./configure --with-iu switch has already been merged to master, so we are building the incomplete IU feature with master as well, to make sure no new development inadvertently destroys the IU parts. (To really build 3G, we would need to also use the openbsc vlr_3G branch instead of master, which is not an option in a gerrit build.).
~N
Would anyway make sense to build dependencies once per build slave in a project where each build slave is a different OS or config instead of sharing load.
Actually the current approach works like that. Each build slave simply has its own directory for holding compiled-dependencies-artifacts. The osmocom docker image might need an additional volume to mount the artifact store, in case /build_bin/artifactStore - which is already mounted - isn't suitable.
One thing probably worth discussing is which job(s) should expose/archive artifacts. In the current approach every job that cannot find the latest dependencies will try to archive/expose them after building from scratch.
Of course we could dedicate this role to a specific job e.g. [1], but I simply didn't (want to) think about such approach. So currently all jobs that profit from dependency artifacts, will archive compiled dependencies in case no artifact holding the latest dependencies was found (and no other project, which compiled the same dependencies finished before).
Practically, the mentioned job [1] will almost always update artifacts, because the chances that a gerrit job finishes before [1] correlates proportionally with the polling intervals of all upstream projects of [1].
In theory a patchset could profit by the artifact archived by its previous patchset, when the duration of both verifications is shorter than the above mentioned polling intervals and no change to any dependency has been introduced meanwhile.
What do you think makes most sense one/several/all job(s) to update artifacts?
The good news is that the dep branches have become unnecessary just two days ago. I will remove them from the build scripts now.
Nice, I will apply these changes for openbsc jobs on my jenkins as well.
blobb
[1] http://jenkins.osmocom.org/jenkins/view/OpenBSC/job/OpenBSC/
On Thu, Mar 30, 2017 at 04:08:33AM +0200, André Boddenberg wrote:
[2] git clone https://github.com/blobbsen/diy-artifacts.git [3] https://jenkins.blobb.me/job/openBSC_multi-configuration_withArtifacts_testR...
Cool, thanks! I see it, but I can't immediately grasp how it works :) Let me take a look at it in a quiet moment and come back with comments later...
~N
-
André Boddenberg -
Harald Welte -
Holger Freyther -
Neels Hofmeyr