30 Mar
2017
30 Mar
'17
4:08 a.m.
Hi,
some weeks a go the topic about using dependency-artifacts to speed up
gerrit verifications was mentioned by Neels in the "RFC: jenkins
pipeline" thread [1].
I was curious and wrote a bash script (osmo-artifacts.sh) [2], which
enables build jobs to archive and fetch dependencies. Instead of
Docker it simply uses archives to store them. An artifact's name has
the following form:
( n(${project}.${branch}.${shortRev}_) ).tar.gz
Openbsc builds are 50 % faster when using artifacts. [3]
The jenkins.sh [2] script had to be slightly modified i.e. basically
split into two functions:
1) buildProjectDeps()
2) buildProject()
A third function to generate the artifact name has been added. The
template.sh [2] script illustrates this description.
What do you think about the result/artifactStore/approach in general?
A cleanUpArtifactStore.sh script running as a cron job is already on
my mind, but imo it's a good point to discuss the intermediate result.
blobb
[1] http://lists.osmocom.org/pipermail/openbsc/2017-March/010400.html
[2] git clone https://github.com/blobbsen/diy-artifacts.git
[3]
https://jenkins.blobb.me/job/openBSC_multi-configuration_withArtifacts_test…
30 Mar
30 Mar
10:17 a.m.
Hi Andre,
just a heads-up about my complete absence from jenkins/CI related
discussions: That doesn't mean I'm not interested, it just means that
I'm happy to defer the decisions on this to those who understand
anything about it (which I don't claim to), i.e. currently most likely
Neels and Holger.
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
5:20 p.m.
On 30. Mar 2017, at 10:17, Harald Welte
<laforge(a)gnumonks.org> wrote:
Hi Andre,
Hey!
just a heads-up about my complete absence from
jenkins/CI related
discussions: That doesn't mean I'm not interested, it just means that
I'm happy to defer the decisions on this to those who understand
anything about it (which I don't claim to), i.e. currently most likely
Neels and Holger.
I was very happy to see it. One nice aspect of running in a namespace
is that all jobs can install to /somedir without interfering with each
other. This makes caching intermediate results easy and as shown by
Andre a nice speed-up!
I try to look at it over the next couple of days. My uneducated questions
right now would be. Why not use the archive artifact feature of jenkins?
Why do we need to keep more than one? In general we build all software
against master of libraries (e.g. even for the 3G support).
But this feature is something we want, no need to rebuild software that
has not changed. :)
holger
31 Mar
31 Mar
1:39 a.m.
> Why not use the archive artifact feature of
jenkins?
True, archiving artifacts comes ootb, but the functionality to copy/fetch
artifacts from another buildjob/project as a build step has to be installed
via plugin [1].
Would you consider installing the copyArtifact plugin?
In case installing such plugin is not an option one can simply use:
"wget $URL_OF_LATEST_SUCCESSFUL_ARTIFACT" [2]
to fetch artifacts. Such command can be put in an additional
"execute shell" build step, so it wouldn't touch any build script...
But a matrixConfigurationJob [3] archives artifacts per axis, so jenkins would
store 8 artifacts [4]. Unfortunately the "archive artifacts" post-build-step
cannot be triggered by a specific axis configuration (afaik).
A quick brainstorming how to change this behavior ended in:
- additional "execute shell" step evaluating its axes configuration
and delete artifact accordingly to avoid redundant archiving.
- change "archive artifact" "post build step" to not fail build if no
artifact
could be archived. Unfortunately this configuration applies for all axes,
so when archiving of the "chosen" artifact fails no one will complain.
Furthermore the "latest successful artifact" might already be outdated when
fetching it. This happens every time when a openbsc [5] build is triggered by
an "upstream project" until its artifact has been archived. So a mechanism is
needed to verify whether fetched artifact is still valid and compiling all
dependencies elsewise.
At this point I just got the feeling that writing a custom script and using a
directory as an artifact store may will be much more straight forward. I agrre,
that this directory approach might be to prototyp'ish.
But let's discuss it. We can change everything. :)
> Why do we need to keep more than one? In general
we build all software
> against master of libraries (e.g. even for the 3G support).
Good question! If one artifact is sufficient, let's archive only one.
(sure thing)
But according to contrib/jenkins.sh in openbsc repo, some non-master branches
(of libosmo-(netif && sccp)) are used in all axes where "IU" is
enabled:
20) if [ "x$IU" = "x--enable-iu" ]; then
21) netif_branch="sysmocom/sctp"
22) sccp_branch="sysmocom/iu"
23) fi
--
26) osmo-build-dep.sh libosmo-netif $netif_branch
27) osmo-build-dep.sh libosmo-sccp $sccp_branch
To be honest I'm a bit confused or am I just missing something?
blobb
[1] https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin
[2] wget
https://jenkins.blobb.me/job/matrixJobArtifactHandlingExample/HELLO=Salut,W…
[3] https://jenkins.blobb.me/job/matrixJobArtifactHandlingExample
[4]
https://jenkins.blobb.me/job/matrixJobArtifactHandlingExample/HELLO=Salut,W…
[5] http://jenkins.osmocom.org/jenkins/view/OpenBSC/job/OpenBSC/
2017-03-30 17:20 GMT+02:00 Holger Freyther <holger(a)freyther.de>de>:
On 30. Mar 2017, at 10:17, Harald Welte
<laforge(a)gnumonks.org> wrote:
Hi Andre,
Hey!
just a heads-up about my complete absence from
jenkins/CI related
discussions: That doesn't mean I'm not interested, it just means that
I'm happy to defer the decisions on this to those who understand
anything about it (which I don't claim to), i.e. currently most likely
Neels and Holger.
I was very happy to see it. One nice aspect of running in a namespace
is that all jobs can install to /somedir without interfering with each
other. This makes caching intermediate results easy and as shown by
Andre a nice speed-up!
I try to look at it over the next couple of days. My uneducated questions
right now would be. Why not use the archive artifact feature of jenkins?
Why do we need to keep more than one? In general we build all software
against master of libraries (e.g. even for the 3G support).
But this feature is something we want, no need to rebuild software that
has not changed. :)
holger 
1:46 p.m.
On Fri, Mar 31, 2017 at 01:39:44AM +0200, André Boddenberg wrote:
[...]
> Why not
use the archive artifact feature of jenkins?
But a matrixConfigurationJob [3]
archives artifacts per axis, so jenkins would
store 8 artifacts [4]. Unfortunately the "archive artifacts" post-build-step
cannot be triggered by a specific axis configuration (afaik). At this point I just got the feeling that writing a
custom script and using a
directory as an artifact store may will be much more straight forward. I agrre,
that this directory approach might be to prototyp'ish.
To me it sounds that an own approach is indeed simpler. If we need to add a
plugin, resolve URLs and go crazy, might as well use the file system that's
already there and working. Maybe one could benefit from distributed builds,
i.e. when one build slave's artifact can be used across other build slaves, but
that doesn't apply to our situation. Would anyway make sense to build
dependencies once per build slave in a project where each build slave is a
different OS or config instead of sharing load.
But according to contrib/jenkins.sh in openbsc repo,
some non-master branches
(of libosmo-(netif && sccp)) are used in all axes where "IU" is
enabled:
20) if [ "x$IU" = "x--enable-iu" ]; then
21) netif_branch="sysmocom/sctp"
22) sccp_branch="sysmocom/iu"
23) fi
--
26) osmo-build-dep.sh libosmo-netif $netif_branch
27) osmo-build-dep.sh libosmo-sccp $sccp_branch
To be honest I'm a bit confused or am I just missing something?
The good news is that the dep branches have become unnecessary just two days
ago. I will remove them from the build scripts now.
The IU prepares for the vlr_3G branch of openbsc, which needed these branched
dependencies. The ./configure --with-iu switch has already been merged to
master, so we are building the incomplete IU feature with master as well, to
make sure no new development inadvertently destroys the IU parts. (To really
build 3G, we would need to also use the openbsc vlr_3G branch instead of
master, which is not an option in a gerrit build.).
~N
4:23 p.m.
> Would anyway make sense to build dependencies once
per build slave in a project
> where each build slave is a different OS or config instead of sharing load.
Actually the current approach works like that. Each build slave simply
has its own directory for holding compiled-dependencies-artifacts. The
osmocom docker image might need an additional volume to mount the
artifact store, in case /build_bin/artifactStore - which is already
mounted - isn't suitable.
One thing probably worth discussing is which job(s) should
expose/archive artifacts. In the current approach every job that
cannot find the latest dependencies will try to archive/expose them
after building from scratch.
Of course we could dedicate this role to a specific job e.g. [1], but
I simply didn't (want to) think about such approach. So currently all
jobs that profit from dependency artifacts, will archive compiled
dependencies in case no artifact holding the latest dependencies was
found (and no other project, which compiled the same dependencies
finished before).
Practically, the mentioned job [1] will almost always update
artifacts, because the chances that a gerrit job finishes before [1]
correlates proportionally with the polling intervals of all upstream
projects of [1].
In theory a patchset could profit by the artifact archived by its
previous patchset, when the duration of both verifications is shorter
than the above mentioned polling intervals and no change to any
dependency has been introduced meanwhile.
What do you think makes most sense one/several/all job(s) to update artifacts?
> The good news is that the dep branches have become
unnecessary just two days
> ago. I will remove them from the build scripts now.
Nice, I will apply these changes for openbsc jobs on my jenkins as well.
blobb
[1] http://jenkins.osmocom.org/jenkins/view/OpenBSC/job/OpenBSC/
30 Mar
30 Mar
6:14 p.m.
On Thu, Mar 30, 2017 at 04:08:33AM +0200, André Boddenberg wrote:
Cool, thanks!
I see it, but I can't immediately grasp how it works :)
Let me take a look at it in a quiet moment and come back with comments later...
~N
2606
days inactive
2607
days old
6 comments
4 participants
participants (4)
-
André Boddenberg -
Harald Welte -
Holger Freyther -
Neels Hofmeyr