3 Mar
2017
3 Mar
'17
7:27 p.m.
Hi.
Right now we have rather sophisticated CI/CD setup which is spread over
several services (OBS for nightly packages, jenkins for CI) and
repositories.
It works well, but if I've got to somehow extend/alter it than I have to
look in at least 3 different places:
- ocmo-ci repo for common scripts
- jenkins job in jenkins web ui
- project's jenkins*.sh scripts
I think it's not meant to be that way, it's just what has grown
organically over the years.
Right now we use recent enough jenkins which support Pipelines [1]
plugin. The basic idea behind it is that each project has Jenkinsfile
[2] in the repository which is self-contained configuration for CI.
In theory enabling this plugin should not affect existing jobs as it's
entirely separate job type.
I suggest following:
- enable Pipelines plugin in jenkins
- configure new pipelines-based job pointing to non-master branch which
we can use as a playground
- once we comfortable with it - migrate existing CI (and possibly CD) to
pipelines (as time permit)
- switch gerrit from old jobs to new pipelines
What do you think?
Details are available via links below but here are some quick takeaways
which motivated me to write this email:
- we'll have actual CI job description under version control (right now
it just sits in web ui)
- we'll have single place to look for CI-related things (ok, maybe 2
places if we still need common library)
- it's more eye-candy (it's clear which build step we're in and how long
each step took)
- it's easier to parallelize (e. g. run sanitizer and regular builds in
parallel)
On a related note: in general, who's point of contact for things like
"please update jenkins plugin X to version Y"?
[1] https://jenkins.io/doc/book/pipeline/
[2] https://jenkins.io/doc/book/pipeline/jenkinsfile/
--
Max Suraev <msuraev(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
4 Mar
4 Mar
6:28 a.m.
On Fri, Mar 03, 2017 at 07:27:16PM +0100, Max wrote:
Hi.
Right now we have rather sophisticated CI/CD setup which is spread over
several services (OBS for nightly packages, jenkins for CI) and
repositories.
It works well, but if I've got to somehow extend/alter it than I have to
look in at least 3 different places:
- ocmo-ci repo for common scripts
- jenkins job in jenkins web ui
- project's jenkins*.sh scripts
I think it's not meant to be that way, it's just what has grown organically
over the years.
Right now we use recent enough jenkins which support Pipelines [1] plugin.
Well, whaddaya know, one of the accelerate3g5 contestants actually wanted to
help introduce Jenkins Pipelines. Sounds like it's going to happen :)
dr.blobb, could you join the discussion please?
About the distribution of scripts: the osmo-ci has common scripts, and each
project's jenkins.sh has project-specific steps in it. Either we keep all
project-specific details in one central place instead of with the project
(where one might say it belongs), or we copy the osmo-ci scripts to every
project duplicating the code -- I don't really see a way to get out of that
part... But I've been annoyed by having to edit a dozen different repositories
to tweak the same detail in each jenkins.sh (like adding that value_string
check), so keeping all jenkins.sh in one central place has occurred to me
several times before as being a good idea. It has the disadvantage that you
can't change jenkins.sh at the same time as a patch changes the behavior (like
your --enable-sanitizer change in libosmo-abis, which was all nicely applied in
one patch), but so far the disadvantage of having to edit N separate
repositories has far outweighed that for me.
I suggest following:
- enable Pipelines plugin in jenkins
- configure new pipelines-based job pointing to non-master branch which we
can use as a playground
- once we comfortable with it - migrate existing CI (and possibly CD) to
pipelines (as time permit)
- switch gerrit from old jobs to new pipelines
What do you think?
sounds excellent.
- we'll have actual CI job description under
version control (right now it
just sits in web ui)
except for the parts we relayed into the jenkins.sh scripts for the same
reason -- but this includes the job config as well, right? which sounds
excellent. I'd trade any web interface for text files any time.
- we'll have single place to look for CI-related
things (ok, maybe 2 places
if we still need common library)
ah ok
- it's more eye-candy (it's clear which build
step we're in and how long
each step took)
- it's easier to parallelize (e. g. run sanitizer and regular builds in
parallel)
how does this come about? Separate workspace per pipeline run? The point
being: does jenkins keep separate network interfaces for each? Our 'make check'
often starts up things on ports that must not be reused. That's why docker was
such an improvement for the openbsc job. Ah, I see that pipelines can actually
integrate with docker as "agent"!
On a related note: in general, who's point of
contact for things like
"please update jenkins plugin X to version Y"?
I guess Holger or me, but the decision whether to do that is probably more with
Holger and Harald, besides the community at large.
One thing that catches my attention: do we have to trade shell script for
groovy scripting? That would be a potential drawback, because we're familiar
with shell, not with groovy. Groovy is very Java, we're more C and sh.
It seems that shell commands become
sh 'ls -alh'
so I guess we would put everything except one-liners into an actual .sh file
and call that from the jenkinsfile. I wouldn't want to shell-script with
prepending 'sh' everywhere and losing all of the env on every line.
Overall I would +1 pipeline trials.
The jenkins.osmocom.org plugin page though says:
"
Pipeline
A suite of plugins that lets you orchestrate automation, simple or complex. See the
Jenkins website for more details and documentation.
<red>Warning: This plugin requires dependent plugins be upgraded and at least one of
these dependent plugins claims to use a different settings format than the installed
version. Jobs using that plugin may need to be reconfigured, and/or you may not be able to
cleanly revert to the prior version without manually restoring old settings. Consult the
plugin release notes for details.</red>
"
whatever that means in practice.
~N
8:57 a.m.
On 4 Mar 2017, at 06:28, Neels Hofmeyr
<nhofmeyr(a)sysmocom.de> wrote:
About the distribution of scripts: the osmo-ci has
common scripts, and each
project's jenkins.sh has project-specific steps in it. Either we keep all
project-specific details in one central place instead of with the project
(where one might say it belongs), or we copy the osmo-ci scripts to every
project duplicating the code -- I don't really see a way to get out of that
part... But I've been annoyed by having to edit a dozen different repositories
to tweak the same detail in each jenkins.sh (like adding that value_string
check), so keeping all jenkins.sh in one central place has occurred to me
several times before as being a good idea. It has the disadvantage that you
can't change jenkins.sh at the same time as a patch changes the behavior (like
your --enable-sanitizer change in libosmo-abis, which was all nicely applied in
one patch), but so far the disadvantage of having to edit N separate
repositories has far outweighed that for me.
The reason I moved the build instruction out of the Job into a file is to
have people easily rebuild/reproduce it. E.g. it helps with people that
either don't know make distcheck or can't copy the invocation from the log.
I think if someone wants to reproduce the failure, it will be difficult for
that person to checkout the right repository with the build script. :)
What I think should be avoided is to use Jenkins specific files. You might
need to install Java and tons of jars to locally drive your build. ;)
holger
10:38 a.m.
Hi Holger,
On Sat, Mar 04, 2017 at 08:57:08AM +0100, Holger Freyther wrote:
The reason I moved the build instruction out of the
Job into a file is to
have people easily rebuild/reproduce it. E.g. it helps with people that
either don't know make distcheck or can't copy the invocation from the log.
I think tat makes a lot of sense, and I would generally like to see more
(if possible) of the jenkins jobs move into the repositories, maybe even
those of the sysmocom jenkins.
It should be possible (and documented!) how somebody can locally
reproduce a build (particularly a build error) that is shown in a
jenkins job.
I think if someone wants to reproduce the failure, it
will be difficult for
that person to checkout the right repository with the build script. :)
agreed.
What I think should be avoided is to use Jenkins
specific files. You might
need to install Java and tons of jars to locally drive your build. ;)
also fully agreed here.
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
4:24 p.m.
On Sat, Mar 04, 2017 at 08:57:08AM +0100, Holger Freyther wrote:
At the moment the jenkins.sh actually depends on osmo-ci scripts, so said
someone would also need the right repository as well -- short of understanding
what 'osmo-build-dep.sh libosmocore' means :P
On Sat, Mar 04, 2017 at 10:43:27AM +0100, Harald Welte wrote:
About the
distribution of scripts: the osmo-ci has common scripts, and each
project's jenkins.sh has project-specific steps in it. Either we keep all
The reason I moved the build instruction out of the Job into a file is to
have people easily rebuild/reproduce it. E.g. it helps with people that
either don't know make distcheck or can't copy the invocation from the log.
I think if someone wants to reproduce the failure, it will be difficult for
that person to checkout the right repository with the build script. :) what about git-subtree or git-submodule for osmo-ci?
Probably a good idea. Though, submodules require manual steps to check out,
right? So a README is needed anyway.
With osmo-ci submoduled it's also a small step to actually move the various
jenkins.sh into osmo-ci and be able to edit them centrally? :) anyway, not
high up on the agenda.
On Sat, Mar 04, 2017 at 08:57:08AM +0100, Holger Freyther wrote:
What I think should be avoided is to use Jenkins
specific files. You might
need to install Java and tons of jars to locally drive your build. ;)
[and]
On Sat, Mar 04, 2017 at 10:43:27AM +0100, Harald Welte wrote:
I have a strong opinion against our developers having
to learn a new
programming language just for continuous integration. That would be a
*very* high price to pay.
Have all build steps in shell scripts that can run on their own, and accompany
that with a jenkins specific file to drive the pipeline, which basically calls
the shell scripts?
On Sat, Mar 04, 2017 at 10:43:27AM +0100, Harald Welte wrote:
Also, in general I appreciate steps that improve
productivity. But
please keep in mind that using new tools/toys just because they exist
and may be hyped is not a good reason. I'm not saying this is the case
here, but I'm saying we have to be careful.
So, what would be the actual benefits?
* complete job config in git? (to be confirmed)
* faster because less things have to be rebuilt? (to be confirmed)
* faster because easily parallelizable/easier integration with docker? (to be
confirmed)
If dr.blobb and/or Max could clarify these points that would be great
(while I guess they will clarify in depth only after actual trials).
First tests could run on a privately setup Jenkins ... dr.blobb?
~N
10:43 a.m.
Hi Neels,
On Sat, Mar 04, 2017 at 06:28:24AM +0100, Neels Hofmeyr wrote:
About the distribution of scripts: the osmo-ci has
common scripts, and each
project's jenkins.sh has project-specific steps in it.
Either we keep all project-specific details in one
central place
instead of with the project (where one might say it belongs), or we
copy the osmo-ci scripts to every project duplicating the code -- I
don't really see a way to get out of that part...
what about git-subtree or git-submodule for osmo-ci?
except for the parts we relayed into the jenkins.sh
scripts for the same
reason -- but this includes the job config as well, right? which sounds
excellent. I'd trade any web interface for text files any time.
I also think that having the job configuration in the repository would
be a plus.
One thing that catches my attention: do we have to trade shell script for
groovy scripting? That would be a potential drawback, because we're familiar
with shell, not with groovy. Groovy is very Java, we're more C and sh.
I have a strong opinion against our developers having to learn a new
programming language just for continuous integration. That would be a
*very* high price to pay.
Also, in general I appreciate steps that improve productivity. But
please keep in mind that using new tools/toys just because they exist
and may be hyped is not a good reason. I'm not saying this is the case
here, but I'm saying we have to be careful.
Regards,
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
3035
days inactive
3036
days old
5 comments
4 participants
participants (4)
-
Harald Welte -
Holger Freyther -
Max -
Neels Hofmeyr