Hi!
I'm currently wondering why on earth we'd be running our master builds inside docker containers on the build slaves.
According to https://git.osmocom.org/osmo-ci/tree/jobs/master-builds.yml we're currently building the following projects inside a container: * openbsc.git * osmo-bsc.git * osmo-mgw.git * osmo-msc.git * osmo-sgsn.git
but none of the others.
AFAIR, Holger originally introduced the dockerized builds for speeding up the build tests? I'm not sure how that
In terms of the history that we have on the jenkins job builder files in osmo-ci.git, the dockerized builds in master-builds.yml were copied from the gerrit-verifications.yml. But the git history of course doesn't go back to why the [manual jenkins jobs] used the docker container in the first place.
My line of thought would be to have [at least] all master builds run natively on the build slave. They are all generated by ansible these days, and should contain a well-defined environment.
Any comments?
Regards, Harald
p.s.: The underlying topic is that SSH agent forwarding doesn't exceed into the docker container, and hence it is not possible to use jenkins credentials for uploading resulting build artefacts such as the manuals...
On Thu, Dec 06, 2018 at 11:28:53AM +0100, Harald Welte wrote:
Hi!
I'm currently wondering why on earth we'd be running our master builds inside docker containers on the build slaves.
According to https://git.osmocom.org/osmo-ci/tree/jobs/master-builds.yml we're currently building the following projects inside a container:
- openbsc.git
- osmo-bsc.git
- osmo-mgw.git
- osmo-msc.git
- osmo-sgsn.git
but none of the others.
Those are the projects that run the python-based external tests, which involve opening up VTY and CTRL ports among others, because we start up the program and interact with it on VTY and CTRL. That means each of those projects is not able to run tests more than once per slave, or we get sporadic conflicts of not being able to start the main program. That was really annoying.
Another complex way would be to round robin through a range of ports or localhost addresses or something, and tweak the .cfg for each test run so that the ports never conflict :/
Or some sort of network segmenting?
p.s.: The underlying topic is that SSH agent forwarding doesn't exceed into the docker container, and hence it is not possible to use jenkins credentials for uploading resulting build artefacts such as the manuals...
ah yes. I think the only really good reason to build the manuals in a jenkins build that also runs 'make check' is that we want to catch asciidoc bugs in the patches submitted for the manuals.
But to publish, it could make more sense to have a separate job for the manuals, i.e. build and publish the manuals from a jenkins job that does not use docker to de-collide the ports.
That also doesn't need several build axes, just the one build config that enables the most software components. Why? well, that is the other complication, because we (want to) auto-generate VTY reference data from the main program. That then would also require starting the program and firing up VTY ports and avoiding conflicts. But I'm also having an idea here, that we introduce a cmdline option that directly dumps the VTY reference to stdout instead of having to connect via VTY and without opening any ports.
Another idea would be to somehow run contrib/jenkins.sh in the docker, but in the jenkins job extract the generated manuals from that docker to publish them.
~N
Hi Neels,
2018. dec. 6. dátummal, 13:38 időpontban Neels Hofmeyr nhofmeyr@sysmocom.de írta:
Those are the projects that run the python-based external tests, which involve opening up VTY and CTRL ports among others, because we start up the program and interact with it on VTY and CTRL. That means each of those projects is not able to run tests more than once per slave, or we get sporadic conflicts of not being able to start the main program. That was really annoying.
Another complex way would be to round robin through a range of ports or localhost addresses or something, and tweak the .cfg for each test run so that the ports never conflict :/
Or some sort of network segmenting?
I think Linux network namespaces would be a possible solution for this issue, if I understood correctly, but setting that up feels to me more complicated then using Docker. Just my 2cents.
Cheers, Domi
On 6. Dec 2018, at 12:12, Tomcsányi, Domonkos domi@tomcsanyi.net wrote:
Hi Neels,
- dec. 6. dátummal, 13:38 időpontban Neels Hofmeyr nhofmeyr@sysmocom.de írta:
Those are the projects that run the python-based external tests, which involve opening up VTY and CTRL ports among others, because we start up the program and interact with it on VTY and CTRL. That means each of those projects is not able to run tests more than once per slave, or we get sporadic conflicts of not being able to start the main program. That was really annoying.
Another complex way would be to round robin through a range of ports or localhost addresses or something, and tweak the .cfg for each test run so that the ports never conflict :/
Or some sort of network segmenting?
I think Linux network namespaces would be a possible solution for this issue, if I understood correctly, but setting that up feels to me more complicated then using Docker. Just my 2cents.
I just skimmed through this thread but it seems to have the right answers.
The primary reason was being able to run multiple instances of the VTY tests without having to make the (test)software more complex (e.g. bind to vty port 0 and then parse it) (and the db paths and af_unix paths)
Secondary reasons are making the build environment volatile and managing the dependencies differently.
For the primary one I looked at "ip netns exec" first but IIRC couldn't find tools to automatically clean things up. I briefly considered writing something that does clone+unshare(2) but the toolchain of docker was further and provided immediate results.
holger
Hi Holger and List,
On Thu, Dec 06, 2018 at 04:26:47PM +0000, Holger Freyther wrote:
I just skimmed through this thread but it seems to have the right answers.
The primary reason was being able to run multiple instances of the VTY tests without having to make the (test)software more complex (e.g. bind to vty port 0 and then parse it) (and the db paths and af_unix paths)
ok. That's very valid for gerrit builds, where it's important to be able to build test multiple patches in parallel. For master builds, I would argue that this is not a real concern, as it's fine to have only one master build per repository (e.g. osmo-bsc.git) at any given time [on any given build slave].
Secondary reasons are making the build environment volatile
That's of course still a valid reason, for any build, including master builds - so we'll have to keep the dockerized builds. However, I would argue that we should
1) build all projects, or at least all projects that have dependencies on other [osmocom] projecst inside docker, not some on the build slave and some in a container
2) have a process by which the container images are re-built. Right now those appear to be still jessie-based, for example. It's good to test both jessie and stretch, of course, but doing some project builds on either of the two doesn't really ensure compatibility for both versions. We should either build everything on both stretch and jessie, or one defined version.
3) ensure that gerrit and master builds run in identical environments. This reduces the risk of patches passing gerrit but failing in master.
and managing the dependencies differently.
Can you please elaborate on that?
Regarding the "deployment" of artifacts like pdf manuals or binary firmware builds, I would say it's best to move that into a separate post-build action inside jenkins, one that happens outside e.g. the docker image. This would mean we mount some "output" directory from the build slave into the container where the artifacts are stored to persist beyond the "--rm" docker container.
From there, we should be able to scp/rsync/... the artefact to the [ftp] server,
using e.g. SSH agent forrwarding with credetials provided by the server.
Hi,
- build all projects, or at least all projects that have dependencies on other [osmocom] projecst inside docker, not some on the build slave and some in a container
+1
- have a process by which the container images are re-built. Right now those appear to be still jessie-based, for example. It's good to test both jessie and stretch, of course, but doing some project builds on either of the two doesn't really ensure compatibility for both versions. We should either build everything on both stretch and jessie, or one defined version.
I thought there is a defined process already. All Jenkins jobs generated by gerrit-verifications.yml and master-builds.yml use the "osmocom:amd64" image, which is defined here:
osmo-ci.git/docker/Dockerfile_osmocom_jenkins.amd64
It gets rebuilt automatically for all buildbots, whenever changes to osmo-ci.git are made: https://jenkins.osmocom.org/jenkins/job/update-osmo-ci-on-slaves/
With a few more details, I had documented the process as I understood it here: https://osmocom.org/projects/osmocom-servers/wiki/Jenkins_build_verification...
- ensure that gerrit and master builds run in identical environments. This reduces the risk of patches passing gerrit but failing in master.
Assuming we use Docker for all builds, then this should be solved (as we are using the same image already).
Regarding the "deployment" of artifacts like pdf manuals or binary firmware builds, I would say it's best to move that into a separate post-build action inside jenkins, one that happens outside e.g. the docker image. This would mean we mount some "output" directory from the build slave into the container where the artifacts are stored to persist beyond the "--rm" docker container.
From there, we should be able to scp/rsync/... the artefact to the [ftp] server, using e.g. SSH agent forrwarding with credetials provided by the server.
Sounds good. (Implementation details: the entire build folder is actually mounted inside Docker, so it is writing all build results to the host system. But we clean it up with "osmo-clean-workspace.sh" from osmo-ci.git/scripts before and after each build, hence the built pdfs do not remain. But I'm sure we can find a way to keep the artifacts, no need to discuss that here in detail.)
Regards, Oliver
On 12/7/18 11:58 AM, Oliver Smith wrote:
- build all projects, or at least all projects that have dependencies on other [osmocom] projecst inside docker, not some on the build slave and some in a container
+1
One edge case: libosmocore gets compiled on FreeBSD_amd64. How would we handle that if we want to build everything with Docker?
On Fri, Dec 07, 2018 at 04:59:36PM +0100, Oliver Smith wrote:
One edge case: libosmocore gets compiled on FreeBSD_amd64. How would we handle that if we want to build everything with Docker?
I wouldn't have much hope for any Osmocom code to keep working on non-Linux platforms in the long term. It is simply not a priority for the project. There's no attention being paid to portability concerns at the level required for portability to non-Linux platforms.
As an OpenBSD user/dev, I believe this problem is much too pervasive in the free software community as a whole. But in Osmocom's case I really don't mind the lack of BSD support. The effort required for portability is non-trivial and I don't see any advantages in using FreeBSD instead of Linux as a base for Osmocom deployments. OpenBSD is out anyway since it doesn't have an SCTP stack.
AFAIK none of the BTS hardware supported by Osmocom is supported by any BSD. And most of the active Osmocom developers don't have direct experience with BSD anyway. Their time is better spent making sure that Osmocom runs very very well on Linux.
Hi Stephan,
On Fri, Dec 07, 2018 at 05:18:22PM +0100, Stefan Sperling wrote:
I wouldn't have much hope for any Osmocom code to keep working on non-Linux platforms in the long term. It is simply not a priority for the project. There's no attention being paid to portability concerns at the level required for portability to non-Linux platforms.
Indeed, this is true. We once had build testing for more projects on FreeBSD, but the amount of time fixing those issues was considerable, while at the time we did a quick poll and not a single FreeBSD user identified themselves at this mailing list. So the decision was clear, and we concluded, as you write:
Their time is better spent making sure that Osmocom runs very very well on Linux.
However, let me re-state that we are happy to merge any compatibility/portability related patches, should there be any developers/users on other platforms that contribute such changes. I just want to make clear that we're not ignorant of or somehow opposing other platforms. It's just that we focus our existing resources where we feel they are spent most effectively.
Regards, Harald
-
Harald Welte -
Holger Freyther -
Neels Hofmeyr -
Oliver Smith -
Stefan Sperling -
Tomcsányi, Domonkos