Hi Andre, On Wed, Sep 06, 2017 at 03:05:27PM +0200, Harald Welte wrote: I've prototyped this at http://git.osmocom.org/docker-playground/tree/osmo-gerrit-libosmo/Dockerfile If you're trying this, make sure you're running it using 'make run' or manually including the "--tmpfs /tmpfs:exec" arguments. The VTY tests of course take ages, but the actual build of openbsc.git is pretty fast and executed on /tmpfs. if the jenkins job for openbsc-gerrit would then do * docker build ... * docker run ... Then the latest openbsc.git would be built and automatically all upstream dependencies are updated from latest master - but only if the latest master of those dependencies has changed (unlike current osmo-deps.sh). This is of course all just a prototype, and proper scripts/integration is needed. -- - Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)

Hi Andre, On Wed, Sep 06, 2017 at 08:45:02PM +0200, André Boddenberg wrote: This is good news. Happy you like it. Great. After that much praise, there's also a downside: * we currently install libosmo* to /usr/local using 'sudo'. There's actually no real reason for this, one could install into some other user-writable PREFIX and use that at compile ('docker run') time. Would be great to see some patches cleaning this up * we always have all dependencies installed, i.e. we're no longer trying to do builds with certain libraries not present. Let's say we wnat to do an osmo-msc --disable-smpp build: libsmpp34 will still be present, and we *could* introduce unnoticed bugs that would only show up once libsmpp34 is actually not present. One could either not worry too much about it, or one could do some more PKG_CONFIG_PATH hackery with different PREFIX for the different libraries so that each library is in a different PREFIX and the librray is not found unless we explicitly pass the related paths to ./configure. I'm not sure if it's worth investing too mcuh time into this, given that lots of conditionals go away in the split-nitb scenario: osmo-bsc always implicitly requires "--enable-osmo-bsc" and osmo-sgsn always implicitly requires libgtp. However, there's still the SMPP example. What do others say? Is this important to test? If so, do we have volunteers to look into writing scripts for this? * In terms of artefacts, we should figure out which ones we want to keep. For sure any kind of log files like config.log should be copied from the tmpfs to the workspace before we kill the container. They might contain useful information. One *might* also want to do a "make install" to the workspace? So to me config.log is a must, everything else is "optional, later, if somebody needs it". But then, Pau had some other opionion, AFAIR. That would be much appreciated. I think the biggest missing part is figuring out some helper scripts to easily 'run' that Docker container with the related arguments. I guess a given jenkins job then should only call that helper script and pass the "configure" arguments and some environment variables like our PARALLEL_MAKE? I wouldn't want to clutter the jenkins job definitions with repetetive long hand-crafted 'docker run' commands. The next question is then where to store all of this. Given that this helper script as well as the Dockerfile is quite generic, it should probably go into osmo-ci. But then, the Dockerfile depens on stuff from docker-playground. We could merge the two or keep the Dockerfile in docker-playground and just put the helper script in osmo-ci? Actually, the part of the script that's running inside the container could be included in the image at 'docker build' time. Only the 'docker run' wrapper that's used to start a container is external. In any case, from my point of view a given jenkins gerrit job should do: * git fetch && git checkout -f -B master origin/master on * osmo-ci/docker-playground to make sure we catch any updates to those * 'docker build' of the respective docker image * 'docker run' by means of some helper script, using the respective arguments / build matrix options as required by the given job/project, as well as the exact git commit we want to test-build (instead of master) The current image should work for {openbsc,osmo-{bsc,bts,pcu,mgw,sgsn,ggsn,trx,hlr},openggsn}-gerrit jobs For the library projects {libasn1c,libsmpp34,libosmo{core,-abis,-netif,-sccp}} and others which have some downstream build dependencies, we could also use the same docker base image. However, some additional concerns: * when building e.g. libosmocore on a container that already has a system-wide installation of libosmocore, we could accidentially use include files from the system (/usr/local/include), rather than those of the current branch/commit that we're trying to build. One more reason not to install into /usr/local but into specific prefixes (see above) and then tell each given build which of the prefixes to use or not, depending on its build dependencies * we might want to test to build (some of?) the downstream dependencies, as e.g. a commit in libosmocore might break osmo-bts. Any help / work in the above areas (and anything I might have missed) is much appreciated. I won't have any more time to work on this, too many other topics going on :/ Regards, Harald -- - Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)

Hi Neels, Of course this will work, but afaics it won't suite for gerrit verifications. Because docker always takes the latest available base image. So if libosmocore image is currently rebuild and didn't finish yet, a libosmo-abis docker build, which uses libosmocore as base image wouldn't wait until the "new" libosmocore image is built, it would simply use the "old" one. That's why I really like the "one container" solution for gerrit verifications, which checks whether all dependencies are up to date by a "docker build" invocation. My knowledge about the gsm-tester is quite limited to its manual and I agree with the general idea of reusing things, but it might be better to have Docker images for precise purposes? My spike with Harald's gerrit-verification image [1] and jenkins-job-builder [2] will probably finish next week (mo/tu). BR, André [1] https://git.osmocom.org/docker-playground/tree/osmo-gerrit-libosmo/Dockerfi… [2] https://pypi.python.org/pypi/jenkins-job-builder/

On Thu, Sep 07, 2017 at 03:34:48PM +0200, André Boddenberg wrote: On the tester, all that we want to do is build (usually current master) and keep the binaries in the image, so that we can launch them with specific config files on another computer. That's all we need to know in this context. If a libosmo-abis patch starts building just before the latest merge to libosmo-core master has finished docker building, it doesn't matter much. The libosmo-abis patch usually does not depend on libosmocore work that is just being merged. If it does, the libosmo-abis patch submitter will have to wait for libosmocore to complete. This is the same as our current gerrit patch submission works and "a law of nature". It's expected and not harmful. On Thu, Sep 07, 2017 at 04:09:26PM +0200, Harald Welte wrote: * I expect the libosmocore-master jenkins job to docker build the libosmocore image whenever a patch was merged. * The libosmo-abis image simply builds on the last stable libosmocore docker image it finds in the hub (what was the generic name for the hub again?). * In turn osmo-hlr takes the last stable libosmo-abis image and just adds building osmo-hlr on top. Each jenkins job takes exactly one 'FROM' image, builds exactly one git tree, stores exactly one state in the docker cache. To be precise, the 'master' build jobs would store the built images in the docker hub thing, the gerrit build jobs just take the build rc and discard the image changes (could keep the result in the cache for a short time). IIUC, we win tremendously by only 'docker build'ing when something is merged to master. One goal for the osmo-gem-tester is to anyway have docker images ready for each project's current master. I don't see why that is required? I accept that. But I don't have a clear picture in my mind of how it would look in practice with a joint Dockerfile: So we have one Dockerfile like https://git.osmocom.org/docker-playground/tree/osmo-gerrit-libosmo/Dockerfi… which contains all osmo gits. This file also actually updates from git and builds *all* the various libs when we update the image. How does this translate to us wanting to e.g. have one jenkins job verifying libosmocore, one for libosmo-abis, [...], one for osmo-msc? Each start out with an image where the very project we want to check is already built and installed, so we need to actively remove installed files from /usr/local/{include,lib,bin} first, for only that project under scrutiny. We can't really rely on 'make uninstall' being correct all the time. How about using 'stow' that showed up recently? It allows wiping installs separately, right? I still see chicken-egg problems: when I run a libosmocore jenkins job, I want to update the image first. *) That inherently already may build libosmocore. For a gerrit patch, it possibly builds libosmocore master, and I can later introduce the patch. If I want to test master though, updating the image already *will* build master, which I actually wanted to test; at what stage then do I detect a failure? If a 'docker build' failure already counts as failure, then: *) What if e.g. libosmo-abis fails during image update: does it cause a failure to be counted for the libosmocore build job instead? I.e. does an unrelated broken libosmo-abis master cross-fire onto non-depending builds? How do we solve this? And I see a possibility: say for every libosmocore patch we actually also build the whole chain and verify that this new libosmocore works with all depending projects. That way we would detect whether libosmocore breaks builds down the line in other projects, which we don't do on gerrit level yet. OTOH then we can't easily introduce a change that needs patches in more than one repos; so far we e.g. first change libosmo-sccp (hence temporarily break the build on osmo-msc), then follow right up with a patch on osmo-msc. When we always verify all depending projects as well, we'll never get the first libosmo-sccp patch past the osmo-msc check. To solve we'd need to get both patches in at the same time. We could parse the commit message 'Depends:' marker, which would actually be interesting to explore: We could have only one single process that is identical for *all* gerrit +V builds across all projects, and wherever a patch is submitted, it always rebuilds and verifies the whole ecosystem from that project on downwards to the projects that depend on it. By docker build arguments we can build with specific git hashes, e.g. one with a new patch, the others with master. A "Depends:" commit msg marker could take in N such git hashes at the same time. Non-trivial to implement, takes more build time (longest for libosmocore, shorter the farther we go towards leaf projects), but concept wise quite appealing. For the osmo-gsm-tester, it's actually ok to have only one image with all binaries ready in it (and launch it N times if we have to separate networks). Having separate images per each program is helpful to be able to quickly rebuild only one binary to a different git hash for debugging, but there are easy ways to do so also with one joint image. So... my dream setup is one joint image and one build job for all projects, rebuilding all dependent projects always, using stow to safely clean up for re-building, and automatically building 'Depends:' marked patches together. But it seems to me to be less trouble to manage N separate Dockerfiles that reference each other and are inherently clean every time. Is there something I'm missing? ~N