Proposal: Usage of stow in jenkins build scripts
Hi, The osmocom software that I do not install via apt but build myself gets installed in $HOME/usr and I do things like: export PATH="${PATH}:${HOME}/usr/bin" export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HOME}/usr/lib" export PKG_CONFIG_PATH=$HOME/usr/lib/pkgconfig MANPATH="${MANPATH}:${HOME}/usr/man" Therefore I don't have to install to a destination outside my home directory which in turn means I can avoid some sudo calls. But that actually does not matter so much, you can mentally substitute $HOME with '/' and append '/local' in the following if you want. I don't install to $HOME/usr directly but use stow for 'managing' the installations, so an installation goes like this: $ mkdir -p $HOME/src/telco/osmo/libosmo-abis $ cd $HOME/src/telco/osmo/libosmo-abis $ git clone git://git.osmocom.org/libosmo-abis $ autoreconf -i $ ./configure --prefix=$HOME/usr/stow/libosmo-abis $ make $ make install $ STOW_DIR=$HOME/usr/stow stow libosmo-abis This has two advantages: * Good visibility of what is installed by a package. $ tree $HOME/usr/stow/libosmocore * Possibility of uninstalling a package even if the Makefile is not present. For me those two advantages outweigh the small additional effort of using stow. Because of that workflow I sometimes run into situations in which make runs fail because either header files are not found during compilation or libraries are not found during linking. The reason is that autotools variables pointing to pkgconfig supplied directories point to a destination under the prefix path with which configure was called. If a dependency was not mentioned in a Makefile.am the required files are not found because the other dependencies are in other directories. For a system on which the osmocom-own dependency libs are installed in distinct directories, current openbsc master HEAD requires the attached patch for a build to succeed. I admit that in most situations the dependencies are installed in the same directory so this does not matter, but it is still a small bug. Therefore I propose to modify the jenkins build scripts so that they use distinct, stow managed directories for the dependencies. That should help to detect oversights, although admittedly developers would have to adapt their workflows to use stow as well in their daily hacking. Otherwise an oversight would be detected by jenkings and not locally which in turn would require additional cleanup commits which is not so nice. What is your gut feeling? Does the increased correctness of Makefiles.am contents justify the (IMO rather small) additional effort of using stow? Kind regards, -Alex
The attached patch below indeed looks like errors in our build. Can you please push this as a patch to gerrit? And you are meaning to say: if we used stow in our jenkins builds, we would catch these errors and fail builds if new ones are introduced, right? What role exactly does stow play here -- do I get the same when I install to separate '--prefix'es and then add all those prefixes to the PKG_CONFIG_PATH and LD_LIBRARY_PATH? The jenkins build scripts for each project are included in the contrib/ dir of each git tree, using scripts found in the osmo-ci.git (also on gerrit). Feel free to go ahead and submit patches that use stow, e.g. for the openbsc.git build to begin with. If it improves our build by catching CFLAGS omissions I'll happily merge it. I can also install packages that you need for this on the build slaves. ~N On Mon, May 15, 2017 at 09:39:57PM +0200, Alexander Huemer wrote:
From e9fc0ed9c7b6c1acc70714f703983708f5d8a1ee Mon Sep 17 00:00:00 2001 From: Alexander Huemer <alexander.huemer@xx.vu> Date: Mon, 15 May 2017 20:42:47 +0200 Subject: [PATCH] Add missing _CFLAGS and _LIBS
These missing pieces go unnoticed if dependencies are not installed in distinct directories.
Change-Id: If8d57b72f63d79cc0d8efba7466c6ec177207cbb --- openbsc/src/utils/Makefile.am | 3 +++ openbsc/tests/sndcp_xid/Makefile.am | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/openbsc/src/utils/Makefile.am b/openbsc/src/utils/Makefile.am index 9c3837a36..26494e13d 100644 --- a/openbsc/src/utils/Makefile.am +++ b/openbsc/src/utils/Makefile.am @@ -109,6 +109,7 @@ osmo_meas_pcap2db_LDADD = \ osmo_meas_pcap2db_CFLAGS = \ $(LIBOSMOCORE_CFLAGS) \ $(LIBOSMOGSM_CFLAGS) \ + $(LIBOSMOABIS_CFLAGS) \ $(NULL)
osmo_meas_udp2db_SOURCES = \ @@ -125,6 +126,7 @@ osmo_meas_udp2db_LDADD = \ osmo_meas_udp2db_CFLAGS = \ $(LIBOSMOCORE_CFLAGS) \ $(LIBOSMOGSM_CFLAGS) \ + $(LIBOSMOABIS_CFLAGS) \ $(NULL)
meas_json_SOURCES = \ @@ -140,5 +142,6 @@ meas_json_LDADD = \ meas_json_CFLAGS = \ $(LIBOSMOCORE_CFLAGS) \ $(LIBOSMOGSM_CFLAGS) \ + $(LIBOSMOABIS_CFLAGS) \ $(NULL)
diff --git a/openbsc/tests/sndcp_xid/Makefile.am b/openbsc/tests/sndcp_xid/Makefile.am index 99b9d1a4f..d09c41b28 100644 --- a/openbsc/tests/sndcp_xid/Makefile.am +++ b/openbsc/tests/sndcp_xid/Makefile.am @@ -15,6 +15,7 @@ sndcp_xid_test_LDADD = \ $(LIBOSMOGB_LIBS) \ $(LIBCARES_LIBS) \ $(LIBCRYPTO_LIBS) \ - -lgtp -lrt -lm + $(LIBGTP_LIBS) \ + -lrt -lm
-- 2.11.0
-- - Neels Hofmeyr <nhofmeyr@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschäftsführer / Managing Directors: Harald Welte
Hi! On Mon, May 15, 2017 at 10:18:03PM +0200, Neels Hofmeyr wrote:
The attached patch below indeed looks like errors in our build. Can you please push this as a patch to gerrit?
Will do.
And you are meaning to say: if we used stow in our jenkins builds, we would catch these errors and fail builds if new ones are introduced, right?
That is the intention, yes, although stow is a convenience layer, not strictly required.
What role exactly does stow play here -- do I get the same when I install to separate '--prefix'es and then add all those prefixes to the PKG_CONFIG_PATH and LD_LIBRARY_PATH?
Yes, the effect would be the same. Stow just makes all of that much more convenient and straight-forward, as you end up with just one location where (symlinks to) libs and so forth have to be searched. stow takes care of that.
The jenkins build scripts for each project are included in the contrib/ dir of each git tree, using scripts found in the osmo-ci.git (also on gerrit). Feel free to go ahead and submit patches that use stow, e.g. for the openbsc.git build to begin with. If it improves our build by catching CFLAGS omissions I'll happily merge it. I can also install packages that you need for this on the build slaves.
I will take a look into that as well.
On Mon, May 15, 2017 at 09:39:57PM +0200, Alexander Huemer wrote:
From e9fc0ed9c7b6c1acc70714f703983708f5d8a1ee Mon Sep 17 00:00:00 2001 From: Alexander Huemer <alexander.huemer@xx.vu> Date: Mon, 15 May 2017 20:42:47 +0200 Subject: [PATCH] Add missing _CFLAGS and _LIBS
These missing pieces go unnoticed if dependencies are not installed in distinct directories.
Change-Id: If8d57b72f63d79cc0d8efba7466c6ec177207cbb --- openbsc/src/utils/Makefile.am | 3 +++ openbsc/tests/sndcp_xid/Makefile.am | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/openbsc/src/utils/Makefile.am b/openbsc/src/utils/Makefile.am index 9c3837a36..26494e13d 100644 --- a/openbsc/src/utils/Makefile.am +++ b/openbsc/src/utils/Makefile.am @@ -109,6 +109,7 @@ osmo_meas_pcap2db_LDADD = \ osmo_meas_pcap2db_CFLAGS = \ $(LIBOSMOCORE_CFLAGS) \ $(LIBOSMOGSM_CFLAGS) \ + $(LIBOSMOABIS_CFLAGS) \ $(NULL)
osmo_meas_udp2db_SOURCES = \ @@ -125,6 +126,7 @@ osmo_meas_udp2db_LDADD = \ osmo_meas_udp2db_CFLAGS = \ $(LIBOSMOCORE_CFLAGS) \ $(LIBOSMOGSM_CFLAGS) \ + $(LIBOSMOABIS_CFLAGS) \ $(NULL)
meas_json_SOURCES = \ @@ -140,5 +142,6 @@ meas_json_LDADD = \ meas_json_CFLAGS = \ $(LIBOSMOCORE_CFLAGS) \ $(LIBOSMOGSM_CFLAGS) \ + $(LIBOSMOABIS_CFLAGS) \ $(NULL)
diff --git a/openbsc/tests/sndcp_xid/Makefile.am b/openbsc/tests/sndcp_xid/Makefile.am index 99b9d1a4f..d09c41b28 100644 --- a/openbsc/tests/sndcp_xid/Makefile.am +++ b/openbsc/tests/sndcp_xid/Makefile.am @@ -15,6 +15,7 @@ sndcp_xid_test_LDADD = \ $(LIBOSMOGB_LIBS) \ $(LIBCARES_LIBS) \ $(LIBCRYPTO_LIBS) \ - -lgtp -lrt -lm + $(LIBGTP_LIBS) \ + -lrt -lm
-- 2.11.0
-- - Neels Hofmeyr <nhofmeyr@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschäftsführer / Managing Directors: Harald Welte
Hi! On Mon, May 15, 2017 at 10:25:18PM +0200, Alexander Huemer wrote:
On Mon, May 15, 2017 at 10:18:03PM +0200, Neels Hofmeyr wrote:
The attached patch below indeed looks like errors in our build. Can you please push this as a patch to gerrit?
Will do.
I just pushed patch set 2 to gerrit. Patch set 1 got Code-Review+2 and Verified+1. Patch set 2 got a broken pipe for --enable-smpp --enable-iu, not sure if that is really caused by the patch.
And you are meaning to say: if we used stow in our jenkins builds, we would catch these errors and fail builds if new ones are introduced, right?
That is the intention, yes, although stow is a convenience layer, not strictly required.
What role exactly does stow play here -- do I get the same when I install to separate '--prefix'es and then add all those prefixes to the PKG_CONFIG_PATH and LD_LIBRARY_PATH?
Yes, the effect would be the same. Stow just makes all of that much more convenient and straight-forward, as you end up with just one location where (symlinks to) libs and so forth have to be searched. stow takes care of that.
The jenkins build scripts for each project are included in the contrib/ dir of each git tree, using scripts found in the osmo-ci.git (also on gerrit). Feel free to go ahead and submit patches that use stow, e.g. for the openbsc.git build to begin with. If it improves our build by catching CFLAGS omissions I'll happily merge it. I can also install packages that you need for this on the build slaves.
I will take a look into that as well.
I created a patch for osmo-ci, but cannot do a $ git push gerrit HEAD:refs/for/master Lack of permissions? For now the patch is attached to this email. On my local machine the following worked then: $ cd ~/src/telco/osmo/openbsc $ MAKE=make PARALLEL_MAKE="-j$(nproc)" \ PATH="$PATH:$HOME/src/telco/osmo/osmo-ci/scripts" ./contrib/jenkins.sh [...] make[1]: Leaving directory '/home/blackbit/src/telco/osmo/openbsc/openbsc/openbsc-0.15.0.770-71124-dirty/_build/sub' if test -d "openbsc-0.15.0.770-71124-dirty"; then find "openbsc-0.15.0.770-71124-dirty" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "openbsc-0.15.0.770-71124-dirty" || { sleep 5 && rm -rf "openbsc-0.15.0.770-71124-dirty"; }; else :; fi ================================================================ openbsc-0.15.0.770-71124-dirty archives ready for distribution: openbsc-0.15.0.770-71124-dirty.tar.gz openbsc-0.15.0.770-71124-dirty.tar.bz2 ================================================================ $ ls deps/install/stow/ libosmo-abis libosmocore libosmo-netif libosmo-sccp libsmpp34 openggsn $ Kind regards, -Alex
Hi! On Fri, May 19, 2017 at 02:09:13PM +0200, Alexander Huemer wrote:
On Mon, May 15, 2017 at 10:25:18PM +0200, Alexander Huemer wrote:
On Mon, May 15, 2017 at 10:18:03PM +0200, Neels Hofmeyr wrote:
The jenkins build scripts for each project are included in the contrib/ dir of each git tree, using scripts found in the osmo-ci.git (also on gerrit). Feel free to go ahead and submit patches that use stow, e.g. for the openbsc.git build to begin with. If it improves our build by catching CFLAGS omissions I'll happily merge it. I can also install packages that you need for this on the build slaves.
I will take a look into that as well.
I created a patch for osmo-ci, but cannot do a $ git push gerrit HEAD:refs/for/master Lack of permissions? For now the patch is attached to this email. On my local machine the following worked then:
My bad, there was an error in my git config. I now pushed the patch to gerrit as 2691[1], the change for openbsc is 2652[2]. Kind regards, -Alex [1] https://gerrit.osmocom.org/2691 [2] https://gerrit.osmocom.org/2652
Hi! On Sat, May 20, 2017 at 12:08:47PM +0200, Alexander Huemer wrote:
I now pushed the patch to gerrit as 2691[1], the change for openbsc is 2652[2].
[1] https://gerrit.osmocom.org/2691 [2] https://gerrit.osmocom.org/2652
Gentle 'ping'. Neels, are you in favor of change 2691? Kind regards, -Alex
participants (2)
-
Alexander Huemer -
Neels Hofmeyr