Hello Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/c/osmo-ci/+/29605
to look at the new patch set (#2).
Change subject: obs/README: add overview with build_binpkg.py
......................................................................
obs/README: add overview with build_binpkg.py
Related: OS#2385
Change-Id: I1b741ad325e6541e6d59dfd50de66bee805e56fe
---
M scripts/obs/README
1 file changed, 19 insertions(+), 0 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/05/29605/2
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29605
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I1b741ad325e6541e6d59dfd50de66bee805e56fe
Gerrit-Change-Number: 29605
Gerrit-PatchSet: 2
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-MessageType: newpatchset
Hello Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/c/osmo-ci/+/29604
to look at the new patch set (#2).
Change subject: jobs/gerrit-binpkgs: new job for deb, rpm packages
......................................................................
jobs/gerrit-binpkgs: new job for deb, rpm packages
Build deb and rpm packages for each patch submitted to gerrit for the
projects in the list.
Example:
https://gerrit.osmocom.org/c/osmo-bsc-nat/+/29492
Related: OS#2385
Change-Id: I7ca8869c2e9f2e7c74a360933be12b5c2b47c2fc
---
A jobs/gerrit-binpkgs.yml
1 file changed, 178 insertions(+), 0 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/04/29604/2
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29604
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I7ca8869c2e9f2e7c74a360933be12b5c2b47c2fc
Gerrit-Change-Number: 29604
Gerrit-PatchSet: 2
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-MessageType: newpatchset
Hello Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/c/osmo-ci/+/29606
to look at the new patch set (#2).
Change subject: obs: config: update note about testing packages
......................................................................
obs: config: update note about testing packages
Related: OS#2385
Change-Id: If6d772052ca64ef93536bf8729a72271f67fbecb
---
M scripts/obs/lib/config.py
1 file changed, 2 insertions(+), 3 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/06/29606/2
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29606
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: If6d772052ca64ef93536bf8729a72271f67fbecb
Gerrit-Change-Number: 29606
Gerrit-PatchSet: 2
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-MessageType: newpatchset
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29605 )
Change subject: obs/README: add overview with build_binpkg.py
......................................................................
obs/README: add overview with build_binpkg.py
Related: OS#2385
Change-Id: I1b741ad325e6541e6d59dfd50de66bee805e56fe
---
M scripts/obs/README
1 file changed, 19 insertions(+), 0 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/05/29605/1
diff --git a/scripts/obs/README b/scripts/obs/README
index cd0955d..f761f1e 100644
--- a/scripts/obs/README
+++ b/scripts/obs/README
@@ -1,3 +1,22 @@
+Overview
+========
+This directory contains the following scripts, related to building Osmocom
+projects in OBS (Open Build Service) and building binary packages. Here is an
+overview of the scripts, run them with -h to get a more detailed description.
+
+* build_srcpkg.py: build one source package for an Osmocom project
+* update_obs_project.py: generate source packages and upload them to OBS
+* build_binpkg.py: build rpm/deb packages for one Osmocom project
+
+Docker
+------
+The above scripts have a -d parameter, that allows running them inside docker.
+This is how we run it in jenkins to avoid installing dependencies on the host.
+
+Note that when building binary packages, libosmocore socket tests and possibly
+others will fail unless docker has ipv6 configured:
+https://osmocom.org/issues/4700#note-3
+
Usage Example: Submitting source packages to Osmocom's OBS build server
=======================================================================
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29605
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I1b741ad325e6541e6d59dfd50de66bee805e56fe
Gerrit-Change-Number: 29605
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-MessageType: newchange
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29606 )
Change subject: obs: config: update note about testing packages
......................................................................
obs: config: update note about testing packages
Related: OS#2385
Change-Id: If6d772052ca64ef93536bf8729a72271f67fbecb
---
M scripts/obs/lib/config.py
1 file changed, 2 insertions(+), 3 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/06/29606/1
diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py
index 393251b..c888220 100644
--- a/scripts/obs/lib/config.py
+++ b/scripts/obs/lib/config.py
@@ -39,9 +39,8 @@
# Osmocom projects: generated source packages will depend on a meta package,
# such as osmocom-nightly, osmocom-latest or osmocom-2022q1. This meta package
# prevents that packages from different feeds are mixed by accident.
-# NOTE: Before adding new projects, make sure the rpm and deb build in OBS!
-# Test it in your own namespace (home:youruser), see README for
-# instructions and/or ask osmith for help.
+# NOTE: Before adding new projects here, add them to jobs/gerrit-binpkgs.yml
+# and ensure the rpm and deb packages build successfully in jenkins.
projects_osmocom = [
"erlang/osmo_dia2gsup",
"libasn1c",
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29606
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: If6d772052ca64ef93536bf8729a72271f67fbecb
Gerrit-Change-Number: 29606
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-MessageType: newchange
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29602 )
Change subject: obs: build_binpkg: debian: don't install manpages
......................................................................
obs: build_binpkg: debian: don't install manpages
Configure dpkg to not extract man pages. Otherwise it will spend some
time regenerating the man page index whenever installing build
dependencies before starting to build a package.
Related: OS#2385
Change-Id: I1c9e3883b976e023c96dfd59eb147770f7ad99a7
---
M scripts/obs/data/build_binpkg.Dockerfile
1 file changed, 5 insertions(+), 0 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/02/29602/1
diff --git a/scripts/obs/data/build_binpkg.Dockerfile b/scripts/obs/data/build_binpkg.Dockerfile
index feacbc9..de53478 100644
--- a/scripts/obs/data/build_binpkg.Dockerfile
+++ b/scripts/obs/data/build_binpkg.Dockerfile
@@ -12,8 +12,13 @@
# build recipe. For rpm-based distributions, there is no build-essential or
# similar package. Instead add relevant packages from prjconf, e.g.:
# https://build.opensuse.org/projects/CentOS:CentOS-8/prjconf
+# For debian, make sure we don't have man pages as otherwise it takes some time
+# to regenerate the manuals database when installing build dependencies.
RUN case "$DISTRO" in \
debian*) \
+ echo "path-exclude=/usr/share/man/*" \
+ > /etc/dpkg/dpkg.cfg.d/exclude-man-pages && \
+ rm -rf /usr/share/man/ && \
apt-get update && \
apt-get install -y --no-install-recommends \
build-essential \
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29602
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I1c9e3883b976e023c96dfd59eb147770f7ad99a7
Gerrit-Change-Number: 29602
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-MessageType: newchange
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29603 )
Change subject: obs: build_binpkg: optimize osmo-gsm-manuals-dev
......................................................................
obs: build_binpkg: optimize osmo-gsm-manuals-dev
Installing osmo-gsm-manuals-dev plus depends takes a long time. Don't do
this for every build, instead do it once when building a second docker
container and then use that.
Related: OS#2385
Change-Id: I8475bd954352b572197795ad4cd9461e39896d48
---
M scripts/obs/build_binpkg.py
A scripts/obs/data/build_binpkg_manuals.Dockerfile
M scripts/obs/lib/docker.py
M scripts/obs/lib/srcpkg.py
4 files changed, 57 insertions(+), 4 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/03/29603/1
diff --git a/scripts/obs/build_binpkg.py b/scripts/obs/build_binpkg.py
index b670af9..27817e2 100755
--- a/scripts/obs/build_binpkg.py
+++ b/scripts/obs/build_binpkg.py
@@ -59,9 +59,17 @@
script_path = "data/build_rpm.sh"
if args.docker:
+ image_type = "build_binpkg"
+
+ # Optimization: use docker container with osmo-gsm-manuals-dev already
+ # installed if it is in build depends
+ if distro.startswith("debian:") \
+ and lib.srcpkg.requires_osmo_gsm_manuals_dev(args.package):
+ image_type += "_manuals"
+
env["BUILDUSER"] = "user"
lib.docker.run_in_docker_and_exit(script_path,
- image_type="build_binpkg",
+ image_type=image_type,
distro=distro,
pass_argv=False, env=env)
else:
diff --git a/scripts/obs/data/build_binpkg_manuals.Dockerfile b/scripts/obs/data/build_binpkg_manuals.Dockerfile
new file mode 100644
index 0000000..fd4c709
--- /dev/null
+++ b/scripts/obs/data/build_binpkg_manuals.Dockerfile
@@ -0,0 +1,20 @@
+# Optimization: installing osmo-gsm-manuals-dev and its many, many dependencies
+# takes quite a long time - sometimes longer than building the package itself
+# (related: OS#4132). Instead of doing this every time before starting a build,
+# here is a second docker container that already has it installed. This gets
+# used by build_binpkg.py in case the package to build depends on
+# osmo-gsm-manuals-dev and the build is done for Debian. Note that right now we
+# don't build the manuals for rpm-based distributions.
+ARG DISTRO_FROM
+FROM ${DISTRO_FROM}
+ARG DISTRO
+
+RUN case "$DISTRO" in \
+ debian*) \
+ apt-get update && \
+ apt-get install -y --no-install-recommends \
+ osmo-gsm-manuals-dev \
+ && \
+ apt-get clean \
+ ;; \
+ esac
diff --git a/scripts/obs/lib/docker.py b/scripts/obs/lib/docker.py
index 81464bb..136aeeb 100644
--- a/scripts/obs/lib/docker.py
+++ b/scripts/obs/lib/docker.py
@@ -14,7 +14,11 @@
return ret
-def get_distro_from(distro):
+def get_distro_from(distro, image_type):
+ # Manuals: depend on regular image (data/build_binpkg_manuals.Dockerfile)
+ if image_type.endswith("_manuals"):
+ return get_image_name(distro, image_type.replace("_manuals", ""))
+
# CentOS 8 is EOL (SYS#5818)
if distro == "centos:8":
return "almalinux:8"
@@ -24,7 +28,7 @@
def build_image(distro, image_type):
image_name = get_image_name(distro, image_type)
- distro_from = get_distro_from(distro)
+ distro_from = get_distro_from(distro, image_type)
print(f"docker: building image {image_name}")
@@ -75,10 +79,15 @@
if add_oscrc:
oscrc = get_oscrc()
- # Build the docker image. Unless it is up-to-date, this will take a few
+ # Unless the docker image is up-to-date, building will take a few
# minutes or so, therefore print the output. No need to restore
# set_cmds_verbose, as we use subprocess.run() below and exit afterwards.
lib.set_cmds_verbose(True)
+
+ # Manuals: build regular image first (data/build_binpkg_manuals.Dockerfile)
+ if image_type.endswith("_manuals"):
+ build_image(distro, image_type.replace("_manuals",""))
+
build_image(distro, image_type)
cmd = ["docker", "run",
diff --git a/scripts/obs/lib/srcpkg.py b/scripts/obs/lib/srcpkg.py
index 1711f70..b37b8b3 100644
--- a/scripts/obs/lib/srcpkg.py
+++ b/scripts/obs/lib/srcpkg.py
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright 2022 sysmocom - s.f.m.c. GmbH <info(a)sysmocom.de>
+import glob
import os
import pathlib
import lib.config
@@ -173,3 +174,18 @@
lib.remove_cache_extra_files()
return version_epoch
+
+
+def requires_osmo_gsm_manuals_dev(project):
+ """ Check if an already built source package has osmo-gsm-manuals-dev in
+ Build-Depends of the .dsc file """
+ path_dsc = glob.glob(f"{lib.get_output_path(project)}/*.dsc")
+ assert len(path_dsc) == 1, f"failed to get dsc path for {project}"
+
+ with open(path_dsc[0], "r") as handle:
+ for line in handle.readlines():
+ if line.startswith("Build-Depends:") \
+ and "osmo-gsm-manuals-dev" in line:
+ return True
+
+ return False
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29603
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I8475bd954352b572197795ad4cd9461e39896d48
Gerrit-Change-Number: 29603
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-MessageType: newchange
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29599 )
Change subject: obs: run_in_docker_and_exit: no os.path.basename
......................................................................
obs: run_in_docker_and_exit: no os.path.basename
Don't use os.path.basename on the script_path parameter passed to the
function. A future patch will pass a script that is inside the data
subdirectory to this function, therefore this is needed.
Related: OS#2385
Change-Id: Ide78d976f9af445c4c8d8748bc274d7289064769
---
M scripts/obs/build_srcpkg.py
M scripts/obs/lib/docker.py
M scripts/obs/update_obs_project.py
3 files changed, 5 insertions(+), 6 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/99/29599/1
diff --git a/scripts/obs/build_srcpkg.py b/scripts/obs/build_srcpkg.py
index 61d8d64..4c848b8 100755
--- a/scripts/obs/build_srcpkg.py
+++ b/scripts/obs/build_srcpkg.py
@@ -31,7 +31,7 @@
lib.set_cmds_verbose(args.verbose)
if args.docker:
- lib.docker.run_in_docker_and_exit(__file__)
+ lib.docker.run_in_docker_and_exit("build_srcpkg.py")
if not args.ignore_req:
lib.check_required_programs()
diff --git a/scripts/obs/lib/docker.py b/scripts/obs/lib/docker.py
index 0a1f748..0c5ec0f 100644
--- a/scripts/obs/lib/docker.py
+++ b/scripts/obs/lib/docker.py
@@ -40,7 +40,7 @@
def run_in_docker_and_exit(script_path, add_oscrc=False,
image_type="build_srcpkg", distro=None):
"""
- :param script_path: what to run inside docker
+ :param script_path: what to run inside docker, relative to scripts/obs/
:param add_oscrc: put user's oscrc in docker (contains obs credentials!)
:param image_type: which Dockerfile to use (data/{image_type}.Dockerfile)
:param distro: which Linux distribution to use, e.g. "debian:11"
@@ -75,9 +75,8 @@
if oscrc:
cmd += ["-v", f"{oscrc}:/home/user/.oscrc"]
- script_path = f"/obs/{os.path.basename(script_path)}"
- cmd += [image_name, script_path] + sys.argv[1:]
+ cmd += [image_name, f"/obs/{script_path}"] + sys.argv[1:]
- print(f"docker: running: {os.path.basename(script_path)} inside docker")
+ print(f"docker: running: {script_path} inside docker")
ret = subprocess.run(cmd)
exit(ret.returncode)
diff --git a/scripts/obs/update_obs_project.py b/scripts/obs/update_obs_project.py
index 58e5888..43748e9 100755
--- a/scripts/obs/update_obs_project.py
+++ b/scripts/obs/update_obs_project.py
@@ -202,7 +202,7 @@
lib.set_cmds_verbose(args.verbose)
if args.docker:
- lib.docker.run_in_docker_and_exit(__file__, True)
+ lib.docker.run_in_docker_and_exit("update_obs_project.py", True)
lib.osc.check_proj(proj)
lib.osc.check_oscrc()
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29599
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: Ide78d976f9af445c4c8d8748bc274d7289064769
Gerrit-Change-Number: 29599
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-MessageType: newchange