October 2022 - gerrit-log - lists.osmocom.org

Change in osmo-ci[master]: obs: add build_binpkg.py

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29601 ) Change subject: obs: add build_binpkg.py ...................................................................... obs: add build_binpkg.py Add a script to build deb/rpm packages, as it would be done on obs.osmocom.org. This will be used by jenkins to verify deb and rpm builds for each submitted gerrit patch. I have attempted to use 'osc build' instead of directly calling apt-get build-dep and dpkg-buildpackage (and rpm equivalents). Using 'osc build' would have the advantage that the build works as close to the OBS build as possible. However it would try to install dependencies with sudo, so we would need to have sudo available in the docker container that builds the untrusted code from gerrit. Let's not do that. Related: OS#2385 Change-Id: I4c6b5d61af35df98cbc70d9ddc8ad36d38a9ce18 --- A scripts/obs/build_binpkg.py A scripts/obs/data/build_binpkg.Dockerfile A scripts/obs/data/build_deb.sh A scripts/obs/data/build_rpm.sh M scripts/obs/lib/__init__.py A scripts/obs/lib/binpkg_deb.py A scripts/obs/lib/binpkg_rpm.py M scripts/obs/lib/config.py M scripts/obs/lib/docker.py 9 files changed, 275 insertions(+), 3 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/01/29601/1 diff --git a/scripts/obs/build_binpkg.py b/scripts/obs/build_binpkg.py new file mode 100755 index 0000000..b670af9 --- /dev/null +++ b/scripts/obs/build_binpkg.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright 2022 sysmocom - s.f.m.c. GmbH <info(a)sysmocom.de> +import argparse +import multiprocessing +import os +import lib +import lib.binpkg_deb +import lib.config +import lib.docker +import lib.git +import lib.metapkg +import lib.srcpkg + + +def main(): + distro_default = lib.config.docker_distro_default + distro_choices = [distro_default] + lib.config.docker_distro_other + jobs_default = multiprocessing.cpu_count() + 1 + + parser = argparse.ArgumentParser( + description="Build a deb or rpm package as it would be done on" + " obs.osmocom.org. Use after building a source package" + " with build_srcpkg.py." + f" Output dir: {lib.config.path_temp}/binpkgs") + parser.add_argument("-d", "--docker", choices=distro_choices, + const=distro_default, nargs="?", + help="build the package in docker for a specific" + f" distro (default: {distro_default})") + parser.add_argument("-j", "--jobs", type=int, default=jobs_default, + help=f"parallel running jobs (default: {jobs_default})") + parser.add_argument("-v", "--verbose", action="store_true", + help="always print shell commands and their output," + " instead of only printing them on error") + parser.add_argument("package", + help="package name, e.g. libosmocore") + args = parser.parse_args() + + lib.set_cmds_verbose(args.verbose) + + srcdir = f"{lib.config.path_temp}/srcpkgs/{args.package}" + if not os.path.exists(srcdir): + print(f"ERROR: {args.package}: no srcpkg found, run build_srcpkg.py" + " first!") + exit(1) + + bindir = f"{lib.config.path_temp}/binpkgs" + lib.run_cmd(["rm", "-rf", bindir]) + os.makedirs(bindir) + + distro = args.docker if args.docker else distro_default + + env = {"JOBS": str(args.jobs), + "PACKAGE": args.package, + "BUILDUSER": os.environ["USER"]} + + script_path = "data/build_deb.sh" + if not distro.startswith("debian:"): + script_path = "data/build_rpm.sh" + + if args.docker: + env["BUILDUSER"] = "user" + lib.docker.run_in_docker_and_exit(script_path, + image_type="build_binpkg", + distro=distro, + pass_argv=False, env=env) + else: + lib.run_cmd(["sudo", "-E", script_path], env=env, + cwd=lib.config.path_top) + +if __name__ == "__main__": + main() diff --git a/scripts/obs/data/build_binpkg.Dockerfile b/scripts/obs/data/build_binpkg.Dockerfile new file mode 100644 index 0000000..feacbc9 --- /dev/null +++ b/scripts/obs/data/build_binpkg.Dockerfile @@ -0,0 +1,72 @@ +ARG DISTRO_FROM +FROM ${DISTRO_FROM} +ARG DISTRO +ARG UID + +COPY Release.key /tmp/Release.key + +RUN useradd --uid=${UID} -m user + +# Only install build-essential here, and what's needed to add the Osmocom +# repository. Everything else must be defined as dependency in the package +# build recipe. For rpm-based distributions, there is no build-essential or +# similar package. Instead add relevant packages from prjconf, e.g.: +# https://build.opensuse.org/projects/CentOS:CentOS-8/prjconf +RUN case "$DISTRO" in \ + debian*) \ + apt-get update && \ + apt-get install -y --no-install-recommends \ + build-essential \ + ca-certificates \ + fakeroot \ + git \ + gnupg2 \ + && \ + apt-get clean \ + ;; \ + centos*) \ + dnf -y install \ + autoconf \ + automake \ + binutils \ + dnf-utils \ + gcc \ + gcc-c++ \ + glibc-devel \ + libtool \ + make \ + redhat-rpm-config \ + rpm-build \ + rpmdevtools \ + wget && \ + yum config-manager --set-enabled powertools && \ + su user -c rpmdev-setuptree \ + ;; \ + esac + +# Add master repository, where packages immediately get updated after merging +# patches to master. +RUN case "$DISTRO" in \ + debian:11) \ + apt-key add /tmp/Release.key && \ + rm /tmp/Release.key && \ + echo "deb https://downloads.osmocom.org/packages/osmocom:/master/Debian_11/ ./" \ + > /etc/apt/sources.list.d/osmocom-master.list \ + ;; \ + centos:8) \ + { echo "[network_osmocom_master]"; \ + echo "name=Nightly packages of the Osmocom project (CentOS_8)"; \ + echo "type=rpm-md"; \ + echo "baseurl=https://downloads.osmocom.org/packages/osmocom:/master/CentOS_8/"; \ + echo "gpgcheck=1"; \ + echo "gpgkey=https://downloads.osmocom.org/packages/osmocom:/master/CentOS_8/repo…"; \ + echo "enabled=1"; \ + } > /etc/yum.repos.d/network:osmocom:master.repo \ + ;; \ + *) \ + echo "can't install repo for $DISTRO" && \ + exit 1 \ + ;; \ + esac + +WORKDIR /obs/ diff --git a/scripts/obs/data/build_deb.sh b/scripts/obs/data/build_deb.sh new file mode 100755 index 0000000..0030278 --- /dev/null +++ b/scripts/obs/data/build_deb.sh @@ -0,0 +1,14 @@ +#!/bin/sh -ex + +apt_get="apt-get" +if [ -n "$INSIDE_DOCKER" ]; then + export DEBIAN_FRONTEND=noninteractive + apt_get="apt-get -y" +fi + +su "$BUILDUSER" -c "tar -C _temp/binpkgs -xvf _temp/srcpkgs/$PACKAGE/*.tar.*" +cd _temp/binpkgs/* + +$apt_get update +$apt_get build-dep . +su "$BUILDUSER" -c "dpkg-buildpackage -us -uc -j$JOBS" diff --git a/scripts/obs/data/build_rpm.sh b/scripts/obs/data/build_rpm.sh new file mode 100755 index 0000000..39f49ee --- /dev/null +++ b/scripts/obs/data/build_rpm.sh @@ -0,0 +1,30 @@ +#!/bin/sh -ex + +if ! [ -d /home/$BUILDUSER/rpmbuild/SOURCES ]; then + set +x + echo "ERROR: rpmdev-setuptree did not run" + echo "If this is an rpm based system and you want to build the package" + echo "here, run rpmdev-setuptree. Otherwise consider building the" + echo "package in docker (-d)." + exit 1 +fi + +yum_builddep="yum-builddep" +if [ -n "$INSIDE_DOCKER" ]; then + yum_builddep="yum-builddep -y" +fi + +spec="$(basename "$(find _temp/srcpkgs/"$PACKAGE" -name '*.spec')")" + +su "$BUILDUSER" -c "cp _temp/srcpkgs/$PACKAGE/$spec ~/rpmbuild/SPECS" +su "$BUILDUSER" -c "cp _temp/srcpkgs/$PACKAGE/*.tar.* ~/rpmbuild/SOURCES" +su "$BUILDUSER" -c "cp _temp/srcpkgs/$PACKAGE/rpmlintrc ~/rpmbuild/SOURCES" + +$yum_builddep "/home/$BUILDUSER/rpmbuild/SPECS/$spec" + +su "$BUILDUSER" -c "rpmbuild -bb ~/rpmbuild/SPECS/$spec" + +# Make built rpms available outside of docker +if [ -n "$INSIDE_DOCKER" ]; then + su "$BUILDUSER" -c "mv ~/rpmbuild/RPMS/*/*.rpm _temp/binpkgs/" +fi diff --git a/scripts/obs/lib/__init__.py b/scripts/obs/lib/__init__.py index 8f52dba..5979623 100644 --- a/scripts/obs/lib/__init__.py +++ b/scripts/obs/lib/__init__.py @@ -14,6 +14,7 @@ def add_shared_arguments(parser): + """ Arguments shared between build_srcpkg.py and update_obs_project.py. """ parser.add_argument("-f", "--feed", help="package feed (default: nightly). The feed" " determines the git revision to be built:" diff --git a/scripts/obs/lib/binpkg_deb.py b/scripts/obs/lib/binpkg_deb.py new file mode 100644 index 0000000..b26623c --- /dev/null +++ b/scripts/obs/lib/binpkg_deb.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright 2022 sysmocom - s.f.m.c. GmbH <info(a)sysmocom.de> +import os +import glob +import lib.config + + +def extract_source(srcdir, bindir): + tarball = glob.glob(f"{srcdir}/*.tar.*")[0] + + print(f"extracting {os.path.basename(tarball)}") + lib.run_cmd(["tar", "-xf", tarball], cwd=bindir) + + return glob.glob(f"{bindir}/*/")[0] + + +def build(srcdir, jobs): + bindir = f"{lib.config.path_temp}/binpkg" + extractdir = extract_source(srcdir, bindir) + + lib.set_cmds_verbose(True) + + # install deps + lib.run_cmd(["apt-get", "-y", "build-dep", "."], cwd=extractdir) + + print("running dpkg-buildpackage") + lib.run_cmd(["dpkg-buildpackage", "-us", "-uc", f"-j{jobs}"], + cwd=extractdir) diff --git a/scripts/obs/lib/binpkg_rpm.py b/scripts/obs/lib/binpkg_rpm.py new file mode 100644 index 0000000..b26623c --- /dev/null +++ b/scripts/obs/lib/binpkg_rpm.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright 2022 sysmocom - s.f.m.c. GmbH <info(a)sysmocom.de> +import os +import glob +import lib.config + + +def extract_source(srcdir, bindir): + tarball = glob.glob(f"{srcdir}/*.tar.*")[0] + + print(f"extracting {os.path.basename(tarball)}") + lib.run_cmd(["tar", "-xf", tarball], cwd=bindir) + + return glob.glob(f"{bindir}/*/")[0] + + +def build(srcdir, jobs): + bindir = f"{lib.config.path_temp}/binpkg" + extractdir = extract_source(srcdir, bindir) + + lib.set_cmds_verbose(True) + + # install deps + lib.run_cmd(["apt-get", "-y", "build-dep", "."], cwd=extractdir) + + print("running dpkg-buildpackage") + lib.run_cmd(["dpkg-buildpackage", "-us", "-uc", f"-j{jobs}"], + cwd=extractdir) diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py index 52b84fb..393251b 100644 --- a/scripts/obs/lib/config.py +++ b/scripts/obs/lib/config.py @@ -112,3 +112,6 @@ } docker_distro_default = "debian:11" +docker_distro_other = [ + "centos:8", +] diff --git a/scripts/obs/lib/docker.py b/scripts/obs/lib/docker.py index 2944cb3..81464bb 100644 --- a/scripts/obs/lib/docker.py +++ b/scripts/obs/lib/docker.py @@ -14,11 +14,23 @@ return ret +def get_distro_from(distro): + # CentOS 8 is EOL (SYS#5818) + if distro == "centos:8": + return "almalinux:8" + + return distro + + def build_image(distro, image_type): - image_name = get_image_name(distro, image_type), + image_name = get_image_name(distro, image_type) + distro_from = get_distro_from(distro) + print(f"docker: building image {image_name}") + lib.run_cmd(["docker", "build", "--build-arg", f"DISTRO={distro}", + "--build-arg", f"DISTRO_FROM={distro_from}", "--build-arg", f"UID={os.getuid()}", "-t", image_name, "-f", f"{lib.config.path_top}/data/{image_type}.Dockerfile", @@ -38,12 +50,15 @@ def run_in_docker_and_exit(script_path, add_oscrc=False, - image_type="build_srcpkg", distro=None): + image_type="build_srcpkg", distro=None, + pass_argv=True, env={}): """ :param script_path: what to run inside docker, relative to scripts/obs/ :param add_oscrc: put user's oscrc in docker (contains obs credentials!) :param image_type: which Dockerfile to use (data/{image_type}.Dockerfile) :param distro: which Linux distribution to use, e.g. "debian:11" + :param pass_argv: pass arguments from sys.argv to the script + :param env: dict of environment variables """ if "INSIDE_DOCKER" in os.environ: return @@ -72,10 +87,17 @@ "-e", "PYTHONUNBUFFERED=1", "-v", f"{lib.config.path_top}:/obs"] + + for env_key, env_val in env.items(): + cmd += ["-e", f"{env_key}={env_val}"] + if oscrc: cmd += ["-v", f"{oscrc}:/home/user/.oscrc"] - cmd += [image_name, f"/obs/{script_path}"] + sys.argv[1:] + cmd += [image_name, f"/obs/{script_path}"] + + if pass_argv: + cmd += sys.argv[1:] print(f"docker: running: {script_path} inside docker") ret = subprocess.run(cmd) -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29601 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I4c6b5d61af35df98cbc70d9ddc8ad36d38a9ce18 Gerrit-Change-Number: 29601 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: run_in_docker_and_exit: remove args param

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29597 ) Change subject: obs: run_in_docker_and_exit: remove args param ...................................................................... obs: run_in_docker_and_exit: remove args param No need to pass args here. It was only used for args.verbose, and that isn't necessary since we don't need to restore the verbose mode. Related: OS#2385 Change-Id: I620cdef46e18f4c66644f14003caf2183c89686f --- M scripts/obs/build_srcpkg.py M scripts/obs/lib/docker.py M scripts/obs/update_obs_project.py 3 files changed, 5 insertions(+), 5 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/97/29597/1 diff --git a/scripts/obs/build_srcpkg.py b/scripts/obs/build_srcpkg.py index 262ead8..61d8d64 100755 --- a/scripts/obs/build_srcpkg.py +++ b/scripts/obs/build_srcpkg.py @@ -31,7 +31,7 @@ lib.set_cmds_verbose(args.verbose) if args.docker: - lib.docker.run_in_docker_and_exit(__file__, args) + lib.docker.run_in_docker_and_exit(__file__) if not args.ignore_req: lib.check_required_programs() diff --git a/scripts/obs/lib/docker.py b/scripts/obs/lib/docker.py index b4ec157..629f239 100644 --- a/scripts/obs/lib/docker.py +++ b/scripts/obs/lib/docker.py @@ -28,7 +28,7 @@ exit(1) -def run_in_docker_and_exit(script_path, args, add_oscrc=False): +def run_in_docker_and_exit(script_path, add_oscrc=False): if "INSIDE_DOCKER" in os.environ: return @@ -41,10 +41,10 @@ oscrc = get_oscrc() # Build the docker image. Unless it is up-to-date, this will take a few - # minutes or so, therefore print the output. + # minutes or so, therefore print the output. No need to restore + # set_cmds_verbose, as we use subprocess.run() below and exit afterwards. lib.set_cmds_verbose(True) build_image() - lib.set_cmds_verbose(args.verbose) cmd = ["docker", "run", "--rm", diff --git a/scripts/obs/update_obs_project.py b/scripts/obs/update_obs_project.py index 9bd46ff..58e5888 100755 --- a/scripts/obs/update_obs_project.py +++ b/scripts/obs/update_obs_project.py @@ -202,7 +202,7 @@ lib.set_cmds_verbose(args.verbose) if args.docker: - lib.docker.run_in_docker_and_exit(__file__, args, True) + lib.docker.run_in_docker_and_exit(__file__, True) lib.osc.check_proj(proj) lib.osc.check_oscrc() -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29597 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I620cdef46e18f4c66644f14003caf2183c89686f Gerrit-Change-Number: 29597 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: run_in_docker_and_exit: --rm on extra line

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29596 ) Change subject: obs: run_in_docker_and_exit: --rm on extra line ...................................................................... obs: run_in_docker_and_exit: --rm on extra line Put it on an extra line like the other arguments to prevent it from getting overlooked. Related: OS#2385 Change-Id: Ie7dea2734e7c47766dfe64b7091bc4f5f72d53a5 --- M scripts/obs/lib/docker.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/96/29596/1 diff --git a/scripts/obs/lib/docker.py b/scripts/obs/lib/docker.py index a5dbcd9..b4ec157 100644 --- a/scripts/obs/lib/docker.py +++ b/scripts/obs/lib/docker.py @@ -47,9 +47,10 @@ lib.set_cmds_verbose(args.verbose) cmd = ["docker", "run", + "--rm", "-e", "INSIDE_DOCKER=1", "-e", "PYTHONUNBUFFERED=1", - "--rm", "-v", f"{lib.config.path_top}:/obs"] + "-v", f"{lib.config.path_top}:/obs"] if oscrc: cmd += ["-v", f"{oscrc}:/home/user/.oscrc"] -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29596 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Ie7dea2734e7c47766dfe64b7091bc4f5f72d53a5 Gerrit-Change-Number: 29596 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: prepare for multiple dockerfiles

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29598 ) Change subject: obs: prepare for multiple dockerfiles ...................................................................... obs: prepare for multiple dockerfiles Rename Dockerfile to build_srcpkg.Dockerfile and adjust related code to allow using a different Dockerfile for building the binary packages in a future patch. Related: OS#2385 Change-Id: I8ef7944a4a81acd6c915998f37139eebad2b2d3e --- R scripts/obs/data/build_srcpkg.Dockerfile M scripts/obs/lib/config.py M scripts/obs/lib/docker.py 3 files changed, 27 insertions(+), 7 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/98/29598/1 diff --git a/scripts/obs/data/Dockerfile b/scripts/obs/data/build_srcpkg.Dockerfile similarity index 100% rename from scripts/obs/data/Dockerfile rename to scripts/obs/data/build_srcpkg.Dockerfile diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py index 79a7849..52b84fb 100644 --- a/scripts/obs/lib/config.py +++ b/scripts/obs/lib/config.py @@ -111,4 +111,4 @@ "open5gs": "^v[0-9]*\\.[0-9]*\\.[0-9]*$", } -docker_image_name = "debian-bullseye-osmocom-obs" +docker_distro_default = "debian:11" diff --git a/scripts/obs/lib/docker.py b/scripts/obs/lib/docker.py index 629f239..0a1f748 100644 --- a/scripts/obs/lib/docker.py +++ b/scripts/obs/lib/docker.py @@ -8,11 +8,20 @@ import lib.config -def build_image(): - print(f"docker: building image {lib.config.docker_image_name}") +def get_image_name(distro, image_type): + ret = f"{distro}-osmocom-obs-{image_type}" + ret = ret.replace(":","-").replace("_","-") + return ret + + +def build_image(distro, image_type): + image_name = get_image_name(distro, image_type), + print(f"docker: building image {image_name}") lib.run_cmd(["docker", "build", + "--build-arg", f"DISTRO={distro}", "--build-arg", f"UID={os.getuid()}", - "-t", lib.config.docker_image_name, + "-t", get_image_name(distro, image_type), + "-f", f"{lib.config.path_top}/data/{image_type}.Dockerfile", f"{lib.config.path_top}/data"]) @@ -28,7 +37,14 @@ exit(1) -def run_in_docker_and_exit(script_path, add_oscrc=False): +def run_in_docker_and_exit(script_path, add_oscrc=False, + image_type="build_srcpkg", distro=None): + """ + :param script_path: what to run inside docker + :param add_oscrc: put user's oscrc in docker (contains obs credentials!) + :param image_type: which Dockerfile to use (data/{image_type}.Dockerfile) + :param distro: which Linux distribution to use, e.g. "debian:11" + """ if "INSIDE_DOCKER" in os.environ: return @@ -36,6 +52,10 @@ print("ERROR: docker is not installed") exit(1) + if not distro: + distro = lib.config.docker_distro_default + image_name = get_image_name(distro, image_type) + oscrc = None if add_oscrc: oscrc = get_oscrc() @@ -44,7 +64,7 @@ # minutes or so, therefore print the output. No need to restore # set_cmds_verbose, as we use subprocess.run() below and exit afterwards. lib.set_cmds_verbose(True) - build_image() + build_image(distro, image_type) cmd = ["docker", "run", "--rm", @@ -56,7 +76,7 @@ cmd += ["-v", f"{oscrc}:/home/user/.oscrc"] script_path = f"/obs/{os.path.basename(script_path)}" - cmd += [lib.config.docker_image_name, script_path] + sys.argv[1:] + cmd += [image_name, script_path] + sys.argv[1:] print(f"docker: running: {os.path.basename(script_path)} inside docker") ret = subprocess.run(cmd) -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29598 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I8ef7944a4a81acd6c915998f37139eebad2b2d3e Gerrit-Change-Number: 29598 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: build_srcpkg_if_needed: switch if branches

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29593 ) Change subject: obs: build_srcpkg_if_needed: switch if branches ...................................................................... obs: build_srcpkg_if_needed: switch if branches Make it slightly easier to read, before extending the latest branch with master too in a follow-up patch. Related: OS#2385 Change-Id: I4f1d4e250eb4d8163f1ded29106a44deb0d0feff --- M scripts/obs/update_obs_project.py 1 file changed, 5 insertions(+), 3 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/93/29593/1 diff --git a/scripts/obs/update_obs_project.py b/scripts/obs/update_obs_project.py index aa33984..168193d 100755 --- a/scripts/obs/update_obs_project.py +++ b/scripts/obs/update_obs_project.py @@ -66,9 +66,9 @@ fetch, is_meta_pkg, skip_up_to_date): global srcpkgs_skipped - if feed != "latest": - print(f"{package}: building source package (feed is {feed})") - else: + if feed == "latest": + """ Check if we can skip this package by comparing the OBS version with + the git remote. """ if is_meta_pkg: latest_version = conflict_version if conflict_version else "1.0.0" else: @@ -95,6 +95,8 @@ else: print(f"{package}: building source package (outdated:" f" {latest_version} <=> {obs_version} in OBS)") + else: + print(f"{package}: building source package (feed is {feed})") build_srcpkg(feed, branch, package, conflict_version, fetch, is_meta_pkg) -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29593 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I4f1d4e250eb4d8163f1ded29106a44deb0d0feff Gerrit-Change-Number: 29593 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: update_obs_project: add master feed

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29594 ) Change subject: obs: update_obs_project: add master feed ...................................................................... obs: update_obs_project: add master feed Add a new master feed, where packages are updated as soon as patches get merged to master. Upload a commit_$COMMIT.txt file for each package in this feed and use it to determine if the package needs to be updated or not. In most packages the commit is already part of the version in the dsc file, e.g. "libosmocore_1.7.0.38.c3b90.dsc", but that's harder to parse and is more likely to have a hash collision (just 5 characters). Related: OS#2385 Change-Id: I3b0b4f4876b8c1eeb61f20d903a6f2cac6e99638 --- M scripts/obs/lib/__init__.py M scripts/obs/lib/config.py M scripts/obs/lib/git.py M scripts/obs/lib/osc.py M scripts/obs/lib/srcpkg.py M scripts/obs/update_obs_project.py 6 files changed, 58 insertions(+), 10 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/94/29594/1 diff --git a/scripts/obs/lib/__init__.py b/scripts/obs/lib/__init__.py index f94c0a8..8f52dba 100644 --- a/scripts/obs/lib/__init__.py +++ b/scripts/obs/lib/__init__.py @@ -17,7 +17,7 @@ parser.add_argument("-f", "--feed", help="package feed (default: nightly). The feed" " determines the git revision to be built:" - " 'nightly' builds 'origin/master'," + " 'nightly' and 'master' build 'origin/master'," " 'latest' builds the last signed tag," " other feeds build their respective branch.", metavar="FEED", default="nightly", diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py index c9df74a..79a7849 100644 --- a/scripts/obs/lib/config.py +++ b/scripts/obs/lib/config.py @@ -32,6 +32,7 @@ "2022q1", "2022q2", "latest", + "master", "nightly", ] diff --git a/scripts/obs/lib/git.py b/scripts/obs/lib/git.py index aab0533..8faf5b2 100644 --- a/scripts/obs/lib/git.py +++ b/scripts/obs/lib/git.py @@ -76,6 +76,28 @@ checkout(project, f"origin/{branch}") +def get_head(project): + repo_path = get_repo_path(project) + ret = lib.run_cmd(["git", "rev-parse", "HEAD"], cwd=repo_path) + return ret.output.rstrip() + + +def get_head_remote(project, branch): + if not branch: + branch = get_default_branch(project) + repo_url = get_repo_url(project) + + print(f"{project}: getting head from git remote for {branch}") + ls_remote = lib.run_cmd(["git", "ls-remote", repo_url, branch]) + + ret = ls_remote.output.split("\t")[0] + if not ret: + lib.exit_error_cmd(ls_remote, f"failed to find head commit for" + "{project} in output") + + return ret + + def get_latest_tag(project): pattern_str = get_latest_tag_pattern(project) pattern = re.compile(pattern_str) diff --git a/scripts/obs/lib/osc.py b/scripts/obs/lib/osc.py index 51e2f5f..df5c34c 100644 --- a/scripts/obs/lib/osc.py +++ b/scripts/obs/lib/osc.py @@ -62,7 +62,7 @@ return ret.output.rstrip().split("\n") -def get_package_version(proj, package): +def get_package_version(proj, package, feed): print(f"{package}: getting OBS version") ret = run_osc(["list", proj, os.path.basename(package)]) @@ -70,15 +70,21 @@ if ret.output == '\n': return "0" - # Extract the version from the dsc filename + # Extract the version from the file list for line in ret.output.split('\n'): line = line.rstrip() - if line.endswith(".dsc"): - return line.split("_")[-1][:-4] + if feed == "master" and package != "osmocom-master": + # Use commit_*.txt + if line.startswith("commit_") and line.endswith(".txt"): + return line.split("_")[1].split(".")[0] + else: + # Use *.dsc + if line.endswith(".dsc"): + return line.split("_")[-1][:-4] lib.exit_error_cmd(ret, "failed to find package version on OBS by" - " extracting the version from the .dsc filename") + " extracting the version from the file list") def create_package(proj, package): diff --git a/scripts/obs/lib/srcpkg.py b/scripts/obs/lib/srcpkg.py index 159f4ed..1711f70 100644 --- a/scripts/obs/lib/srcpkg.py +++ b/scripts/obs/lib/srcpkg.py @@ -2,6 +2,7 @@ # SPDX-License-Identifier: GPL-2.0-or-later # Copyright 2022 sysmocom - s.f.m.c. GmbH <info(a)sysmocom.de> import os +import pathlib import lib.config import lib.debian import lib.rpm_spec @@ -13,7 +14,7 @@ lib.git.checkout(project, f"origin/{branch}") elif feed == "latest": lib.git.checkout_latest_tag(project) - elif feed == "nightly": + elif feed in ["master", "nightly"]: lib.git.checkout_default_branch(project) else: # 2022q1 etc lib.git.checkout(project, f"origin/{feed}") @@ -117,6 +118,18 @@ f.write(f"{version}\n") +def write_commit_txt(project): + """ Write the current git commit to commit_$commit.txt file, so it gets + uploaded to OBS along with the rest of the source package. This allows + figuring out if the source package is still up-to-date or not for the + master feed. """ + output_path = lib.get_output_path(project) + commit = lib.git.get_head(project) + + print(f"{project}: adding commit_{commit}.txt") + pathlib.Path(f"{output_path}/commit_{commit}.txt").touch() + + def build(project, feed, branch, conflict_version, fetch, gerrit_id=0): lib.git.clone(project, fetch) lib.git.clean(project) @@ -155,5 +168,8 @@ lib.rpm_spec.generate(project, version, epoch) lib.rpm_spec.copy_to_output(project) + if feed == "master": + write_commit_txt(project) + lib.remove_cache_extra_files() return version_epoch diff --git a/scripts/obs/update_obs_project.py b/scripts/obs/update_obs_project.py index 168193d..9bd46ff 100755 --- a/scripts/obs/update_obs_project.py +++ b/scripts/obs/update_obs_project.py @@ -66,13 +66,16 @@ fetch, is_meta_pkg, skip_up_to_date): global srcpkgs_skipped - if feed == "latest": + if feed in ["master", "latest"]: """ Check if we can skip this package by comparing the OBS version with the git remote. """ if is_meta_pkg: latest_version = conflict_version if conflict_version else "1.0.0" else: - latest_version = lib.git.get_latest_tag_remote(package) + if feed == "master": + latest_version = lib.git.get_head_remote(package, branch) + else: + latest_version = lib.git.get_latest_tag_remote(package) if latest_version is None: print(f"{package}: skipping (no git tag found)") @@ -82,7 +85,7 @@ if os.path.basename(package) not in pkgs_remote: print(f"{package}: building source package (not in OBS)") else: - obs_version = lib.osc.get_package_version(proj, package) + obs_version = lib.osc.get_package_version(proj, package, feed) if is_up_to_date(obs_version, latest_version): if skip_up_to_date: print(f"{package}: skipping ({obs_version} is up-to-date)") -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29594 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I3b0b4f4876b8c1eeb61f20d903a6f2cac6e99638 Gerrit-Change-Number: 29594 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: jobs/osmocom-obs: add job for master feed

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29595 ) Change subject: jobs/osmocom-obs: add job for master feed ...................................................................... jobs/osmocom-obs: add job for master feed Add one new job for building source packages and sending them to obs.osmocom.org. Trigger it from all master-* jobs. I've also considered adding one job per existing master job that would only update one package at a time (master-libosmocore-obs, master-osmo-bsc-obs, ...). With some additional development effort it should be possible, and it would make each individual master OBS job faster. But given that with the current implementation it only takes 20s to 30s for *all* packages if there are no changes, as it compares git remote HEAD with the version currently on OBS before starting to clone repositories and building the source packages (similar to Osmocom_OBS_latest_obs.osmocom.org), it didn't seem worth optimizing. Set concurrent to false as the triggers from master-builds will likely cause it to run multiple times in parallel otherwise. Related: https://jenkins.osmocom.org/jenkins/view/OBS/job/Osmocom_OBS_master_obs.osm… Related: https://obs.osmocom.org/project/show/osmocom:master Related: OS#2385 Change-Id: I53a494f13f81ae837f2d362c54e1bdf13f121db3 --- M jobs/master-builds.yml M jobs/osmocom-obs.yml M scripts/obs/lib/git.py 3 files changed, 11 insertions(+), 1 deletion(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/95/29595/1 diff --git a/jobs/master-builds.yml b/jobs/master-builds.yml index 7fc306f..e7b4c61 100644 --- a/jobs/master-builds.yml +++ b/jobs/master-builds.yml @@ -491,6 +491,8 @@ resolve-relative-paths: true - trigger: project: '{obj:trigger}' + - trigger: + project: 'Osmocom_OBS_master_obs.osmocom.org' - email: recipients: '{obj:email}' send-to-individuals: true diff --git a/jobs/osmocom-obs.yml b/jobs/osmocom-obs.yml index ff2a18a..0036493 100644 --- a/jobs/osmocom-obs.yml +++ b/jobs/osmocom-obs.yml @@ -4,6 +4,8 @@ jobs: - Osmocom_OBS_{type}_{server} type: + - master: + conflict_version: "" - nightly: # For nightly we don't provide ABI compatibility, make sure packages # from different build dates are not mixed by accident @@ -15,10 +17,16 @@ proj: "network:osmocom" - obs.osmocom.org: proj: "osmocom" + # Pushing to build.opensuse.org is legacy, will be disabled soon (OS#5557). + # Don't push the new master repository there. + exclude: + - type: master + server: build.opensuse.org - job-template: name: 'Osmocom_OBS_{type}_{server}' project-type: freestyle + concurrent: false defaults: global description: | See <a href="https://osmocom.org/projects/cellular-infrastructure/wiki/Binary_Packages">Wiki: binary packages</a> diff --git a/scripts/obs/lib/git.py b/scripts/obs/lib/git.py index 8faf5b2..8081a54 100644 --- a/scripts/obs/lib/git.py +++ b/scripts/obs/lib/git.py @@ -88,7 +88,7 @@ repo_url = get_repo_url(project) print(f"{project}: getting head from git remote for {branch}") - ls_remote = lib.run_cmd(["git", "ls-remote", repo_url, branch]) + ls_remote = lib.run_cmd(["git", "ls-remote", repo_url, f"heads/{branch}"]) ret = ls_remote.output.split("\t")[0] if not ret: -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29595 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I53a494f13f81ae837f2d362c54e1bdf13f121db3 Gerrit-Change-Number: 29595 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: lib.git.get_default_branch: add function

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29592 ) Change subject: obs: lib.git.get_default_branch: add function ...................................................................... obs: lib.git.get_default_branch: add function Refactor checkout_default_branch so the get_default_branch logic can be used from another function in a future patch. Related: OS#2385 Change-Id: If3f0ab500ff165d81f61b19ad41af90bd39dce8f --- M scripts/obs/lib/git.py 1 file changed, 6 insertions(+), 3 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/92/29592/1 diff --git a/scripts/obs/lib/git.py b/scripts/obs/lib/git.py index 8dc57ae..aab0533 100644 --- a/scripts/obs/lib/git.py +++ b/scripts/obs/lib/git.py @@ -65,11 +65,14 @@ lib.run_cmd(["git", "review", "-d", str(gerrit_id)], cwd=repo_path) -def checkout_default_branch(project): - branch = lib.config.git_branch_default +def get_default_branch(project): if project in lib.config.git_branch_other: - branch = lib.config.git_branch_other[project] + return lib.config.git_branch_other[project] + return lib.config.git_branch_default + +def checkout_default_branch(project): + branch = get_default_branch(project) checkout(project, f"origin/{branch}") -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29592 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: If3f0ab500ff165d81f61b19ad41af90bd39dce8f Gerrit-Change-Number: 29592 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in libosmocore[master]: msgb: dont return a length when msgb->lXh is NULL

by laforge

Attention is currently required from: dexter. laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/libosmocore/+/29591 ) Change subject: msgb: dont return a length when msgb->lXh is NULL ...................................................................... Patch Set 1: (1 comment) Patchset: PS1: one could actually argue that we should OSMO_ASSERT in this case, as using msgb_lXlen() on a msgb without lXh pointer set is almost certainly a programming error. -- To view, visit https://gerrit.osmocom.org/c/libosmocore/+/29591 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: libosmocore Gerrit-Branch: master Gerrit-Change-Id: I1795c559f190713ebbabfbabf3453ab77da46a49 Gerrit-Change-Number: 29591 Gerrit-PatchSet: 1 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Attention: dexter <pmaier(a)sysmocom.de> Gerrit-Comment-Date: Tue, 04 Oct 2022 12:53:39 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment

2 years, 8 months

1
0
0 0

Change in libosmocore[master]: msgb: dont return a length when msgb->lXh is NULL

by laforge

Attention is currently required from: dexter. laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/libosmocore/+/29591 ) Change subject: msgb: dont return a length when msgb->lXh is NULL ...................................................................... Patch Set 1: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/libosmocore/+/29591 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: libosmocore Gerrit-Branch: master Gerrit-Change-Id: I1795c559f190713ebbabfbabf3453ab77da46a49 Gerrit-Change-Number: 29591 Gerrit-PatchSet: 1 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Attention: dexter <pmaier(a)sysmocom.de> Gerrit-Comment-Date: Tue, 04 Oct 2022 12:52:57 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

2 years, 8 months

1
0
0 0

2025

2024

2023

2022

gerrit-log October 2022