Attention is currently required from: osmith.
pespin has posted comments on this change by osmith. ( https://gerrit.osmocom.org/c/osmo-ci/+/41369?usp=email )
Change subject: OBS: set default distro to debian 13
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/41369?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I7c1dcca90e46645a497fd547ce1243b414c53201
Gerrit-Change-Number: 41369
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>
Gerrit-Attention: osmith <osmith(a)sysmocom.de>
Gerrit-Comment-Date: Thu, 06 Nov 2025 14:45:55 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/41370?usp=email )
Change subject: OBS: support building pyhss
......................................................................
OBS: support building pyhss
Build source packages for the python project pyhss similar to how we do
it for erlang projects: by downloading all dependencies ahead of time
and vendoring them in the source package.
Related: SYS#6819
Change-Id: I321090e811f7c5c142bf973c616d83cd5b2219ab
---
M scripts/obs/data/build_srcpkg.Dockerfile
A scripts/obs/data/pyhss_download_deps.sh
M scripts/obs/lib/__init__.py
M scripts/obs/lib/srcpkg.py
4 files changed, 113 insertions(+), 0 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/70/41370/1
diff --git a/scripts/obs/data/build_srcpkg.Dockerfile b/scripts/obs/data/build_srcpkg.Dockerfile
index 8b57931..e0368fb 100644
--- a/scripts/obs/data/build_srcpkg.Dockerfile
+++ b/scripts/obs/data/build_srcpkg.Dockerfile
@@ -3,13 +3,19 @@
FROM ${DISTRO_FROM}
ARG UID
+# default-libmysqlclient-dev: needed for fetching the source package
+# "mysqlclient" with pip that PyHSS depends on. Pip actually compiles the
+# package to figure out its dependency tree and aborts if libmysqlclient-dev is
+# missing (https://github.com/pypa/pip/issues/1884).
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y --no-install-recommends \
ca-certificates \
colordiff \
debhelper \
+ default-libmysqlclient-dev \
dh-python \
+ dh-virtualenv \
dpkg-dev \
erlang-nox \
fakeroot \
@@ -20,7 +26,9 @@
lsb-release \
meson \
osc \
+ pkgconf \
python3-packaging \
+ python3-pip \
python3-setuptools \
quilt \
sed \
diff --git a/scripts/obs/data/pyhss_download_deps.sh b/scripts/obs/data/pyhss_download_deps.sh
new file mode 100644
index 0000000..7dfd5d9
--- /dev/null
+++ b/scripts/obs/data/pyhss_download_deps.sh
@@ -0,0 +1,97 @@
+#!/bin/sh -e
+# Copyright 2025 sysmocom - s.f.m.c. GmbH
+# SPDX-License-Identifier: GPL-3.0-or-later
+# This script downloads all dependencies of PyHSS from the python package
+# index, either as binary package for all python versions and CPU architectures
+# we care about, or as source package depending on what is available.
+
+check_cwd() {
+ if ! [ -e services/hssService.py ]; then
+ echo "ERROR: run this script from the PyHSS directory"
+ exit 1
+ fi
+ if [ -d debian/deps ]; then
+ echo "ERROR: debian/deps exists already!"
+ exit 1
+ fi
+}
+
+download_deps() {
+ local srcpkgs=_temp/requirements-source.txt
+ local binpkgs=_temp/requirements-binary.txt
+ local py_ver
+ local python_versions="
+ 3.11
+ 3.12
+ 3.13
+ "
+ # See e.g. https://pypi.org/project/SQLAlchemy/#files
+ local platform
+ local platforms="
+ manylinux_2_17_aarch64
+ manylinux_2_17_x86_64
+ "
+
+ rm -rf _temp
+ mkdir _temp
+
+ while IFS= read -r line; do
+ case "$line" in
+ # These packages are only available as sources
+ mongo*|pymongo*|mysqlclient*|pysctp*|pycryptodome=*)
+ echo "$line" >>"$srcpkgs"
+ ;;
+ # The rest is available as binary packages. This is
+ # preferred as dependencies for building some of these
+ # are not always available in the target distributions
+ # (e.g. pydantic-core is written in rust and tooling
+ # for building python + rust is not in debian 12).
+ *)
+ echo "$line" >>"$binpkgs"
+ ;;
+ esac
+ done < "requirements.txt"
+
+ # Build system dependencies must also be installed as we will build
+ # offline with --no-index and pip won't use system libraries:
+ # https://github.com/pypa/pip/issues/5696
+ echo "setuptools" >>"$binpkgs"
+ echo "wheel" >>"$binpkgs"
+ echo "hatchling" >>"$binpkgs"
+
+ echo ":: Downloading source packages"
+ pip download \
+ --dest debian/deps \
+ --no-binary=:all: \
+ -r "$srcpkgs"
+
+ for py_ver in $python_versions; do
+ for platform in $platforms; do
+ echo ":: Downloading binary packages (python $py_ver, platform $platform)"
+ local binpkgs_extra=""
+
+ # Redis depends on async-timeout, which has been upstreamed
+ # into Python 3.11+. This means "pip download" may not download
+ # it if it runs with a more recent python version, but older
+ # distros (debian 12) will need it.
+ if [ "$py_ver" = "3.11" ]; then
+ binpkgs_extra="async-timeout"
+ fi
+
+ pip download \
+ --dest debian/deps \
+ --python-version "$py_ver" \
+ --platform "$platform" \
+ --only-binary=:all: \
+ -r "$binpkgs" \
+ $binpkgs_extra
+ done
+ done
+
+ rm -r _temp
+}
+
+check_cwd
+download_deps
+
+echo ":: Success"
diff --git a/scripts/obs/lib/__init__.py b/scripts/obs/lib/__init__.py
index 66090f2..4d0d624 100644
--- a/scripts/obs/lib/__init__.py
+++ b/scripts/obs/lib/__init__.py
@@ -16,6 +16,8 @@
# Print output of commands as they run, not only on error
cmds_verbose = False
+ci_obs_dir = os.path.realpath(f"{__file__}/../..")
+
def add_shared_arguments(parser):
""" Arguments shared between build_srcpkg.py and update_obs_project.py. """
diff --git a/scripts/obs/lib/srcpkg.py b/scripts/obs/lib/srcpkg.py
index 78e49d1..1b31f84 100644
--- a/scripts/obs/lib/srcpkg.py
+++ b/scripts/obs/lib/srcpkg.py
@@ -141,6 +141,12 @@
)
+def prepare_project_pyhss():
+ repo_path = lib.git.get_repo_path("pyhss")
+ script = os.path.join(lib.ci_obs_dir, "data/pyhss_download_deps.sh")
+ lib.run_cmd(["sh", "-e", script], cwd=repo_path)
+
+
def run_generate_build_dep(project):
"""Run contrib/generate_build_dep.sh if it exists in the given project, to
to download sources for dependencies (see e.g. osmo_dia2gsup.git)."""
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/41370?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I321090e811f7c5c142bf973c616d83cd5b2219ab
Gerrit-Change-Number: 41370
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/41369?usp=email )
Change subject: OBS: set default distro to debian 13
......................................................................
OBS: set default distro to debian 13
Change the distribution that is used to build the source packages to
debian 13 so we are on the latest one again.
Change-Id: I7c1dcca90e46645a497fd547ce1243b414c53201
---
M scripts/obs/lib/config.py
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/69/41369/1
diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py
index 8282d9e..0a8d60d 100644
--- a/scripts/obs/lib/config.py
+++ b/scripts/obs/lib/config.py
@@ -134,7 +134,7 @@
"wireshark": tag_pattern('v', c=r'\.[0-9a-z]+'),
}
-docker_distro_default = "debian:12"
+docker_distro_default = "debian:13"
docker_distro_other = [
"almalinux:*", # instead of centos (SYS#5818)
"centos:7", # SYS#6760
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/41369?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I7c1dcca90e46645a497fd547ce1243b414c53201
Gerrit-Change-Number: 41369
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/41366?usp=email )
Change subject: OBS: lib/srcpkg: format with ruff
......................................................................
OBS: lib/srcpkg: format with ruff
Change-Id: I2cfb8fc5bd78449e62526758ddb9263cf3d8ced1
---
M .ruff.toml
M scripts/obs/lib/srcpkg.py
2 files changed, 59 insertions(+), 38 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/66/41366/1
diff --git a/.ruff.toml b/.ruff.toml
index a75a783..c0b9b9d 100644
--- a/.ruff.toml
+++ b/.ruff.toml
@@ -16,7 +16,6 @@
"scripts/obs/lib/metapkg.py",
"scripts/obs/lib/osc.py",
"scripts/obs/lib/rpm_spec.py",
- "scripts/obs/lib/srcpkg.py",
"scripts/obs/update_obs_project.py",
"scripts/osmo-depcheck/buildstack.py",
"scripts/osmo-depcheck/config.py",
diff --git a/scripts/obs/lib/srcpkg.py b/scripts/obs/lib/srcpkg.py
index 7b93937..78e49d1 100644
--- a/scripts/obs/lib/srcpkg.py
+++ b/scripts/obs/lib/srcpkg.py
@@ -10,7 +10,7 @@
def checkout_for_feed(project):
- """ checkout a commit, either latest tag or master or 20YY branch """
+ """checkout a commit, either latest tag or master or 20YY branch"""
feed = lib.args.feed
branch = lib.args.git_branch
if branch:
@@ -24,7 +24,7 @@
def get_git_version(project):
- """ :returns: the string from git-version-gen, e.g. '1.7.0.10-76bdb' """
+ """:returns: the string from git-version-gen, e.g. '1.7.0.10-76bdb'"""
repo_path = lib.git.get_repo_path(project)
# Run git-version-gen if it is in the repository
@@ -41,18 +41,29 @@
pattern = lib.git.get_latest_tag_pattern(project)
pattern = pattern.replace("^", "", 1)
pattern = pattern.replace("$", "", -1)
- result = lib.run_cmd(["git", "describe",
- "--abbrev=4",
- "--tags",
- f"--match={pattern}",
- "HEAD"], cwd=repo_path, check=False)
+ result = lib.run_cmd(
+ [
+ "git",
+ "describe",
+ "--abbrev=4",
+ "--tags",
+ f"--match={pattern}",
+ "HEAD",
+ ],
+ cwd=repo_path,
+ check=False,
+ )
if result.returncode == 128:
print(f"{project}: has no git tags")
- commit = lib.run_cmd(["git", "rev-parse", "HEAD"],
- cwd=repo_path).output[0:4]
- count = lib.run_cmd(["git", "rev-list", "--count", "HEAD"],
- cwd=repo_path).output.rstrip()
+ commit = lib.run_cmd(
+ ["git", "rev-parse", "HEAD"],
+ cwd=repo_path,
+ ).output[0:4]
+ count = lib.run_cmd(
+ ["git", "rev-list", "--count", "HEAD"],
+ cwd=repo_path,
+ ).output.rstrip()
try:
print(f"{project}: getting version from debian/changelog")
version = lib.debian.get_last_version_from_changelog(project)
@@ -105,14 +116,14 @@
def get_epoch(project):
- """ The osmo-gbproxy used to have the same package version as osmo-sgsn
- until 2021 where it was split into its own git repository. From then
- on, osmo-gbproxy has a 0.*.* package version, which is smaller than
- the previous 1.*.* from osmo-sgsn. We had to set the epoch to 1 for
- osmo-gbproxy so package managers know these 0.*.* versions are higher
- than the previous 1.*.* ones that are still found in e.g. debian 11.
- The epoch is set in debian/changelog, retrieve it from there.
- :returns: the epoch number if set, e.g. "1" or an empty string """
+ """The osmo-gbproxy used to have the same package version as osmo-sgsn
+ until 2021 where it was split into its own git repository. From then on,
+ osmo-gbproxy has a 0.*.* package version, which is smaller than the
+ previous 1.*.* from osmo-sgsn. We had to set the epoch to 1 for
+ osmo-gbproxy so package managers know these 0.*.* versions are higher than
+ the previous 1.*.* ones that are still found in e.g. debian 11. The epoch
+ is set in debian/changelog, retrieve it from there.
+ :returns: the epoch number if set, e.g. "1" or an empty string"""
version_epoch = lib.debian.get_last_version_from_changelog(project)
if ":" in version_epoch:
@@ -122,15 +133,17 @@
def prepare_project_open5gs():
- """ Download the subproject sources here, so the package can be built in
- OBS without Internet access. """
- lib.run_cmd(["meson", "subprojects", "download"],
- cwd=lib.git.get_repo_path("open5gs"))
+ """Download the subproject sources here, so the package can be built in
+ OBS without Internet access."""
+ lib.run_cmd(
+ ["meson", "subprojects", "download"],
+ cwd=lib.git.get_repo_path("open5gs"),
+ )
def run_generate_build_dep(project):
- """ Run contrib/generate_build_dep.sh if it exists in the given project, to
- to download sources for dependencies (see e.g. osmo_dia2gsup.git). """
+ """Run contrib/generate_build_dep.sh if it exists in the given project, to
+ to download sources for dependencies (see e.g. osmo_dia2gsup.git)."""
repo_path = lib.git.get_repo_path(project)
script_path = "contrib/generate_build_dep.sh"
@@ -147,10 +160,10 @@
def write_commit_txt(project):
- """ Write the current git commit to commit_$commit.txt file, so it gets
- uploaded to OBS along with the rest of the source package. This allows
- figuring out if the source package is still up-to-date or not for the
- master feed. """
+ """Write the current git commit to commit_$commit.txt file, so it gets
+ uploaded to OBS along with the rest of the source package. This allows
+ figuring out if the source package is still up-to-date or not for the
+ master feed."""
output_path = lib.get_output_path(project)
commit = lib.git.get_head(project)
@@ -160,16 +173,26 @@
def set_asciidoc_style_without_draft_watermark(project):
repo_path = lib.git.get_repo_path(project)
- doc_makefiles = lib.run_cmd(["grep", "-r", "-l", "include $(OSMO_GSM_MANUALS_DIR)/build/Makefile.asciidoc.inc"], cwd=repo_path, check=False)
+ doc_makefiles = lib.run_cmd(
+ ["grep", "-r", "-l", "include $(OSMO_GSM_MANUALS_DIR)/build/Makefile.asciidoc.inc"],
+ cwd=repo_path,
+ check=False,
+ )
doc_makefiles = doc_makefiles.output.rstrip().split("\n")
for doc_makefile in doc_makefiles:
if doc_makefile == "":
continue
print(f"{project}: setting asciidoc style to remove draft watermark in {doc_makefile}")
- lib.run_cmd(["sed", "-i",
- '/\\/build\\/Makefile\\.asciidoc\\.inc/s/^/ ASCIIDOCSTYLE = $(BUILDDIR)\\/custom-dblatex.sty\\n/',
- doc_makefile], cwd=repo_path)
+ lib.run_cmd(
+ [
+ "sed",
+ "-i",
+ "/\\/build\\/Makefile\\.asciidoc\\.inc/s/^/ ASCIIDOCSTYLE = $(BUILDDIR)\\/custom-dblatex.sty\\n/",
+ doc_makefile,
+ ],
+ cwd=repo_path,
+ )
def build(project, gerrit_id=0):
@@ -237,15 +260,14 @@
def requires_osmo_gsm_manuals_dev(project):
- """ Check if an already built source package has osmo-gsm-manuals-dev in
- Build-Depends of the .dsc file """
+ """Check if an already built source package has osmo-gsm-manuals-dev in
+ Build-Depends of the .dsc file"""
path_dsc = glob.glob(f"{lib.get_output_path(project)}/*.dsc")
assert len(path_dsc) == 1, f"failed to get dsc path for {project}"
with open(path_dsc[0], "r") as handle:
for line in handle.readlines():
- if line.startswith("Build-Depends:") \
- and "osmo-gsm-manuals-dev" in line:
+ if line.startswith("Build-Depends:") and "osmo-gsm-manuals-dev" in line:
return True
return False
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/41366?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I2cfb8fc5bd78449e62526758ddb9263cf3d8ced1
Gerrit-Change-Number: 41366
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>