September 2023 - gerrit-log - lists.osmocom.org

[S] Change in pysim[master]: filesystem: add attribute "leftpad" to class LinFixedEF
by dexter 07 Sep '23

07 Sep '23

dexter has uploaded this change for review. ( https://gerrit.osmocom.org/c/pysim/+/34336?usp=email ) Change subject: filesystem: add attribute "leftpad" to class LinFixedEF ...................................................................... filesystem: add attribute "leftpad" to class LinFixedEF In some cases, the specs do not specify an absolute record length. Instead there may be only a minimum record length specified. The card vendor may then chose to use larger record length at will. This usually is no problem since the data is usually written from the left and the remaining bytes are padded at the end (right side) of the data. However in some rare cases (EF.MSISDN, see also 3GPP TS 51.011, section 10.5.5) the data must be written right-aligned towards the physical record length. This means that the data is padded from the left in this case. To fix this: Let's add a "leftpad" flag to LinFixedEF, which we set to true in those corner cases. The code that updates the record in commands.py must then check this flag and padd the data accordingly. Change-Id: I241d9fd656f9064a3ebb4e8e01a52b6b030f9923 Related: OS#5714 --- M pySim/commands.py M pySim/filesystem.py M pySim/runtime.py M pySim/ts_51_011.py 4 files changed, 38 insertions(+), 8 deletions(-) git pull ssh://gerrit.osmocom.org:29418/pysim refs/changes/36/34336/1 diff --git a/pySim/commands.py b/pySim/commands.py index 477cb2b..e072069 100644 --- a/pySim/commands.py +++ b/pySim/commands.py @@ -26,7 +26,7 @@ from construct import * from pySim.construct import LV -from pySim.utils import rpad, b2h, h2b, sw_match, bertlv_encode_len, Hexstr, h2i, str_sanitize, expand_hex +from pySim.utils import rpad, lpad, b2h, h2b, sw_match, bertlv_encode_len, Hexstr, h2i, str_sanitize, expand_hex from pySim.utils import Hexstr, SwHexstr, ResTuple from pySim.exceptions import SwMatchError from pySim.transport import LinkBase @@ -269,7 +269,7 @@ data.lower(), res[0].lower())) def update_record(self, ef: Path, rec_no: int, data: Hexstr, force_len: bool = False, - verify: bool = False, conserve: bool = False) -> ResTuple: + verify: bool = False, conserve: bool = False, leftpad: bool = False) -> ResTuple: """Execute UPDATE RECORD. Args: @@ -279,6 +279,7 @@ force_len : enforce record length by using the actual data length verify : verify data by re-reading the record conserve : read record and compare it with data, skip write on match + leftpad : apply 0xff padding from the left instead from the right side. """ res = self.select_path(ef) @@ -295,7 +296,10 @@ raise ValueError('Data length exceeds record length (expected max %d, got %d)' % ( rec_length, len(data) // 2)) elif (len(data) // 2 < rec_length): - data = rpad(data, rec_length * 2) + if leftpad: + data = lpad(data, rec_length * 2) + else: + data = rpad(data, rec_length * 2) # Save write cycles by reading+comparing before write if conserve: diff --git a/pySim/filesystem.py b/pySim/filesystem.py index 9f3ee17..5950ad1 100644 --- a/pySim/filesystem.py +++ b/pySim/filesystem.py @@ -920,7 +920,7 @@ self._cmd.poutput_json(data) def __init__(self, fid: str, sfid: str = None, name: str = None, desc: str = None, - parent: Optional[CardDF] = None, rec_len: Size = (1, None), **kwargs): + parent: Optional[CardDF] = None, rec_len: Size = (1, None), leftpad: bool = False, **kwargs): """ Args: fid : File Identifier (4 hex digits) @@ -929,9 +929,11 @@ desc : Description of the file parent : Parent CardFile object within filesystem hierarchy rec_len : Tuple of (minimum_length, recommended_length) + leftpad: On write, data must be padded from the left to fit pysical record length """ super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, parent=parent, **kwargs) self.rec_len = rec_len + self.leftpad = leftpad self.shell_commands = [self.ShellCommands()] self._construct = None self._tlv = None diff --git a/pySim/runtime.py b/pySim/runtime.py index 422e916..d6c6d19 100644 --- a/pySim/runtime.py +++ b/pySim/runtime.py @@ -458,7 +458,9 @@ """ if not isinstance(self.selected_file, LinFixedEF): raise TypeError("Only works with Linear Fixed EF") - return self.rs.card._scc.update_record(self.selected_file.fid, rec_nr, data_hex, conserve=self.rs.conserve_write) + return self.rs.card._scc.update_record(self.selected_file.fid, rec_nr, data_hex, + conserve=self.rs.conserve_write, + leftpad=self.selected_file.leftpad) def update_record_dec(self, rec_nr: int, data: dict): """Update a record with given abstract data. Will encode abstract to binary data diff --git a/pySim/ts_51_011.py b/pySim/ts_51_011.py index c81bfdf..d8cbabd 100644 --- a/pySim/ts_51_011.py +++ b/pySim/ts_51_011.py @@ -181,7 +181,7 @@ # TS 51.011 Section 10.5.5 class EF_MSISDN(LinFixedEF): def __init__(self, fid='6f40', sfid=None, name='EF.MSISDN', desc='MSISDN', **kwargs): - super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(15, 34), **kwargs) + super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(15, 34), leftpad=True, **kwargs) def _decode_record_hex(self, raw_hex_data, **kwargs): return {'msisdn': dec_msisdn(raw_hex_data)} @@ -192,8 +192,7 @@ encoded_msisdn = enc_msisdn(msisdn) else: encoded_msisdn = enc_msisdn(msisdn[2], msisdn[0], msisdn[1]) - alpha_identifier = (list(self.rec_len)[ - 0] - len(encoded_msisdn) // 2) * "ff" + alpha_identifier = (list(self.rec_len)[0] - len(encoded_msisdn) // 2) * "ff" return alpha_identifier + encoded_msisdn # TS 51.011 Section 10.5.6 -- To view, visit https://gerrit.osmocom.org/c/pysim/+/34336?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: pysim Gerrit-Branch: master Gerrit-Change-Id: I241d9fd656f9064a3ebb4e8e01a52b6b030f9923 Gerrit-Change-Number: 34336 Gerrit-PatchSet: 1 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[S] Change in osmo-ci[master]: lint: Add osmo_ss7_asp_rx_unknown_cb to typedefs_osmo.txt
by osmith 07 Sep '23

07 Sep '23

Attention is currently required from: pespin. osmith has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/34330?usp=email ) Change subject: lint: Add osmo_ss7_asp_rx_unknown_cb to typedefs_osmo.txt ...................................................................... Patch Set 1: Code-Review+2 -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/34330?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Ifeb2ed66f32b2c3802ea1aab4857f334c3748da3 Gerrit-Change-Number: 34330 Gerrit-PatchSet: 1 Gerrit-Owner: pespin <pespin(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Attention: pespin <pespin(a)sysmocom.de> Gerrit-Comment-Date: Thu, 07 Sep 2023 12:15:14 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

1 0

[S] Change in osmo-ci[master]: OBS: lib/osc: add private proj variable
by osmith 07 Sep '23

07 Sep '23

Hello Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/c/osmo-ci/+/34332?usp=email to look at the new patch set (#2). Change subject: OBS: lib/osc: add private proj variable ...................................................................... OBS: lib/osc: add private proj variable Adjust lib/osc.py to take the OBS project from a global variable that can be set with lib.osc.set_apiurl(), instead of using lib.args.obs_project. Move the sanity check to set_apiurl(). This is in preparation for adding a new sync script, which will operate on multiple OBS instances. Related: OS#6165 Change-Id: Iabb871bcb432f2fbdaec9cbcab3d663ccf969901 --- M scripts/obs/lib/osc.py M scripts/obs/update_obs_project.py 2 files changed, 29 insertions(+), 18 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/32/34332/2 -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/34332?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Iabb871bcb432f2fbdaec9cbcab3d663ccf969901 Gerrit-Change-Number: 34332 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-MessageType: newpatchset

1 0

[M] Change in osmo-ci[master]: jobs/osmocom-obs-sync: new job
by osmith 07 Sep '23

07 Sep '23

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/34334?usp=email ) Change subject: jobs/osmocom-obs-sync: new job ...................................................................... jobs/osmocom-obs-sync: new job Related: OS#6165 Change-Id: Ibcf02eb5feee2b314ff28861835040c21aea4294 --- A jobs/osmocom-obs-sync.yml 1 file changed, 64 insertions(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/34/34334/1 diff --git a/jobs/osmocom-obs-sync.yml b/jobs/osmocom-obs-sync.yml new file mode 100644 index 0000000..2dd9888 --- /dev/null +++ b/jobs/osmocom-obs-sync.yml @@ -0,0 +1,54 @@ +--- +- job: + name: 'Osmocom_OBS_sync' + project-type: freestyle + concurrent: false + defaults: global + description: | + Sync base distributions (Debian:12 etc.) from openSUSE's OBS to our OBS + (<a href="https://osmocom.org/issues/6165">OS#6165</a>). + properties: + - build-discarder: + days-to-keep: 30 + num-to-keep: 30 + node: obs + parameters: + - string: + name: CI_BRANCH + description: osmo-ci.git branch + default: 'master' + scm: + - git: + branches: + - '$CI_BRANCH' + url: https://gerrit.osmocom.org/osmo-ci + git-config-name: 'Jenkins Builder' + git-config-email: 'jenkins(a)osmocom.org' + builders: + - shell: | + export PYTHONUNBUFFERED=1 + export DISTROS=" + AlmaLinux:8 + Debian:10 + Debian:11 + Debian:12 + Debian:Next + Debian:Testing + Raspbian:10 + Raspbian:11 + Ubuntu:20.04 + Ubuntu:22.04 + Ubuntu:23.04 + " + ./scripts/obs/sync_obs_projects.py \ + --docker \ + $DISTROS + triggers: + - timed: "H H * * *" + wrappers: + - ansicolor: + colormap: xterm + publishers: + - email: + notify-every-unstable-build: true + recipients: 'jenkins-notifications(a)lists.osmocom.org' -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/34334?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Ibcf02eb5feee2b314ff28861835040c21aea4294 Gerrit-Change-Number: 34334 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[S] Change in osmo-ci[master]: OBS: lib/osc: add private proj variable
by osmith 07 Sep '23

07 Sep '23

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/34332?usp=email ) Change subject: OBS: lib/osc: add private proj variable ...................................................................... OBS: lib/osc: add private proj variable Adjust lib/osc.py to take the OBS project from a global variable that can be set with lib.osc.set_apiurl(), instead of using lib.args.obs_project. Move the sanity check to set_apiurl(). This is in preparation for adding a new sync script, which will operate on multiple OBS projects. Related: OS#6165 Change-Id: Iabb871bcb432f2fbdaec9cbcab3d663ccf969901 --- M scripts/obs/lib/osc.py M scripts/obs/update_obs_project.py 2 files changed, 29 insertions(+), 18 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/32/34332/1 diff --git a/scripts/obs/lib/osc.py b/scripts/obs/lib/osc.py index c9040ec..1113032 100644 --- a/scripts/obs/lib/osc.py +++ b/scripts/obs/lib/osc.py @@ -10,15 +10,7 @@ import lib.config apiurl = None - - -def check_proj(): - proj = lib.args.obs_project - if ":" in proj: - return - - print(f"ERROR: this doesn't look like a valid OBS project: {proj}") - exit(1) +proj = None def check_oscrc(): @@ -33,8 +25,16 @@ exit(1) -def set_apiurl(url): +def set_apiurl(url, obs_proj=None): global apiurl + global proj + + if obs_proj is not None: + if ":" not in obs_proj: + print(f"ERROR: this doesn't look like a valid OBS project: {obs_proj}") + exit(1) + proj = obs_proj + apiurl = url @@ -58,7 +58,6 @@ def get_remote_pkgs(): - proj = lib.args.obs_project print(f"OBS: getting packages in {proj}") ret = run_osc(["list", proj]) return ret.output.rstrip().split("\n") @@ -66,7 +65,6 @@ def get_package_version(package): feed = lib.args.feed - proj = lib.args.obs_project print(f"{package}: getting OBS version") ret = run_osc(["list", proj, os.path.basename(package)]) @@ -92,7 +90,6 @@ def create_package(package): - proj = lib.args.obs_project print(f"{package}: creating new OBS package") # cut off repository prefix like in "python/osmo-python-tests" @@ -117,7 +114,6 @@ def update_package(package, version): - proj = lib.args.obs_project print(f"{package}: updating OBS package") # cut off repository prefix like in "python/osmo-python-tests" @@ -146,5 +142,4 @@ def delete_package(package, commit_msg): print(f"{package}: removing from OBS ({commit_msg})") - run_osc(["rdelete", "-m", commit_msg, lib.args.obs_project, - os.path.basename(package)]) + run_osc(["rdelete", "-m", commit_msg, proj, os.path.basename(package)]) diff --git a/scripts/obs/update_obs_project.py b/scripts/obs/update_obs_project.py index 599e5cd..eee58a7 100755 --- a/scripts/obs/update_obs_project.py +++ b/scripts/obs/update_obs_project.py @@ -231,9 +231,8 @@ if args.docker: lib.docker.run_in_docker_and_exit("update_obs_project.py", True) - lib.osc.check_proj() lib.osc.check_oscrc() - lib.osc.set_apiurl(args.apiurl) + lib.osc.set_apiurl(args.apiurl, args.obs_project) if not args.ignore_req: lib.check_required_programs() -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/34332?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Iabb871bcb432f2fbdaec9cbcab3d663ccf969901 Gerrit-Change-Number: 34332 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[S] Change in osmo-ci[master]: OBS: build_srcpkg.Dockerfile: use bookworm
by osmith 07 Sep '23

07 Sep '23

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/34331?usp=email ) Change subject: OBS: build_srcpkg.Dockerfile: use bookworm ...................................................................... OBS: build_srcpkg.Dockerfile: use bookworm rebar3 is in debian bookworm's repositories, hence it is not needed anymore to use osmocom:latest here. Change-Id: Iddd0295ee928fc9a47d4d206e414b78a2561d13d --- M scripts/obs/data/build_srcpkg.Dockerfile M scripts/obs/lib/config.py 2 files changed, 17 insertions(+), 15 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/31/34331/1 diff --git a/scripts/obs/data/build_srcpkg.Dockerfile b/scripts/obs/data/build_srcpkg.Dockerfile index f5d72cd..ebb30cd 100644 --- a/scripts/obs/data/build_srcpkg.Dockerfile +++ b/scripts/obs/data/build_srcpkg.Dockerfile @@ -1,29 +1,19 @@ -FROM debian:bullseye +# Change distro in lib/config.py:docker_distro_default +ARG DISTRO_FROM +FROM ${DISTRO_FROM} ARG UID RUN apt-get update && \ apt-get upgrade -y && \ apt-get install -y --no-install-recommends \ ca-certificates \ - gnupg2 \ - && \ - apt-get clean - -COPY Release.key /tmp/Release.key -RUN apt-key add /tmp/Release.key && \ - rm /tmp/Release.key && \ - echo "deb https://downloads.osmocom.org/packages/osmocom:/latest/Debian_11/ ./" \ - > /etc/apt/sources.list.d/osmocom-latest.list - -RUN apt-get update && \ - apt-get upgrade -y && \ - apt-get install -y --no-install-recommends \ debhelper \ dh-python \ dpkg-dev \ fakeroot \ git \ git-review \ + gnupg2 \ meson \ osc \ python3-setuptools \ diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py index 3942f43..c3b172d 100644 --- a/scripts/obs/lib/config.py +++ b/scripts/obs/lib/config.py @@ -111,7 +111,7 @@ "open5gs": "^v[0-9]*\\.[0-9]*\\.[0-9]*$", } -docker_distro_default = "debian:11" +docker_distro_default = "debian:12" docker_distro_other = [ "almalinux:*", # instead of centos (SYS#5818) "debian:*", -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/34331?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Iddd0295ee928fc9a47d4d206e414b78a2561d13d Gerrit-Change-Number: 34331 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[L] Change in osmo-ci[master]: OBS: add sync_obs_projects.py
by osmith 07 Sep '23

07 Sep '23

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/34333?usp=email ) Change subject: OBS: add sync_obs_projects.py ...................................................................... OBS: add sync_obs_projects.py Add a new script that syncronizes base distributions from openSUSE's OBS, like Debian:12. This is done by getting "meta" and "prjconf" from the source OBS, adjusting it and writing it to our OBS instance. The motivation for this is making our OBS more resilient, by using these synchronized projects we can build packages even if the openSUSE OBS API is not reachable. Related: OS#6165 Change-Id: I00122f7819ba682cb791c370d781377cfb302629 --- M scripts/obs/README M scripts/obs/data/build_srcpkg.Dockerfile M scripts/obs/lib/config.py M scripts/obs/lib/osc.py A scripts/obs/sync_obs_projects.py 5 files changed, 392 insertions(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/33/34333/1 diff --git a/scripts/obs/README b/scripts/obs/README index f761f1e..cf8e201 100644 --- a/scripts/obs/README +++ b/scripts/obs/README @@ -7,6 +7,7 @@ * build_srcpkg.py: build one source package for an Osmocom project * update_obs_project.py: generate source packages and upload them to OBS * build_binpkg.py: build rpm/deb packages for one Osmocom project +* sync_obs_projects.py: sync projects from another instance (OS#6165) Docker ------ diff --git a/scripts/obs/data/build_srcpkg.Dockerfile b/scripts/obs/data/build_srcpkg.Dockerfile index ebb30cd..75482bc 100644 --- a/scripts/obs/data/build_srcpkg.Dockerfile +++ b/scripts/obs/data/build_srcpkg.Dockerfile @@ -7,6 +7,7 @@ apt-get upgrade -y && \ apt-get install -y --no-install-recommends \ ca-certificates \ + colordiff \ debhelper \ dh-python \ dpkg-dev \ @@ -14,6 +15,7 @@ git \ git-review \ gnupg2 \ + libxml2-utils \ meson \ osc \ python3-setuptools \ diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py index c3b172d..038122b 100644 --- a/scripts/obs/lib/config.py +++ b/scripts/obs/lib/config.py @@ -117,3 +117,17 @@ "debian:*", "ubuntu:*", ] + +# +# Options related to sync from build.opensuse.org (OS#6165) +# + +sync_remove_paths = [ + # This path has a kernel-obs-build package that other OBS instances use to + # build armv7l/hl packages, but we don't need it + "OBS:DefaultKernel", +] + +sync_set_maintainers = [ + "osmocom-jenkins", +] diff --git a/scripts/obs/lib/osc.py b/scripts/obs/lib/osc.py index 1113032..656783e 100644 --- a/scripts/obs/lib/osc.py +++ b/scripts/obs/lib/osc.py @@ -143,3 +143,39 @@ def delete_package(package, commit_msg): print(f"{package}: removing from OBS ({commit_msg})") run_osc(["rdelete", "-m", commit_msg, proj, os.path.basename(package)]) + + +def get_prjconf(output_file): + print(f"{proj}: getting prjconf") + prjconf = lib.osc.run_osc(["meta", "prjconf", proj]).output + with open(output_file, "w") as h: + h.write(prjconf) + + +def update_prjconf(prjconf_file, commit_msg): + print(f"{proj}: updating prjconf") + lib.osc.run_osc(["meta", + "prjconf", + "-F", prjconf_file, + "-m", commit_msg, + proj]) + + +def get_meta(output_file): + print(f"{proj}: getting meta") + meta = lib.osc.run_osc(["meta", "prj", proj]).output + with open(output_file, "w") as h: + h.write(meta) + + +def update_meta(meta_file, commit_msg): + print(f"{proj}: updating meta") + lib.osc.run_osc(["meta", + "prj", + "-F", meta_file, + "-m", commit_msg, + proj]) + +def get_projects(): + print(f"OBS: getting list of projects") + return lib.osc.run_osc(["ls"]).output.rstrip().split("\n") diff --git a/scripts/obs/sync_obs_projects.py b/scripts/obs/sync_obs_projects.py new file mode 100755 index 0000000..85ceba3 --- /dev/null +++ b/scripts/obs/sync_obs_projects.py @@ -0,0 +1,321 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright 2023 sysmocom - s.f.m.c. GmbH <info(a)sysmocom.de> +import argparse +import glob +import hashlib +import html +import os +import shlex +import shutil +import xml.etree.ElementTree + +import lib +import lib.docker +import lib.osc + +temp_source_prjconf = f"{lib.config.path_temp}/sync_source_prjconf" +temp_source_meta = f"{lib.config.path_temp}/sync_source_meta" +temp_dest_old_meta = f"{lib.config.path_temp}/sync_dest_old_meta" +temp_dest_old_prjconf = f"{lib.config.path_temp}/sync_dest_old_prjconf" +temp_dest_new_meta = f"{lib.config.path_temp}/sync_dest_new_meta" +temp_dest_new_prjconf = f"{lib.config.path_temp}/sync_dest_new_prjconf" + + +def parse_args(): + parser = argparse.ArgumentParser(description="Sync OBS projects (prjconf," + " meta) from another instance (OS#6165)") + parser.add_argument("-d", "--docker", + help="run in docker to avoid installing required pkgs", + action="store_true") + parser.add_argument("-n", "--no-skip-up-to-date", + dest="skip_up_to_date", action="store_false", + help="always assume projects are outdated") + parser.add_argument("-v", "--verbose", action="store_true", + help="always print shell commands and their output," + " instead of only printing them on error") + parser.add_argument("projects", + help="source OBS project, e.g. Debian:12", + nargs="+") + + advanced = parser.add_argument_group("advanced options") + advanced.add_argument("-A", "--apiurl", help="source OBS API URL" + " (default: https://api.opensuse.org)", + default="https://api.opensuse.org") + advanced.add_argument("-p", "--prefix", default="openSUSE.org-mirror", + help="destination OBS prefix" + " (default: openSUSE.org-mirror)") + advanced.add_argument("-t", "--to-apiurl", help="destination OBS API URL" + " (default: https://obs.osmocom.org)", + default="https://obs.osmocom.org") + advanced.add_argument("-w", "--weburl", default="https://build.opensuse.org", + help="source OBS web URL (default:" + " https://build.opensuse.org)") + + args = parser.parse_args() + lib.set_args(args) + + lib.osc.check_oscrc() + + if args.docker: + lib.docker.run_in_docker_and_exit("sync_obs_projects.py", add_oscrc=True) + + +def check_required_programs(): + required_programs = [ + "colordiff", + "xmllint", + ] + + ok = True + for program in required_programs: + if not shutil.which(program): + print(f"ERROR: missing program: {program}") + ok = False + + if not ok: + print("Either install them or use the -d argument to run in docker") + exit(1) + + +def generate_prjconf_header(project): + """ This header gets prepended to the prjconf, before it gets written to + the destination OBS. This script uses it to determine whether the + project needs to be updated next time it runs. """ + with open(temp_source_prjconf, "rb") as h: + source_prjconf = h.read() + with open(temp_source_meta, "rb") as h: + source_meta = h.read() + + ret = "### This project gets synced from:\n" + ret += f"### {lib.args.weburl}/project/show/{project}\n" + ret += "### \n" + ret += "### Do not modify manually. See OS#6165.\n" + ret += "### \n" + ret += f"### Sync information:\n" + ret += f"### - source meta: {hashlib.md5(source_meta).hexdigest()}\n" + ret += f"### - source prjconf: {hashlib.md5(source_prjconf).hexdigest()}\n" + ret += "\n" + + return ret + + +def is_up_to_date(header, projects, project): + project_new = f"{lib.args.prefix}:{project}" + + if project_new not in projects: + print(f"{project_new}: is outdated (not in destination OBS)") + return False + + lib.osc.get_prjconf(temp_dest_old_prjconf) + with open(temp_dest_old_prjconf, "r") as h: + dest_prjconf = h.read() + + if dest_prjconf.startswith(header): + if not lib.args.skip_up_to_date: + print(f"{project_new}: is up-to-date, but -n is set") + return False + print(f"{project_new}: is up-to-date") + return True + + print(f"{project_new}: is outdated") + return False + + +def get_relevant_arches(project): + if project.startswith("AlmaLinux:"): + return ["x86_64"] + if project.startswith("Raspbian:"): + return ["armv7l"] + + return ["aarch64", + "armv7l", + "i586", + "x86_64"] + + +def rewrite_meta(project): + project_new = f"{lib.args.prefix}:{project}" + print(f"{project}: rewriting meta for {project_new}") + tree = xml.etree.ElementTree.parse(temp_source_meta) + root = tree.getroot() + arches = get_relevant_arches(project) + + # Update <project name="..."> + assert root.get("name") == project + root.set("name", project_new) + + for description in root.findall("description"): + href = f"{lib.args.weburl}/project/show/{project}" + description.text = ("This project gets synced from:" + f" <a href='{html.escape(href)}'>{project}</a>\n" + "Do not modify manually. See OS#6165.\n") + + for repository in root.findall(".repository"): + repo_name = repository.get("name") + print(f" adjusting repository: {repo_name}") + for path in repository.findall(".path"): + # Update <path project="..."> + path_project_old = path.get("project") + path_project_new = f"{lib.args.prefix}:{path_project_old}" + path.set("project", path_project_new) + + # Remove unneeded paths + for path_check in lib.config.sync_remove_paths: + if path_project_old == path_check: + print(f" removing path: {path_project_old}") + repository.remove(path) + break + + # Remove arches we don't build for + for arch in repository.findall(".arch"): + if arch.text not in arches: + print(f" removing arch: {arch.text}") + repository.remove(arch) + for download in repository.findall(".download"): + if download.get("arch") not in arches: + repository.remove(download) + + # Debian: meta configs on build.opensuse.org reference PGP keys with an + # experimental feature that is not yet in the stable version of OBS + # (e.g. <pubkey>debian-archive-12</pubkey>): + # https://github.com/openSUSE/open-build-service/pull/14528 + # Also we don't have such a pubkeydir set up on our OBS server. Assume + # ftp.de.debian.org is a trusted mirror, switch to HTTPS and skip the + # PGP verification by removing the pubkey blocks. + if project.startswith("Debian:"): + for download in repository.findall(".download"): + url = download.get("url") + print(f" changing url to https: {url}") + assert url.startswith("http://ftp.de.debian.org/debian"), \ + f"unexpected mirror URL" + download.set("url", url.replace("http://", "https://")) + for pubkey in download.findall("pubkey"): + download.remove(pubkey) + + # Remove original maintainers + for person in root.findall(".person"): + root.remove(person) + + # Add new maintainers + for userid in lib.config.sync_set_maintainers: + print(f" set maintainer: {userid}") + person = xml.etree.ElementTree.Element("person") + person.set("userid", userid) + person.set("role", "maintainer") + # Insert into same position: after title and description + root.insert(2, person) + + tree.write(temp_dest_new_meta) + + +def rewrite_prjconf(project, header): + project_new = f"{lib.args.prefix}:{project}" + print(f"{project}: rewriting prjconf for {project_new}") + + prjconf = "" + with open(temp_source_prjconf, "r") as f: + for line in f: + line = line.rstrip() + + # Remove unneeded dependencies + if line == "VMInstall: kernel-obs-build": + print(f" commenting out: {line}") + line = f"# (commented out by sync) {line}" + + prjconf += f"{line}\n" + + # Extend the AlmaLinux prjconf to also set CentOS variables, as some of + # our prjconfigs and spec files rely on them + if project == "AlmaLinux:8": + print(f" appending CentOS 8 variables") + prjconf += "\n" + prjconf += "# CentOS 8 compat added by sync script\n" + prjconf += "%define centos_version 800\n" + prjconf += "%define centos_ver 8\n" + prjconf += "%define centos 8\n" + prjconf += "Macros:\n" + prjconf += "%centos_version 800\n" + prjconf += "%centos_ver 8\n" + prjconf += "%centos 8\n" + prjconf += ":Macros\n" + + with open(temp_dest_new_prjconf, "w") as f: + f.write(header) + f.write(prjconf) + + +def show_diff(projects, project): + project_new = f"{lib.args.prefix}:{project}" + if project_new not in projects: + return + + # Show prjconf diff (old prjconf was retrieved in is_up_to_date()) + diff = lib.run_cmd(["colordiff", + "-c3", + temp_dest_old_prjconf, + temp_dest_new_prjconf], + check=False) + if diff.returncode: + print(f"{project_new}: prjconf changes:") + print(diff.output, end="") + else: + print(f"{project_new}: prjconf is unchanged") + + # Show meta diff + lib.osc.get_meta(temp_dest_old_meta) + for file in [temp_dest_old_meta, temp_dest_new_meta]: + lib.run_cmd(f"xmllint --format {shlex.quote(file)} > {shlex.quote(file)}.pretty", + shell=True) + diff = lib.run_cmd(["colordiff", + "-c3", + f"{temp_dest_old_meta}.pretty", + f"{temp_dest_new_meta}.pretty"], + check=False) + if diff.returncode: + print(f"{project_new}: meta changes:") + print(diff.output, end="") + else: + print(f"{project_new}: meta is unchanged") + + +def main(): + parse_args() + check_required_programs() + + os.makedirs(lib.config.path_temp, exist_ok=True) + + # Get destination OBS projects + lib.osc.set_apiurl(lib.args.to_apiurl, None) + dest_projects = lib.osc.get_projects() + + for project in lib.args.projects: + # Talk to source OBS + lib.osc.set_apiurl(lib.args.apiurl, project) + + # Get source prjconf, meta + lib.osc.get_prjconf(temp_source_prjconf) + lib.osc.get_meta(temp_source_meta) + + # Talk to dest OBS + project_new = f"{lib.args.prefix}:{project}" + lib.osc.set_apiurl(lib.args.to_apiurl, project_new) + + # Check if dest is up-to-date + header = generate_prjconf_header(project) + if is_up_to_date(header, dest_projects, project): + continue + + # Rewrite configs and show diff + rewrite_prjconf(project, header) + rewrite_meta(project) + show_diff(dest_projects, project) + + # Update dest prjconf & meta + commit_msg = f"sync with {lib.args.weburl}/project/show/{project}" + lib.osc.update_meta(temp_dest_new_meta, commit_msg) + lib.osc.update_prjconf(temp_dest_new_prjconf, commit_msg) + + +if __name__ == "__main__": + main() -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/34333?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I00122f7819ba682cb791c370d781377cfb302629 Gerrit-Change-Number: 34333 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[XL] Change in libosmo-sccp[master]: ss7: Split asp and xua_server into their own files
by pespin 07 Sep '23

07 Sep '23

pespin has posted comments on this change. ( https://gerrit.osmocom.org/c/libosmo-sccp/+/34329?usp=email ) Change subject: ss7: Split asp and xua_server into their own files ...................................................................... Patch Set 1: (1 comment) Patchset: PS1: Build failure fixed by: https://gerrit.osmocom.org/c/osmo-ci/+/34330 lint: Add osmo_ss7_asp_rx_unknown_cb to typedefs_osmo.txt -- To view, visit https://gerrit.osmocom.org/c/libosmo-sccp/+/34329?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: libosmo-sccp Gerrit-Branch: master Gerrit-Change-Id: I3d43268459d61b0b9f9bec34bf31dc0851fa5e48 Gerrit-Change-Number: 34329 Gerrit-PatchSet: 1 Gerrit-Owner: pespin <pespin(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Comment-Date: Thu, 07 Sep 2023 11:21:40 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment

1 0

[S] Change in osmo-ci[master]: lint: Add osmo_ss7_asp_rx_unknown_cb to typedefs_osmo.txt
by pespin 07 Sep '23

07 Sep '23

pespin has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/34330?usp=email ) Change subject: lint: Add osmo_ss7_asp_rx_unknown_cb to typedefs_osmo.txt ...................................................................... lint: Add osmo_ss7_asp_rx_unknown_cb to typedefs_osmo.txt Change-Id: Ifeb2ed66f32b2c3802ea1aab4857f334c3748da3 --- M lint/checkpatch/typedefs_osmo.txt 1 file changed, 10 insertions(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/30/34330/1 diff --git a/lint/checkpatch/typedefs_osmo.txt b/lint/checkpatch/typedefs_osmo.txt index 1dae49a..204ba0a 100644 --- a/lint/checkpatch/typedefs_osmo.txt +++ b/lint/checkpatch/typedefs_osmo.txt @@ -2,3 +2,4 @@ A_SET_OF l1if_compl_cb ranap_message +osmo_ss7_asp_rx_unknown_cb -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/34330?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Ifeb2ed66f32b2c3802ea1aab4857f334c3748da3 Gerrit-Change-Number: 34330 Gerrit-PatchSet: 1 Gerrit-Owner: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[XL] Change in libosmo-sccp[master]: ss7: Split asp and xua_server into their own files
by Jenkins Builder 07 Sep '23

07 Sep '23

Jenkins Builder has posted comments on this change. ( https://gerrit.osmocom.org/c/libosmo-sccp/+/34329?usp=email ) Change subject: ss7: Split asp and xua_server into their own files ...................................................................... Patch Set 1: (1 comment) File src/osmo_ss7_asp.c: Robot Comment from checkpatch (run ID jenkins-gerrit-lint-10887): https://gerrit.osmocom.org/c/libosmo-sccp/+/34329/comment/76c42182_f93d2553 PS1, Line 1189: static osmo_ss7_asp_rx_unknown_cb *g_osmo_ss7_asp_rx_unknown_cb; need consistent spacing around '*' (ctx:WxV) (or typedef missing in osmo-ci/lint/checkpatch/typedefs_osmo.txt?) -- To view, visit https://gerrit.osmocom.org/c/libosmo-sccp/+/34329?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: libosmo-sccp Gerrit-Branch: master Gerrit-Change-Id: I3d43268459d61b0b9f9bec34bf31dc0851fa5e48 Gerrit-Change-Number: 34329 Gerrit-PatchSet: 1 Gerrit-Owner: pespin <pespin(a)sysmocom.de> Gerrit-CC: Jenkins Builder Gerrit-Comment-Date: Thu, 07 Sep 2023 11:16:55 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment

1 0