July 2023 - gerrit-log - lists.osmocom.org

[M] Change in osmo-ci[master]: jobs/gerrit: refactor pipeline script
by osmith 31 Jul '23

31 Jul '23

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/33987 ) Change subject: jobs/gerrit: refactor pipeline script ...................................................................... jobs/gerrit: refactor pipeline script * Deduplicate the code to run a job by moving it to a function. * Print the status of jobs right after they finished, instead of waiting until all jobs are done * Make the status print messages more readable Change-Id: I641a5b483721ce2bbf21bd61d8f4e83faf94ac24 --- M jobs/gerrit-verifications.yml M scripts/jenkins-gerrit/comment_generate.py 2 files changed, 73 insertions(+), 115 deletions(-) Approvals: laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve Jenkins Builder: Verified diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 2bbe261..b77b0ca 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -468,173 +468,114 @@ description: Run struct_endianness.py from libosmocore.git default: '{obj:pipeline_endianness}' dsl: | + def run_job(display_name, job_name, comment_type=null) {{ + try {{ + echo "pipeline_${{display_name}}: running job ${{job_name}}" + + build job: job_name, parameters: [ + string(name: "COMMENT_TYPE", value: comment_type), + string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), + string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), + string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), + string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), + string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), + string(name: "GERRIT_PATCHSET_UPLOADER_NAME", value: "${{env.GERRIT_PATCHSET_UPLOADER_NAME}}"), + string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), + string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), + string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), + string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), + string(name: "PIPELINE_BUILD_URL", value: "${{env.BUILD_URL}}"), + string(name: "PROJECT_NAME", value: "{repos}"), + ] + echo "pipeline_${{display_name}}: SUCCESS" + }} catch (Exception e) {{ + echo "pipeline_${{display_name}}: FAILED" + }} + }} + pipeline {{ agent none stages {{ stage("Verification") {{ parallel {{ + + // Run the comment job to add the pipeline link to gerrit stage("Start Comment") {{ steps {{ - // Run the comment job to add the pipeline link to gerrit script {{ - // Keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - build job: 'gerrit-verifications-comment', parameters: [ - string(name: "COMMENT_TYPE", value: "start"), - string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), - string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), - string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), - string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), - string(name: "GERRIT_PATCHSET_UPLOADER_NAME", value: "${{env.GERRIT_PATCHSET_UPLOADER_NAME}}"), - string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "PIPELINE_BUILD_URL", value: "${{env.BUILD_URL}}"), - ] - }} + run_job("comment_start", "gerrit-verifications-comment", "start") }} }} }} + stage("Build") {{ when {{ expression {{ params.PIPELINE_BUILD }} }} steps {{ script {{ - // Run the build job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_BUILD_PASSED = "0" - build job: 'gerrit-{repos}-build', parameters: [ - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - ] - env.PIPELINE_BUILD_PASSED = "1" - }} + run_job("build", "gerrit-{repos}-build") }} }} }} + stage("Lint") {{ when {{ expression {{ params.PIPELINE_LINT }} }} steps {{ script {{ - // Run the lint job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_LINT_PASSED = "0" - build job: 'gerrit-lint', parameters: [ - string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), - string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), - string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), - string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), - string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), - ] - env.PIPELINE_LINT_PASSED = "1" - }} + run_job("lint", "gerrit-lint") }} }} }} + stage("deb") {{ when {{ expression {{ params.PIPELINE_BINPKGS }} }} steps {{ script {{ - // Run the binpkgs job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_DEB_PASSED = "0" - build job: 'gerrit-binpkgs-deb', parameters: [ - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), - string(name: "PROJECT_NAME", value: "{repos}"), - ] - env.PIPELINE_DEB_PASSED = "1" - }} + run_job("deb", "gerrit-binpkgs-deb") }} }} }} + stage("rpm") {{ - when {{ - expression {{ params.PIPELINE_BINPKGS }} - }} - steps {{ - script {{ - // Run the binpkgs job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_RPM_PASSED = "0" - build job: 'gerrit-binpkgs-rpm', parameters: [ - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), - string(name: "PROJECT_NAME", value: "{repos}"), - ] - env.PIPELINE_RPM_PASSED = "1" - }} + when {{ + expression {{ params.PIPELINE_BINPKGS }} + }} + steps {{ + script {{ + run_job("rpm", "gerrit-binpkgs-rpm") }} }} }} + stage("Endianness") {{ when {{ expression {{ params.PIPELINE_ENDIANNESS }} }} steps {{ script {{ - // Run the endianness job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_ENDIANNESS_PASSED = "0" - build job: 'gerrit-pipeline-endianness', parameters: [ - string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), - string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), - string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), - string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), - string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), - ] - env.PIPELINE_ENDIANNESS_PASSED = "1" - }} + run_job("endianness", "gerrit-pipeline-endianness") }} }} }} - }} - }} + + }} // end of parallel + }} // end of Verification stage + + // Run the comment job to get successful/failed links and add a + // comment + vote to gerrit stage("Result Comment") {{ steps {{ - echo "PIPELINE_BUILD_PASSED=${{env.PIPELINE_BUILD_PASSED}}" - echo "PIPELINE_LINT_PASSED=${{env.PIPELINE_LINT_PASSED}}" - echo "PIPELINE_DEB_PASSED=${{env.PIPELINE_DEB_PASSED}}" - echo "PIPELINE_RPM_PASSED=${{env.PIPELINE_RPM_PASSED}}" - echo "PIPELINE_ENDIANNESS_PASSED=${{env.PIPELINE_ENDIANNESS_PASSED}}" - - // Run the comment job to get successful/failed links and add a - // comment + vote to gerrit script {{ - build job: 'gerrit-verifications-comment', parameters: [ - string(name: "COMMENT_TYPE", value: "result"), - string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), - string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), - string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), - string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), - string(name: "GERRIT_PATCHSET_UPLOADER_NAME", value: "${{env.GERRIT_PATCHSET_UPLOADER_NAME}}"), - string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "PIPELINE_BUILD_URL", value: "${{env.BUILD_URL}}"), - ] + run_job("comment_result", "gerrit-verifications-comment", "result") }} }} }} - // The end result is success if all started jobs were successful, - // and failed otherwise. + }} }} disabled: '{obj:disabled}' diff --git a/scripts/jenkins-gerrit/comment_generate.py b/scripts/jenkins-gerrit/comment_generate.py index 2246fb7..6689ee4 100755 --- a/scripts/jenkins-gerrit/comment_generate.py +++ b/scripts/jenkins-gerrit/comment_generate.py @@ -9,7 +9,7 @@ jenkins_url = "https://jenkins.osmocom.org" re_start_build = re.compile("Starting building: gerrit-[a-zA-Z-_0-9]* #[0-9]*") -re_result = re.compile("^PIPELINE_[A-Z]*_PASSED=[01]$") +re_result = re.compile("^pipeline_([a-zA-Z-_0-9]*): (SUCCESS|FAILED)$") re_job_type = re.compile("JOB_TYPE=([a-zA-Z-_0-9]*),") @@ -74,14 +74,17 @@ ret[stage] = {"url": job_url, "name": job_name, "id": job_id} # Parse result lines - if re_result.match(line): - stage = line.split("_")[1].lower() + match = re_result.match(line) + if match: + stage = match.group(1) + if stage.startswith("comment_"): + # Jobs that run this script, not relevant for summary + continue assert stage in ret, f"found result for stage {stage}, but" \ " didn't find where it was started. The" \ " re_start_build regex probably needs to be adjusted" \ " to match the related gerrit-*-build job." - passed = line.split("=")[1].rstrip() == "1" - ret[stage]["passed"] = passed + ret[stage]["passed"] = (match.group(2) == "SUCCESS") return ret -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33987 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I641a5b483721ce2bbf21bd61d8f4e83faf94ac24 Gerrit-Change-Number: 33987 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

1 0

[M] Change in osmo-ci[master]: jobs/gerrit-binpkgs: don't hardcode distros
by osmith 31 Jul '23

31 Jul '23

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/33988 ) Change subject: jobs/gerrit-binpkgs: don't hardcode distros ...................................................................... jobs/gerrit-binpkgs: don't hardcode distros Make it possible to set a list of distributions to check in gerrit-verifications.yml instead of only having one boolean for testing two hardcoded rpm and deb distributions. Change-Id: I59487e3dc2f55057de1b6a322f088fff0d18654c --- M jobs/gerrit-binpkgs.yml M jobs/gerrit-verifications.yml M scripts/jenkins-gerrit/comment_generate.py 3 files changed, 109 insertions(+), 61 deletions(-) Approvals: laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve Jenkins Builder: Verified diff --git a/jobs/gerrit-binpkgs.yml b/jobs/gerrit-binpkgs.yml index 7a4e9e9..ecfbd52 100644 --- a/jobs/gerrit-binpkgs.yml +++ b/jobs/gerrit-binpkgs.yml @@ -5,13 +5,10 @@ - project: name: gerrit-binpkgs jobs: - - 'gerrit-binpkgs-{type}' - type: - - 'deb' - - 'rpm' + - 'gerrit-binpkgs' -- job-template: - name: 'gerrit-binpkgs-{type}' +- job: + name: 'gerrit-binpkgs' project-type: freestyle node: osmocom-gerrit concurrent: true @@ -21,7 +18,7 @@ days-to-keep: 30 num-to-keep: 1000 description: | - {type} package job of CI for patches sent to + binary packages job (deb/rpm) of CI for patches sent to <a href="https://gerrit.osmocom.org">gerrit</a> </br></br> Related issue: <a href="https://osmocom.org/issues/2385">OS#2385</a> @@ -44,6 +41,9 @@ - string: name: PROJECT_NAME description: project to test, e.g. 'libosmocore' (set by pipeline) + - string: + name: DISTRO + description: distribution to build for, e.g. 'debian:12' (set by pipeline) scm: - git: @@ -62,14 +62,13 @@ builders: - shell: |- - # Skip rpm build if .spec.in file is missing - if [ "{type}" = "rpm" ] && [ -z "$(find code-from-gerrit -name '*.spec.in')" ]; then - set +x - echo - echo "No *.spec.in file found, skipping rpm build." - echo - exit 0 - fi + # Make obvious which distro we build for, for humans and for parsing + # by scripts/jenkins-gerrit/comment_generate.py + set +x + echo + echo "Building binary packages for distro: '$DISTRO'" + echo + set -x # Clone osmo-ci.git rm -rf osmo-ci @@ -80,13 +79,6 @@ osmo-ci git -C osmo-ci log --oneline - # Get distro from type - case "{type}" in - deb) distro="debian:11" ;; - rpm) distro="almalinux:8" ;; - *) echo "unknown package format: '{type}'"; exit 1 ;; - esac - # Move code from gerrit to build_srcpkg.py's git cache cache_dir=osmo-ci/scripts/obs/_cache mkdir -p $cache_dir @@ -106,7 +98,7 @@ # Build binary package ./build_binpkg.py \ - --docker "$distro" \ + --docker "$DISTRO" \ "$PROJECT_NAME" wrappers: - ansicolor: diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index b77b0ca..f2c8bec 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -92,20 +92,21 @@ # Which jobs to run in the pipeline pipeline_build: true pipeline_lint: true - pipeline_binpkgs: true + pipeline_binpkgs: "almalinux:8 debian:11" pipeline_endianness: true # in alphabetical order repos: - asn1c: - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_lint: false - - cellmgr-ng + - cellmgr-ng: + pipeline_binpkgs: "debian:11" - docker-playground: pipeline_build: false - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - libasn1c @@ -125,7 +126,7 @@ - libsmpp34 - libtelnet: - pipeline_binpkgs: false + pipeline_binpkgs: "" - libusrp @@ -143,7 +144,7 @@ -e SMPP="$SMPP" \ -e MGCP="$MGCP" \ {docker_img} {timeout_cmd} /build/contrib/jenkins.sh - pipeline_binpkgs: false + pipeline_binpkgs: "" - openggsn: a1_name: GTP @@ -190,6 +191,7 @@ gerrit_project: 'erlang/{repos}' cmd: '{docker_run} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' pipeline_endianness: false + pipeline_binpkgs: "debian:11" - osmo-e1-hardware: a2_name: JOB_TYPE @@ -206,11 +208,11 @@ -e JOB_TYPE="$JOB_TYPE" \ "$DOCKER_IMG" \ {timeout_cmd} /build/contrib/jenkins.sh - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-ci: pipeline_build: false - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - osmo-ggsn: @@ -229,12 +231,13 @@ - osmo-gsm-manuals: pipeline_endianness: false + pipeline_binpkgs: "debian:11" - osmo_gsup: repos_url: '{gerrit_url}/erlang/{repos}' gerrit_project: 'erlang/{repos}' cmd: '{docker_run_ccache} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-hlr - osmo-iuh @@ -290,6 +293,7 @@ repos_url: '{gerrit_url}/python/{repos}' gerrit_project: 'python/{repos}' pipeline_endianness: false + pipeline_binpkgs: "debian:11" - osmo-sgsn: a1_name: IU @@ -309,7 +313,7 @@ repos_url: '{gerrit_url}/erlang/{repos}' gerrit_project: 'erlang/{repos}' cmd: '{docker_run} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - osmo-sip-connector @@ -336,39 +340,41 @@ "$DOCKER_IMG" {timeout_cmd} /build/contrib/jenkins.sh - osmocom-bb: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-tetra: - pipeline_binpkgs: false + pipeline_binpkgs: "" - - osmo-sysmon + - osmo-sysmon: + pipeline_binpkgs: "debian:11" + - osmo-remsim - simtrace2 - osmo-opencm3-projects: cmd: '{docker_run} {docker_img} {timeout_cmd} /build/contrib/jenkins.sh' - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-asf4-dfu: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-ccid-firmware: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-e1d - osmo-cbc - osmo-e1-recorder: docker_img: '$USER/debian-bullseye-jenkins' # needs python2 (OS#5950) - pipeline_binpkgs: false + pipeline_binpkgs: "" - gapk: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-uecups - osmo-el2tpd: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-smlc: docker_img: '$USER/debian-bullseye-jenkins' # needs python2 (OS#5950) @@ -413,7 +419,7 @@ {timeout_cmd} /build/contrib/jenkins.sh ;; esac - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - osmo-ttcn3-hacks: @@ -431,12 +437,12 @@ su build -c 'make clean'; su build -c 'make compile'; " - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - dahdi-tools: pipeline_lint: false - pipeline_binpkgs: false + pipeline_binpkgs: "" jobs: - 'gerrit-{repos}' @@ -459,21 +465,26 @@ name: REPO_URL description: Clone URL, to be passed to jobs started in the pipeline default: '{obj:repos_url}' - - bool: + - string: name: PIPELINE_BINPKGS - description: Enable the binpkgs job (build deb/rpm packages) + description: | + Enable the binpkgs job (build deb/rpm packages) for the given space + separated distributions. Empty list disables the binpkgs job. The + distributions must have binary packages in the osmocom:master + repository. default: '{obj:pipeline_binpkgs}' - bool: name: PIPELINE_ENDIANNESS description: Run struct_endianness.py from libosmocore.git default: '{obj:pipeline_endianness}' dsl: | - def run_job(display_name, job_name, comment_type=null) {{ + def run_job(display_name, job_name, comment_type=null, distro=null) {{ try {{ echo "pipeline_${{display_name}}: running job ${{job_name}}" build job: job_name, parameters: [ string(name: "COMMENT_TYPE", value: comment_type), + string(name: "DISTRO", value: distro), string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), @@ -530,24 +541,42 @@ }} }} - stage("deb") {{ + // Generating binpkgs stages is not possible if we want to run + // them in parallel and not have another job that just waits + // until they complete. + + stage("binpkgs-0") {{ when {{ - expression {{ params.PIPELINE_BINPKGS }} + expression {{ params.PIPELINE_BINPKGS.split().size() > 0 }} }} steps {{ script {{ - run_job("deb", "gerrit-binpkgs-deb") + distro = params.PIPELINE_BINPKGS.split()[0] + run_job(distro, "gerrit-binpkgs", null, distro) }} }} }} - stage("rpm") {{ + stage("binpkgs-1") {{ when {{ - expression {{ params.PIPELINE_BINPKGS }} + expression {{ params.PIPELINE_BINPKGS.split().size() > 1 }} }} steps {{ script {{ - run_job("rpm", "gerrit-binpkgs-rpm") + distro = params.PIPELINE_BINPKGS.split()[1] + run_job(distro, "gerrit-binpkgs", null, distro) + }} + }} + }} + + stage("binpkgs-2") {{ + when {{ + expression {{ params.PIPELINE_BINPKGS.split().size() > 2 }} + }} + steps {{ + script {{ + distro = params.PIPELINE_BINPKGS.split()[2] + run_job(distro, "gerrit-binpkgs", null, distro) }} }} }} diff --git a/scripts/jenkins-gerrit/comment_generate.py b/scripts/jenkins-gerrit/comment_generate.py index 6689ee4..e96893c 100755 --- a/scripts/jenkins-gerrit/comment_generate.py +++ b/scripts/jenkins-gerrit/comment_generate.py @@ -9,8 +9,9 @@ jenkins_url = "https://jenkins.osmocom.org" re_start_build = re.compile("Starting building: gerrit-[a-zA-Z-_0-9]* #[0-9]*") -re_result = re.compile("^pipeline_([a-zA-Z-_0-9]*): (SUCCESS|FAILED)$") +re_result = re.compile("^pipeline_([a-zA-Z-_0-9:]*): (SUCCESS|FAILED)$") re_job_type = re.compile("JOB_TYPE=([a-zA-Z-_0-9]*),") +re_distro = re.compile("Building binary packages for distro: '([a-zA-Z0-9:].*)'") def parse_args(): @@ -31,16 +32,29 @@ return parser.parse_args() -def stage_from_job_name(job_name): +def stage_binpkgs_from_url(job_url): + """ Multiple gerrit-binpkgs jobs may be started to build binary packages + for multiple distributions. It is not clear from the job name / URL of + a job for which distro it is building, so read it from the log output. + :returns: a distro like "debian:12" """ + global re_distro + + url = f"{job_url}/consoleText" + with urllib.request.urlopen(url) as response: + content = response.read().decode("utf-8") + match = re_distro.search(content) + assert match, f"couldn't find distro name in log: {url}" + return match.group(1) + + +def stage_from_job_name(job_name, job_url): if job_name == "gerrit-verifications-comment": # The job that runs this script. Don't include it in the summary. return None if job_name == "gerrit-lint": return "lint" - if job_name == "gerrit-binpkgs-deb": - return "deb" - if job_name == "gerrit-binpkgs-rpm": - return "rpm" + if job_name == "gerrit-binpkgs": + return stage_binpkgs_from_url(job_url) if job_name == "gerrit-pipeline-endianness": return "endianness" if job_name.endswith("-build"): @@ -69,7 +83,7 @@ job_name = match.split(" ")[2] job_id = int(match.split(" ")[3].replace("#", "")) job_url = f"{jenkins_url}/jenkins/job/{job_name}/{job_id}" - stage = stage_from_job_name(job_name) + stage = stage_from_job_name(job_name, job_url) if stage: ret[stage] = {"url": job_url, "name": job_name, "id": job_id} -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33988 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I59487e3dc2f55057de1b6a322f088fff0d18654c Gerrit-Change-Number: 33988 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

1 0

[S] Change in osmo-ci[master]: Cosmetic: jobs/gerrit: update pipeline comment
by osmith 31 Jul '23

31 Jul '23

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/33986 ) Change subject: Cosmetic: jobs/gerrit: update pipeline comment ...................................................................... Cosmetic: jobs/gerrit: update pipeline comment Change-Id: If06af0e955240e1b9d678f1020767bbfb70b1d96 --- M jobs/gerrit-verifications.yml 1 file changed, 11 insertions(+), 6 deletions(-) Approvals: laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve Jenkins Builder: Verified diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 7b6c1f8..2bbe261 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -24,7 +24,8 @@ # NOTE: after updating the job with Jenkins Job Builder as usual, check if a # new pipeline script was generated and approve it here: # https://jenkins.osmocom.org/jenkins/scriptApproval/ -# This happens when changing the pipeline script, when adding new projects etc. +# This used to be necessary when changing the pipeline script, adding new +# projects etc. But it seems to get auto-approved now. - project: name: gerrit @@ -466,11 +467,6 @@ name: PIPELINE_ENDIANNESS description: Run struct_endianness.py from libosmocore.git default: '{obj:pipeline_endianness}' - # NOTE: jenkins pipelines don't run unless the dsl-script was approved. - # Sadly this doesn't happen automatically when updating the job with - # Jenkins Job Builder. So if you change the pipeline script, update the job - # as usually with JJB and then approve the scripts here: - # https://jenkins.osmocom.org/jenkins/scriptApproval/ dsl: | pipeline {{ agent none -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33986 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: If06af0e955240e1b9d678f1020767bbfb70b1d96 Gerrit-Change-Number: 33986 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

1 0

[S] Change in osmo-ci[master]: jobs/gerrit-verifications: fix binpkgs for libusrp
by osmith 31 Jul '23

31 Jul '23

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/33989 ) Change subject: jobs/gerrit-verifications: fix binpkgs for libusrp ...................................................................... jobs/gerrit-verifications: fix binpkgs for libusrp Don't build libusrp for centos8/almalinux. libusrp does have a .spec.in file, but it needs SDCC for the build and SDCC is not available there. In theory we could build the rpm in CI for opensuse tumbleweed, but it requires adding support for building for opensuse in scripts/obs/data/build_binpkg.Dockerfile and scripts/obs/data/build_rpm.sh first, and since it works quite differently than centos8 and libusrp changes rarely (last code change in 2021) I've decided to not do that now. Fixes: OS#5898 Change-Id: If61765fe628321cae004307f4845d8927a1c7019 --- M jobs/gerrit-verifications.yml 1 file changed, 23 insertions(+), 1 deletion(-) Approvals: laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve Jenkins Builder: Verified diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index f2c8bec..7fbc328 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -128,7 +128,9 @@ - libtelnet: pipeline_binpkgs: "" - - libusrp + - libusrp: + # centos8/almalinux doesn't have SDCC (OS#5898) + pipeline_binpkgs: "debian:11" - openbsc: docker_img: '$USER/debian-bullseye-jenkins' # needs python2 (OS#5950) -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33989 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: If61765fe628321cae004307f4845d8927a1c7019 Gerrit-Change-Number: 33989 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

1 0

[S] Change in osmo-ci[master]: jobs/gerrit-verifications: build for debian 10, 12
by osmith 31 Jul '23

31 Jul '23

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/33990 ) Change subject: jobs/gerrit-verifications: build for debian 10, 12 ...................................................................... jobs/gerrit-verifications: build for debian 10, 12 Build packages in CI for the oldest and newest debian release, for which we provide binary packages in the Osmocom OBS repositories. See "jobs: master/gerrit: use debian bookworm (12)" for reasoning: I079e55a1325083714c8d39f922b2563e843fc0bc Related: OS#6057 Change-Id: I959b466865bd327cc72cde4a1763ac13c2c2d797 --- M jobs/gerrit-verifications.yml 1 file changed, 23 insertions(+), 7 deletions(-) Approvals: laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve Jenkins Builder: Verified diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 7fbc328..1324df6 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -92,7 +92,7 @@ # Which jobs to run in the pipeline pipeline_build: true pipeline_lint: true - pipeline_binpkgs: "almalinux:8 debian:11" + pipeline_binpkgs: "almalinux:8 debian:10 debian:12" pipeline_endianness: true # in alphabetical order @@ -102,7 +102,7 @@ pipeline_lint: false - cellmgr-ng: - pipeline_binpkgs: "debian:11" + pipeline_binpkgs: "debian:10 debian:12" - docker-playground: pipeline_build: false @@ -130,7 +130,7 @@ - libusrp: # centos8/almalinux doesn't have SDCC (OS#5898) - pipeline_binpkgs: "debian:11" + pipeline_binpkgs: "debian:10 debian:12" - openbsc: docker_img: '$USER/debian-bullseye-jenkins' # needs python2 (OS#5950) @@ -193,7 +193,7 @@ gerrit_project: 'erlang/{repos}' cmd: '{docker_run} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' pipeline_endianness: false - pipeline_binpkgs: "debian:11" + pipeline_binpkgs: "debian:10 debian:12" - osmo-e1-hardware: a2_name: JOB_TYPE @@ -233,7 +233,7 @@ - osmo-gsm-manuals: pipeline_endianness: false - pipeline_binpkgs: "debian:11" + pipeline_binpkgs: "debian:10 debian:12" - osmo_gsup: repos_url: '{gerrit_url}/erlang/{repos}' @@ -295,7 +295,7 @@ repos_url: '{gerrit_url}/python/{repos}' gerrit_project: 'python/{repos}' pipeline_endianness: false - pipeline_binpkgs: "debian:11" + pipeline_binpkgs: "debian:10 debian:12" - osmo-sgsn: a1_name: IU @@ -348,7 +348,7 @@ pipeline_binpkgs: "" - osmo-sysmon: - pipeline_binpkgs: "debian:11" + pipeline_binpkgs: "debian:10 debian:12" - osmo-remsim - simtrace2 -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33990 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I959b466865bd327cc72cde4a1763ac13c2c2d797 Gerrit-Change-Number: 33990 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

1 0

[S] Change in osmo-ci[master]: scripts/jenkins-gerrit: make assert more verbose
by osmith 31 Jul '23

31 Jul '23

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/33991 ) Change subject: scripts/jenkins-gerrit: make assert more verbose ...................................................................... scripts/jenkins-gerrit: make assert more verbose Change-Id: I0ea40663220f7eb8ee72e43f3f6b776be5835db2 --- M scripts/jenkins-gerrit/comment_generate.py 1 file changed, 11 insertions(+), 1 deletion(-) Approvals: laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve Jenkins Builder: Verified diff --git a/scripts/jenkins-gerrit/comment_generate.py b/scripts/jenkins-gerrit/comment_generate.py index e96893c..147faac 100755 --- a/scripts/jenkins-gerrit/comment_generate.py +++ b/scripts/jenkins-gerrit/comment_generate.py @@ -97,7 +97,8 @@ assert stage in ret, f"found result for stage {stage}, but" \ " didn't find where it was started. The" \ " re_start_build regex probably needs to be adjusted" \ - " to match the related gerrit-*-build job." + " to match the related gerrit-*-build job.\n\n" \ + f"ret: {ret}" ret[stage]["passed"] = (match.group(2) == "SUCCESS") return ret -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33991 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I0ea40663220f7eb8ee72e43f3f6b776be5835db2 Gerrit-Change-Number: 33991 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

1 0

[M] Change in libosmo-netif[master]: ipa: Add segmentation callback
by daniel 31 Jul '23

31 Jul '23

Attention is currently required from: arehbein, laforge, pespin. daniel has posted comments on this change. ( https://gerrit.osmocom.org/c/libosmo-netif/+/33652 ) Change subject: ipa: Add segmentation callback ...................................................................... Patch Set 5: Code-Review+1 (1 comment) File src/ipa.c: https://gerrit.osmocom.org/c/libosmo-netif/+/33652/comment/6939048f_b997576f PS5, Line 375: #define MSG_CB_IPA_INFO_OFFSET 0 Unused, can be removed -- To view, visit https://gerrit.osmocom.org/c/libosmo-netif/+/33652 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: libosmo-netif Gerrit-Branch: master Gerrit-Change-Id: I87ef4c7023126b783dd79e7ed47be31e1b76f975 Gerrit-Change-Number: 33652 Gerrit-PatchSet: 5 Gerrit-Owner: arehbein <arehbein(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: daniel <dwillmann(a)sysmocom.de> Gerrit-CC: laforge <laforge(a)osmocom.org> Gerrit-CC: pespin <pespin(a)sysmocom.de> Gerrit-Attention: arehbein <arehbein(a)sysmocom.de> Gerrit-Attention: laforge <laforge(a)osmocom.org> Gerrit-Attention: pespin <pespin(a)sysmocom.de> Gerrit-Comment-Date: Mon, 31 Jul 2023 06:36:02 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

1 0

[S] Change in libosmo-netif[master]: examples: Add extension header octet to example
by daniel 31 Jul '23

31 Jul '23

Attention is currently required from: arehbein, pespin. daniel has posted comments on this change. ( https://gerrit.osmocom.org/c/libosmo-netif/+/33202 ) Change subject: examples: Add extension header octet to example ...................................................................... Patch Set 13: Code-Review+2 (1 comment) Patchset: PS13: Adding +2 from before -- To view, visit https://gerrit.osmocom.org/c/libosmo-netif/+/33202 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: libosmo-netif Gerrit-Branch: master Gerrit-Change-Id: I962b9edcba65cdc98da00d2f8753dc5acd481502 Gerrit-Change-Number: 33202 Gerrit-PatchSet: 13 Gerrit-Owner: arehbein <arehbein(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: daniel <dwillmann(a)sysmocom.de> Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: arehbein <arehbein(a)sysmocom.de> Gerrit-Attention: pespin <pespin(a)sysmocom.de> Gerrit-Comment-Date: Mon, 31 Jul 2023 06:28:44 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

1 0

[M] Change in osmo-bts[master]: csd_v110_rtp_encode(): properly set E1/E2/E3 bits
by laforge 30 Jul '23

30 Jul '23

Attention is currently required from: fixeria. laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-bts/+/33934 ) Change subject: csd_v110_rtp_encode(): properly set E1/E2/E3 bits ...................................................................... Patch Set 2: Code-Review+1 -- To view, visit https://gerrit.osmocom.org/c/osmo-bts/+/33934 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-bts Gerrit-Branch: master Gerrit-Change-Id: I1a14597dff746cf975140b294400a2cc05badccd Gerrit-Change-Number: 33934 Gerrit-PatchSet: 2 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Comment-Date: Sun, 30 Jul 2023 20:18:10 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

1 0

[L] Change in osmo-bts[master]: l1sap: proper rate adaptation for CSD (RFC4040 'clearmode')
by laforge 30 Jul '23

30 Jul '23

Attention is currently required from: osmith, fixeria, pespin. laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-bts/+/33832 ) Change subject: l1sap: proper rate adaptation for CSD (RFC4040 'clearmode') ...................................................................... Patch Set 5: Code-Review+1 (1 comment) File src/common/csd_v110.c: https://gerrit.osmocom.org/c/osmo-bts/+/33832/comment/6d773655_85fd15ba PS4, Line 150: ra_bits[i] = (rtp[i] >> 7); > I am basically replicating the implementation from libosmocore here. Here is what I. […] Done -- To view, visit https://gerrit.osmocom.org/c/osmo-bts/+/33832 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-bts Gerrit-Branch: master Gerrit-Change-Id: I5e3701ad52d5d428fd02caff037881045f2d0a02 Gerrit-Change-Number: 33832 Gerrit-PatchSet: 5 Gerrit-Owner: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-Attention: osmith <osmith(a)sysmocom.de> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Attention: pespin <pespin(a)sysmocom.de> Gerrit-Comment-Date: Sun, 30 Jul 2023 20:17:29 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Comment-In-Reply-To: laforge <laforge(a)osmocom.org> Comment-In-Reply-To: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-MessageType: comment

1 0