July 2023 - gerrit-log - lists.osmocom.org

[M] Change in osmo-ci[master]: jobs/gerrit: refactor pipeline script
by osmith 28 Jul '23

28 Jul '23

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/33987 ) Change subject: jobs/gerrit: refactor pipeline script ...................................................................... jobs/gerrit: refactor pipeline script * Deduplicate the code to run a job by moving it to a function. * Print the status of jobs right after they finished, instead of waiting until all jobs are done * Make the status print messages more readable Change-Id: I641a5b483721ce2bbf21bd61d8f4e83faf94ac24 --- M jobs/gerrit-verifications.yml M scripts/jenkins-gerrit/comment_generate.py 2 files changed, 73 insertions(+), 115 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/87/33987/1 diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 2bbe261..b77b0ca 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -468,173 +468,114 @@ description: Run struct_endianness.py from libosmocore.git default: '{obj:pipeline_endianness}' dsl: | + def run_job(display_name, job_name, comment_type=null) {{ + try {{ + echo "pipeline_${{display_name}}: running job ${{job_name}}" + + build job: job_name, parameters: [ + string(name: "COMMENT_TYPE", value: comment_type), + string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), + string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), + string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), + string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), + string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), + string(name: "GERRIT_PATCHSET_UPLOADER_NAME", value: "${{env.GERRIT_PATCHSET_UPLOADER_NAME}}"), + string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), + string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), + string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), + string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), + string(name: "PIPELINE_BUILD_URL", value: "${{env.BUILD_URL}}"), + string(name: "PROJECT_NAME", value: "{repos}"), + ] + echo "pipeline_${{display_name}}: SUCCESS" + }} catch (Exception e) {{ + echo "pipeline_${{display_name}}: FAILED" + }} + }} + pipeline {{ agent none stages {{ stage("Verification") {{ parallel {{ + + // Run the comment job to add the pipeline link to gerrit stage("Start Comment") {{ steps {{ - // Run the comment job to add the pipeline link to gerrit script {{ - // Keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - build job: 'gerrit-verifications-comment', parameters: [ - string(name: "COMMENT_TYPE", value: "start"), - string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), - string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), - string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), - string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), - string(name: "GERRIT_PATCHSET_UPLOADER_NAME", value: "${{env.GERRIT_PATCHSET_UPLOADER_NAME}}"), - string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "PIPELINE_BUILD_URL", value: "${{env.BUILD_URL}}"), - ] - }} + run_job("comment_start", "gerrit-verifications-comment", "start") }} }} }} + stage("Build") {{ when {{ expression {{ params.PIPELINE_BUILD }} }} steps {{ script {{ - // Run the build job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_BUILD_PASSED = "0" - build job: 'gerrit-{repos}-build', parameters: [ - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - ] - env.PIPELINE_BUILD_PASSED = "1" - }} + run_job("build", "gerrit-{repos}-build") }} }} }} + stage("Lint") {{ when {{ expression {{ params.PIPELINE_LINT }} }} steps {{ script {{ - // Run the lint job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_LINT_PASSED = "0" - build job: 'gerrit-lint', parameters: [ - string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), - string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), - string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), - string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), - string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), - ] - env.PIPELINE_LINT_PASSED = "1" - }} + run_job("lint", "gerrit-lint") }} }} }} + stage("deb") {{ when {{ expression {{ params.PIPELINE_BINPKGS }} }} steps {{ script {{ - // Run the binpkgs job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_DEB_PASSED = "0" - build job: 'gerrit-binpkgs-deb', parameters: [ - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), - string(name: "PROJECT_NAME", value: "{repos}"), - ] - env.PIPELINE_DEB_PASSED = "1" - }} + run_job("deb", "gerrit-binpkgs-deb") }} }} }} + stage("rpm") {{ - when {{ - expression {{ params.PIPELINE_BINPKGS }} - }} - steps {{ - script {{ - // Run the binpkgs job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_RPM_PASSED = "0" - build job: 'gerrit-binpkgs-rpm', parameters: [ - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), - string(name: "PROJECT_NAME", value: "{repos}"), - ] - env.PIPELINE_RPM_PASSED = "1" - }} + when {{ + expression {{ params.PIPELINE_BINPKGS }} + }} + steps {{ + script {{ + run_job("rpm", "gerrit-binpkgs-rpm") }} }} }} + stage("Endianness") {{ when {{ expression {{ params.PIPELINE_ENDIANNESS }} }} steps {{ script {{ - // Run the endianness job for this repository and keep going on failure - catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ - env.PIPELINE_ENDIANNESS_PASSED = "0" - build job: 'gerrit-pipeline-endianness', parameters: [ - string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), - string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), - string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), - string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), - string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), - ] - env.PIPELINE_ENDIANNESS_PASSED = "1" - }} + run_job("endianness", "gerrit-pipeline-endianness") }} }} }} - }} - }} + + }} // end of parallel + }} // end of Verification stage + + // Run the comment job to get successful/failed links and add a + // comment + vote to gerrit stage("Result Comment") {{ steps {{ - echo "PIPELINE_BUILD_PASSED=${{env.PIPELINE_BUILD_PASSED}}" - echo "PIPELINE_LINT_PASSED=${{env.PIPELINE_LINT_PASSED}}" - echo "PIPELINE_DEB_PASSED=${{env.PIPELINE_DEB_PASSED}}" - echo "PIPELINE_RPM_PASSED=${{env.PIPELINE_RPM_PASSED}}" - echo "PIPELINE_ENDIANNESS_PASSED=${{env.PIPELINE_ENDIANNESS_PASSED}}" - - // Run the comment job to get successful/failed links and add a - // comment + vote to gerrit script {{ - build job: 'gerrit-verifications-comment', parameters: [ - string(name: "COMMENT_TYPE", value: "result"), - string(name: "GERRIT_PROJECT", value: "${{env.GERRIT_PROJECT}}"), - string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), - string(name: "GERRIT_PATCHSET_NUMBER", value: "${{env.GERRIT_PATCHSET_NUMBER}}"), - string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), - string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), - string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), - string(name: "GERRIT_PATCHSET_UPLOADER_NAME", value: "${{env.GERRIT_PATCHSET_UPLOADER_NAME}}"), - string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), - string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), - string(name: "PIPELINE_BUILD_URL", value: "${{env.BUILD_URL}}"), - ] + run_job("comment_result", "gerrit-verifications-comment", "result") }} }} }} - // The end result is success if all started jobs were successful, - // and failed otherwise. + }} }} disabled: '{obj:disabled}' diff --git a/scripts/jenkins-gerrit/comment_generate.py b/scripts/jenkins-gerrit/comment_generate.py index 2246fb7..6689ee4 100755 --- a/scripts/jenkins-gerrit/comment_generate.py +++ b/scripts/jenkins-gerrit/comment_generate.py @@ -9,7 +9,7 @@ jenkins_url = "https://jenkins.osmocom.org" re_start_build = re.compile("Starting building: gerrit-[a-zA-Z-_0-9]* #[0-9]*") -re_result = re.compile("^PIPELINE_[A-Z]*_PASSED=[01]$") +re_result = re.compile("^pipeline_([a-zA-Z-_0-9]*): (SUCCESS|FAILED)$") re_job_type = re.compile("JOB_TYPE=([a-zA-Z-_0-9]*),") @@ -74,14 +74,17 @@ ret[stage] = {"url": job_url, "name": job_name, "id": job_id} # Parse result lines - if re_result.match(line): - stage = line.split("_")[1].lower() + match = re_result.match(line) + if match: + stage = match.group(1) + if stage.startswith("comment_"): + # Jobs that run this script, not relevant for summary + continue assert stage in ret, f"found result for stage {stage}, but" \ " didn't find where it was started. The" \ " re_start_build regex probably needs to be adjusted" \ " to match the related gerrit-*-build job." - passed = line.split("=")[1].rstrip() == "1" - ret[stage]["passed"] = passed + ret[stage]["passed"] = (match.group(2) == "SUCCESS") return ret -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33987 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I641a5b483721ce2bbf21bd61d8f4e83faf94ac24 Gerrit-Change-Number: 33987 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[M] Change in osmo-ci[master]: jobs/gerrit-binpkgs: don't hardcode distros
by osmith 28 Jul '23

28 Jul '23

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/33988 ) Change subject: jobs/gerrit-binpkgs: don't hardcode distros ...................................................................... jobs/gerrit-binpkgs: don't hardcode distros Make it possible to set a list of distributions to check in gerrit-verifications.yml instead of only having one boolean for testing two hardcoded rpm and deb distributions. Change-Id: I59487e3dc2f55057de1b6a322f088fff0d18654c --- M jobs/gerrit-binpkgs.yml M jobs/gerrit-verifications.yml M scripts/jenkins-gerrit/comment_generate.py 3 files changed, 109 insertions(+), 61 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/88/33988/1 diff --git a/jobs/gerrit-binpkgs.yml b/jobs/gerrit-binpkgs.yml index 7a4e9e9..ecfbd52 100644 --- a/jobs/gerrit-binpkgs.yml +++ b/jobs/gerrit-binpkgs.yml @@ -5,13 +5,10 @@ - project: name: gerrit-binpkgs jobs: - - 'gerrit-binpkgs-{type}' - type: - - 'deb' - - 'rpm' + - 'gerrit-binpkgs' -- job-template: - name: 'gerrit-binpkgs-{type}' +- job: + name: 'gerrit-binpkgs' project-type: freestyle node: osmocom-gerrit concurrent: true @@ -21,7 +18,7 @@ days-to-keep: 30 num-to-keep: 1000 description: | - {type} package job of CI for patches sent to + binary packages job (deb/rpm) of CI for patches sent to <a href="https://gerrit.osmocom.org">gerrit</a> </br></br> Related issue: <a href="https://osmocom.org/issues/2385">OS#2385</a> @@ -44,6 +41,9 @@ - string: name: PROJECT_NAME description: project to test, e.g. 'libosmocore' (set by pipeline) + - string: + name: DISTRO + description: distribution to build for, e.g. 'debian:12' (set by pipeline) scm: - git: @@ -62,14 +62,13 @@ builders: - shell: |- - # Skip rpm build if .spec.in file is missing - if [ "{type}" = "rpm" ] && [ -z "$(find code-from-gerrit -name '*.spec.in')" ]; then - set +x - echo - echo "No *.spec.in file found, skipping rpm build." - echo - exit 0 - fi + # Make obvious which distro we build for, for humans and for parsing + # by scripts/jenkins-gerrit/comment_generate.py + set +x + echo + echo "Building binary packages for distro: '$DISTRO'" + echo + set -x # Clone osmo-ci.git rm -rf osmo-ci @@ -80,13 +79,6 @@ osmo-ci git -C osmo-ci log --oneline - # Get distro from type - case "{type}" in - deb) distro="debian:11" ;; - rpm) distro="almalinux:8" ;; - *) echo "unknown package format: '{type}'"; exit 1 ;; - esac - # Move code from gerrit to build_srcpkg.py's git cache cache_dir=osmo-ci/scripts/obs/_cache mkdir -p $cache_dir @@ -106,7 +98,7 @@ # Build binary package ./build_binpkg.py \ - --docker "$distro" \ + --docker "$DISTRO" \ "$PROJECT_NAME" wrappers: - ansicolor: diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index b77b0ca..f2c8bec 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -92,20 +92,21 @@ # Which jobs to run in the pipeline pipeline_build: true pipeline_lint: true - pipeline_binpkgs: true + pipeline_binpkgs: "almalinux:8 debian:11" pipeline_endianness: true # in alphabetical order repos: - asn1c: - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_lint: false - - cellmgr-ng + - cellmgr-ng: + pipeline_binpkgs: "debian:11" - docker-playground: pipeline_build: false - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - libasn1c @@ -125,7 +126,7 @@ - libsmpp34 - libtelnet: - pipeline_binpkgs: false + pipeline_binpkgs: "" - libusrp @@ -143,7 +144,7 @@ -e SMPP="$SMPP" \ -e MGCP="$MGCP" \ {docker_img} {timeout_cmd} /build/contrib/jenkins.sh - pipeline_binpkgs: false + pipeline_binpkgs: "" - openggsn: a1_name: GTP @@ -190,6 +191,7 @@ gerrit_project: 'erlang/{repos}' cmd: '{docker_run} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' pipeline_endianness: false + pipeline_binpkgs: "debian:11" - osmo-e1-hardware: a2_name: JOB_TYPE @@ -206,11 +208,11 @@ -e JOB_TYPE="$JOB_TYPE" \ "$DOCKER_IMG" \ {timeout_cmd} /build/contrib/jenkins.sh - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-ci: pipeline_build: false - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - osmo-ggsn: @@ -229,12 +231,13 @@ - osmo-gsm-manuals: pipeline_endianness: false + pipeline_binpkgs: "debian:11" - osmo_gsup: repos_url: '{gerrit_url}/erlang/{repos}' gerrit_project: 'erlang/{repos}' cmd: '{docker_run_ccache} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-hlr - osmo-iuh @@ -290,6 +293,7 @@ repos_url: '{gerrit_url}/python/{repos}' gerrit_project: 'python/{repos}' pipeline_endianness: false + pipeline_binpkgs: "debian:11" - osmo-sgsn: a1_name: IU @@ -309,7 +313,7 @@ repos_url: '{gerrit_url}/erlang/{repos}' gerrit_project: 'erlang/{repos}' cmd: '{docker_run} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - osmo-sip-connector @@ -336,39 +340,41 @@ "$DOCKER_IMG" {timeout_cmd} /build/contrib/jenkins.sh - osmocom-bb: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-tetra: - pipeline_binpkgs: false + pipeline_binpkgs: "" - - osmo-sysmon + - osmo-sysmon: + pipeline_binpkgs: "debian:11" + - osmo-remsim - simtrace2 - osmo-opencm3-projects: cmd: '{docker_run} {docker_img} {timeout_cmd} /build/contrib/jenkins.sh' - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-asf4-dfu: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-ccid-firmware: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-e1d - osmo-cbc - osmo-e1-recorder: docker_img: '$USER/debian-bullseye-jenkins' # needs python2 (OS#5950) - pipeline_binpkgs: false + pipeline_binpkgs: "" - gapk: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-uecups - osmo-el2tpd: - pipeline_binpkgs: false + pipeline_binpkgs: "" - osmo-smlc: docker_img: '$USER/debian-bullseye-jenkins' # needs python2 (OS#5950) @@ -413,7 +419,7 @@ {timeout_cmd} /build/contrib/jenkins.sh ;; esac - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - osmo-ttcn3-hacks: @@ -431,12 +437,12 @@ su build -c 'make clean'; su build -c 'make compile'; " - pipeline_binpkgs: false + pipeline_binpkgs: "" pipeline_endianness: false - dahdi-tools: pipeline_lint: false - pipeline_binpkgs: false + pipeline_binpkgs: "" jobs: - 'gerrit-{repos}' @@ -459,21 +465,26 @@ name: REPO_URL description: Clone URL, to be passed to jobs started in the pipeline default: '{obj:repos_url}' - - bool: + - string: name: PIPELINE_BINPKGS - description: Enable the binpkgs job (build deb/rpm packages) + description: | + Enable the binpkgs job (build deb/rpm packages) for the given space + separated distributions. Empty list disables the binpkgs job. The + distributions must have binary packages in the osmocom:master + repository. default: '{obj:pipeline_binpkgs}' - bool: name: PIPELINE_ENDIANNESS description: Run struct_endianness.py from libosmocore.git default: '{obj:pipeline_endianness}' dsl: | - def run_job(display_name, job_name, comment_type=null) {{ + def run_job(display_name, job_name, comment_type=null, distro=null) {{ try {{ echo "pipeline_${{display_name}}: running job ${{job_name}}" build job: job_name, parameters: [ string(name: "COMMENT_TYPE", value: comment_type), + string(name: "DISTRO", value: distro), string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), string(name: "GERRIT_CHANGE_NUMBER", value: "${{env.GERRIT_CHANGE_NUMBER}}"), string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), @@ -530,24 +541,42 @@ }} }} - stage("deb") {{ + // Generating binpkgs stages is not possible if we want to run + // them in parallel and not have another job that just waits + // until they complete. + + stage("binpkgs-0") {{ when {{ - expression {{ params.PIPELINE_BINPKGS }} + expression {{ params.PIPELINE_BINPKGS.split().size() > 0 }} }} steps {{ script {{ - run_job("deb", "gerrit-binpkgs-deb") + distro = params.PIPELINE_BINPKGS.split()[0] + run_job(distro, "gerrit-binpkgs", null, distro) }} }} }} - stage("rpm") {{ + stage("binpkgs-1") {{ when {{ - expression {{ params.PIPELINE_BINPKGS }} + expression {{ params.PIPELINE_BINPKGS.split().size() > 1 }} }} steps {{ script {{ - run_job("rpm", "gerrit-binpkgs-rpm") + distro = params.PIPELINE_BINPKGS.split()[1] + run_job(distro, "gerrit-binpkgs", null, distro) + }} + }} + }} + + stage("binpkgs-2") {{ + when {{ + expression {{ params.PIPELINE_BINPKGS.split().size() > 2 }} + }} + steps {{ + script {{ + distro = params.PIPELINE_BINPKGS.split()[2] + run_job(distro, "gerrit-binpkgs", null, distro) }} }} }} diff --git a/scripts/jenkins-gerrit/comment_generate.py b/scripts/jenkins-gerrit/comment_generate.py index 6689ee4..e96893c 100755 --- a/scripts/jenkins-gerrit/comment_generate.py +++ b/scripts/jenkins-gerrit/comment_generate.py @@ -9,8 +9,9 @@ jenkins_url = "https://jenkins.osmocom.org" re_start_build = re.compile("Starting building: gerrit-[a-zA-Z-_0-9]* #[0-9]*") -re_result = re.compile("^pipeline_([a-zA-Z-_0-9]*): (SUCCESS|FAILED)$") +re_result = re.compile("^pipeline_([a-zA-Z-_0-9:]*): (SUCCESS|FAILED)$") re_job_type = re.compile("JOB_TYPE=([a-zA-Z-_0-9]*),") +re_distro = re.compile("Building binary packages for distro: '([a-zA-Z0-9:].*)'") def parse_args(): @@ -31,16 +32,29 @@ return parser.parse_args() -def stage_from_job_name(job_name): +def stage_binpkgs_from_url(job_url): + """ Multiple gerrit-binpkgs jobs may be started to build binary packages + for multiple distributions. It is not clear from the job name / URL of + a job for which distro it is building, so read it from the log output. + :returns: a distro like "debian:12" """ + global re_distro + + url = f"{job_url}/consoleText" + with urllib.request.urlopen(url) as response: + content = response.read().decode("utf-8") + match = re_distro.search(content) + assert match, f"couldn't find distro name in log: {url}" + return match.group(1) + + +def stage_from_job_name(job_name, job_url): if job_name == "gerrit-verifications-comment": # The job that runs this script. Don't include it in the summary. return None if job_name == "gerrit-lint": return "lint" - if job_name == "gerrit-binpkgs-deb": - return "deb" - if job_name == "gerrit-binpkgs-rpm": - return "rpm" + if job_name == "gerrit-binpkgs": + return stage_binpkgs_from_url(job_url) if job_name == "gerrit-pipeline-endianness": return "endianness" if job_name.endswith("-build"): @@ -69,7 +83,7 @@ job_name = match.split(" ")[2] job_id = int(match.split(" ")[3].replace("#", "")) job_url = f"{jenkins_url}/jenkins/job/{job_name}/{job_id}" - stage = stage_from_job_name(job_name) + stage = stage_from_job_name(job_name, job_url) if stage: ret[stage] = {"url": job_url, "name": job_name, "id": job_id} -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33988 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I59487e3dc2f55057de1b6a322f088fff0d18654c Gerrit-Change-Number: 33988 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[S] Change in osmo-ci[master]: Cosmetic: jobs/gerrit: update pipeline comment
by osmith 28 Jul '23

28 Jul '23

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/33986 ) Change subject: Cosmetic: jobs/gerrit: update pipeline comment ...................................................................... Cosmetic: jobs/gerrit: update pipeline comment Change-Id: If06af0e955240e1b9d678f1020767bbfb70b1d96 --- M jobs/gerrit-verifications.yml 1 file changed, 11 insertions(+), 6 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/86/33986/1 diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 7b6c1f8..2bbe261 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -24,7 +24,8 @@ # NOTE: after updating the job with Jenkins Job Builder as usual, check if a # new pipeline script was generated and approve it here: # https://jenkins.osmocom.org/jenkins/scriptApproval/ -# This happens when changing the pipeline script, when adding new projects etc. +# This used to be necessary when changing the pipeline script, adding new +# projects etc. But it seems to get auto-approved now. - project: name: gerrit @@ -466,11 +467,6 @@ name: PIPELINE_ENDIANNESS description: Run struct_endianness.py from libosmocore.git default: '{obj:pipeline_endianness}' - # NOTE: jenkins pipelines don't run unless the dsl-script was approved. - # Sadly this doesn't happen automatically when updating the job with - # Jenkins Job Builder. So if you change the pipeline script, update the job - # as usually with JJB and then approve the scripts here: - # https://jenkins.osmocom.org/jenkins/scriptApproval/ dsl: | pipeline {{ agent none -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/33986 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: If06af0e955240e1b9d678f1020767bbfb70b1d96 Gerrit-Change-Number: 33986 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

1 0

[S] Change in pysim[master]: utils: tolerate uninitialized fields in dec_addr_tlv
by laforge 28 Jul '23

28 Jul '23

laforge has submitted this change. ( https://gerrit.osmocom.org/c/pysim/+/33954 ) Change subject: utils: tolerate uninitialized fields in dec_addr_tlv ...................................................................... utils: tolerate uninitialized fields in dec_addr_tlv TLV fields holding an address may still be uninitialized and hence filled with 0xff bytes. Lets interpret those fields in the same way as we interpret empty fields. Related: OS#6094 Change-Id: Idc0a92ea88756266381c8da2ad62de061a8ea7a1 --- M pySim/utils.py 1 file changed, 18 insertions(+), 0 deletions(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved diff --git a/pySim/utils.py b/pySim/utils.py index 735468e..1541e2e 100644 --- a/pySim/utils.py +++ b/pySim/utils.py @@ -771,6 +771,10 @@ if tlv[1] == 0: continue + # Uninitialized field + if all([v == 0xff for v in tlv[2]]): + continue + # First byte in the value has the address type addr_type = tlv[2][0] # TODO: Support parsing of IPv6 -- To view, visit https://gerrit.osmocom.org/c/pysim/+/33954 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: pysim Gerrit-Branch: master Gerrit-Change-Id: Idc0a92ea88756266381c8da2ad62de061a8ea7a1 Gerrit-Change-Number: 33954 Gerrit-PatchSet: 2 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-MessageType: merged

1 0

[S] Change in pysim[master]: pySim-trace: mark card reset in the trace
by laforge 28 Jul '23

28 Jul '23

laforge has submitted this change. ( https://gerrit.osmocom.org/c/pysim/+/33960 ) Change subject: pySim-trace: mark card reset in the trace ...................................................................... pySim-trace: mark card reset in the trace The trace log currently does not contain any information about card resets. This makes the trace difficult to follow. Let's use the CardReset object to display the ATR in the trace. Related: OS#6094 Change-Id: Ia550a8bd2f45d2ad622cb2ac2a2905397db76bce --- M pySim-trace.py M pySim/apdu/__init__.py M pySim/apdu_source/gsmtap.py M pySim/apdu_source/pyshark_gsmtap.py M pySim/apdu_source/pyshark_rspro.py 5 files changed, 32 insertions(+), 4 deletions(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved diff --git a/pySim-trace.py b/pySim-trace.py index 8a9fad7..8313c27 100755 --- a/pySim-trace.py +++ b/pySim-trace.py @@ -93,6 +93,11 @@ print("%02u %-16s %-35s %-8s %s %s" % (inst.lchan_nr, inst._name, inst.path_str, inst.col_id, inst.col_sw, inst.processed)) print("===============================") + def format_reset(self, apdu: CardReset): + """Output a single decoded CardReset.""" + print(apdu) + print("===============================") + def main(self): """Main loop of tracer: Iterates over all Apdu received from source.""" while True: @@ -101,6 +106,7 @@ if isinstance(apdu, CardReset): self.rs.reset() + self.format_reset(apdu) continue # ask ApduDecoder to look-up (INS,CLA) + instantiate an ApduCommand derived diff --git a/pySim/apdu/__init__.py b/pySim/apdu/__init__.py index cc0f701..b884e23 100644 --- a/pySim/apdu/__init__.py +++ b/pySim/apdu/__init__.py @@ -448,4 +448,11 @@ class CardReset: - pass + def __init__(self, atr: bytes): + self.atr = atr + + def __str__(self): + if (self.atr): + return '%s(%s)' % (type(self).__name__, b2h(self.atr)) + else: + return '%s' % (type(self).__name__) diff --git a/pySim/apdu_source/gsmtap.py b/pySim/apdu_source/gsmtap.py index fe450e2..aaf97ad 100644 --- a/pySim/apdu_source/gsmtap.py +++ b/pySim/apdu_source/gsmtap.py @@ -49,7 +49,7 @@ return ApduCommands.parse_cmd_bytes(gsmtap_msg['body']) elif sub_type == 'atr': # card has been reset - return CardReset() + return CardReset(gsmtap_msg['body']) elif sub_type in ['pps_req', 'pps_rsp']: # simply ignore for now pass diff --git a/pySim/apdu_source/pyshark_gsmtap.py b/pySim/apdu_source/pyshark_gsmtap.py index 8ea9ae7..aa7b624 100644 --- a/pySim/apdu_source/pyshark_gsmtap.py +++ b/pySim/apdu_source/pyshark_gsmtap.py @@ -69,7 +69,7 @@ return ApduCommands.parse_cmd_bytes(gsmtap_msg['body']) elif sub_type == 'atr': # card has been reset - return CardReset() + return CardReset(gsmtap_msg['body']) elif sub_type in ['pps_req', 'pps_rsp']: # simply ignore for now pass diff --git a/pySim/apdu_source/pyshark_rspro.py b/pySim/apdu_source/pyshark_rspro.py index 499e9ff..8c614ff 100644 --- a/pySim/apdu_source/pyshark_rspro.py +++ b/pySim/apdu_source/pyshark_rspro.py @@ -117,7 +117,8 @@ vccPresent, resetActive, clkActive = self.get_pstatus(slot_pstatus) if vccPresent and clkActive and not resetActive: logger.debug("RESET") - return CardReset() + #TODO: extract ATR from RSPRO message and use it here + return CardReset(None) else: print("Unhandled msg type %s: %s" % (msg_type, rspro_msg)) -- To view, visit https://gerrit.osmocom.org/c/pysim/+/33960 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: pysim Gerrit-Branch: master Gerrit-Change-Id: Ia550a8bd2f45d2ad622cb2ac2a2905397db76bce Gerrit-Change-Number: 33960 Gerrit-PatchSet: 2 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-MessageType: merged

1 0

[S] Change in pysim[master]: pySim-trace: catch StopIteration exception on trace file end
by laforge 28 Jul '23

28 Jul '23

laforge has submitted this change. ( https://gerrit.osmocom.org/c/pysim/+/33962 ) Change subject: pySim-trace: catch StopIteration exception on trace file end ...................................................................... pySim-trace: catch StopIteration exception on trace file end When the trace file end is reaced, pyShark raises a StopIteration exception. Let's catch this exception and exit gracefully. Related: OS#6094 Change-Id: I6ab5689b909333531d08bf46e5dfea59b161a79e --- M pySim-trace.py 1 file changed, 20 insertions(+), 1 deletion(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved diff --git a/pySim-trace.py b/pySim-trace.py index 8313c27..325fb8c 100755 --- a/pySim-trace.py +++ b/pySim-trace.py @@ -100,9 +100,15 @@ def main(self): """Main loop of tracer: Iterates over all Apdu received from source.""" + apdu_counter = 0 while True: # obtain the next APDU from the source (blocking read) - apdu = self.source.read() + try: + apdu = self.source.read() + apdu_counter = apdu_counter + 1 + except StopIteration: + print("%i APDUs parsed, stop iteration." % apdu_counter) + return 0 if isinstance(apdu, CardReset): self.rs.reset() -- To view, visit https://gerrit.osmocom.org/c/pysim/+/33962 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: pysim Gerrit-Branch: master Gerrit-Change-Id: I6ab5689b909333531d08bf46e5dfea59b161a79e Gerrit-Change-Number: 33962 Gerrit-PatchSet: 2 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-MessageType: merged

1 0

[S] Change in pysim[master]: utils: tolerate uninitialized fields in dec_addr_tlv
by laforge 28 Jul '23

28 Jul '23

Attention is currently required from: fixeria, dexter. laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/pysim/+/33954 ) Change subject: utils: tolerate uninitialized fields in dec_addr_tlv ...................................................................... Patch Set 2: Code-Review+2 (1 comment) File pySim/utils.py: https://gerrit.osmocom.org/c/pysim/+/33954/comment/83a0474c_15e9717d PS1, Line 775: if tlv[2] == [0xff] * len(tlv[2]): > Self explanatory is in the eye of the beholder. I really love your comment, dexter :) -- To view, visit https://gerrit.osmocom.org/c/pysim/+/33954 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: pysim Gerrit-Branch: master Gerrit-Change-Id: Idc0a92ea88756266381c8da2ad62de061a8ea7a1 Gerrit-Change-Number: 33954 Gerrit-PatchSet: 2 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Attention: dexter <pmaier(a)sysmocom.de> Gerrit-Comment-Date: Fri, 28 Jul 2023 09:07:33 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Comment-In-Reply-To: fixeria <vyanitskiy(a)sysmocom.de> Comment-In-Reply-To: dexter <pmaier(a)sysmocom.de> Gerrit-MessageType: comment

1 0

[S] Change in pysim[master]: tests: add test script for pySim-trace
by laforge 28 Jul '23

28 Jul '23

Attention is currently required from: dexter. laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/pysim/+/33963 ) Change subject: tests: add test script for pySim-trace ...................................................................... Patch Set 4: Code-Review+1 (1 comment) Patchset: PS4: this is good for a very basic test, but *not raising an exception* is of course a rather vague "pass criteria". I guess we have to think about what can be done in terms of further result verification (in follow-up patches). A low-hanging fruit would be to match the number of APDUs received. One *could* match the console output (similar to our autotest unit-tests in the osmo-{bts,bsc,msc,...} software. However, as we also know from there, every minor whitespace or cosmetic change will then require updating of the test results. Given that it should be simple to implement (famous last words), I think it might be worth a shot to have an expected stdout/stderr text file against we compare the pySim-trace output. -- To view, visit https://gerrit.osmocom.org/c/pysim/+/33963 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: pysim Gerrit-Branch: master Gerrit-Change-Id: Icfabfa7c59968021eef0399991bd05b92467d8d2 Gerrit-Change-Number: 33963 Gerrit-PatchSet: 4 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Attention: dexter <pmaier(a)sysmocom.de> Gerrit-Comment-Date: Fri, 28 Jul 2023 09:06:19 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

1 0

[S] Change in pysim[master]: pySim-trace: catch StopIteration exception on trace file end
by laforge 28 Jul '23

28 Jul '23

Attention is currently required from: fixeria, dexter. laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/pysim/+/33962 ) Change subject: pySim-trace: catch StopIteration exception on trace file end ...................................................................... Patch Set 2: Code-Review+2 -- To view, visit https://gerrit.osmocom.org/c/pysim/+/33962 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: pysim Gerrit-Branch: master Gerrit-Change-Id: I6ab5689b909333531d08bf46e5dfea59b161a79e Gerrit-Change-Number: 33962 Gerrit-PatchSet: 2 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Attention: dexter <pmaier(a)sysmocom.de> Gerrit-Comment-Date: Fri, 28 Jul 2023 09:02:54 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

1 0

[S] Change in pysim[master]: pySim-trace: mark card reset in the trace
by laforge 28 Jul '23

28 Jul '23

Attention is currently required from: dexter. laforge has posted comments on this change. ( https://gerrit.osmocom.org/c/pysim/+/33960 ) Change subject: pySim-trace: mark card reset in the trace ...................................................................... Patch Set 2: Code-Review+2 -- To view, visit https://gerrit.osmocom.org/c/pysim/+/33960 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: pysim Gerrit-Branch: master Gerrit-Change-Id: Ia550a8bd2f45d2ad622cb2ac2a2905397db76bce Gerrit-Change-Number: 33960 Gerrit-PatchSet: 2 Gerrit-Owner: dexter <pmaier(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Attention: dexter <pmaier(a)sysmocom.de> Gerrit-Comment-Date: Fri, 28 Jul 2023 09:02:32 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

1 0