October 2022 - gerrit-log - lists.osmocom.org

Change in osmo-ci[master]: jobs/gerrit-verifications*: drop unsupported args

by Jenkins Builder

Jenkins Builder has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/29688 ) Change subject: jobs/gerrit-verifications*: drop unsupported args ...................................................................... Patch Set 1: Verified+1 1 passed: [lint] https://jenkins.osmocom.org/jenkins/job/gerrit-lint/16/consoleFull Build Successful -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29688 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I6a0057738955bc2507cb5d8c4d48edf652c3529f Gerrit-Change-Number: 29688 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Comment-Date: Tue, 11 Oct 2022 11:35:27 +0000 Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: jobs/gerrit-verifications: add approvals comment

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29687 ) Change subject: jobs/gerrit-verifications: add approvals comment ...................................................................... jobs/gerrit-verifications: add approvals comment Change-Id: I4dfc0f84ffe584523bbff4c41295968a07ba097d --- M jobs/gerrit-verifications.yml 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/87/29687/1 diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 0e64ff3..d9a8ccf 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -364,6 +364,11 @@ name: PIPELINE_BINPKGS description: Enable the binpkgs job (build deb/rpm packages) default: '{obj:pipeline_binpkgs}' + # NOTE: jenkins pipelines don't run unless the dsl-script was approved. + # Sadly this doesn't happen automatically when updating the job with + # Jenkins Job Builder. So if you change the pipeline script, update the job + # as usually with JJB and then approve the scripts here: + # https://jenkins.osmocom.org/jenkins/scriptApproval/ dsl: | pipeline {{ agent {{ label "osmocom-gerrit" }} -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29687 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I4dfc0f84ffe584523bbff4c41295968a07ba097d Gerrit-Change-Number: 29687 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: jobs/gerrit-verifications*: drop unsupported args

by osmith

osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/29688 ) Change subject: jobs/gerrit-verifications*: drop unsupported args ...................................................................... jobs/gerrit-verifications*: drop unsupported args Fix warnings when updating the job with Jenkins Job Builder: WARNING:jenkins_jobs.modules.triggers:Gerrit Trigger property 'no-name-and-email' is deprecated WARNING:jenkins_jobs.modules.triggers:Gerrit Trigger property 'trigger-for-unreviewed-patches' is not supported in this plugin version Change-Id: I6a0057738955bc2507cb5d8c4d48edf652c3529f --- M jobs/gerrit-verifications-dahdi.yml M jobs/gerrit-verifications.yml 2 files changed, 0 insertions(+), 4 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/88/29688/1 diff --git a/jobs/gerrit-verifications-dahdi.yml b/jobs/gerrit-verifications-dahdi.yml index 51bfb04..a117ff3 100644 --- a/jobs/gerrit-verifications-dahdi.yml +++ b/jobs/gerrit-verifications-dahdi.yml @@ -76,8 +76,6 @@ notbuilt: false silent: false escape-quotes: false - no-name-and-email: false - trigger-for-unreviewed-patches: true server-name: gerrit.osmocom.org builders: diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index d9a8ccf..60ec639 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -529,8 +529,6 @@ notbuilt: false silent: true # comment + vote is done in gerrit-pipeline-result.yml escape-quotes: false - no-name-and-email: false - trigger-for-unreviewed-patches: true server-name: gerrit.osmocom.org - job-template: -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29688 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I6a0057738955bc2507cb5d8c4d48edf652c3529f Gerrit-Change-Number: 29688 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: support Osmocom projects without prefix

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/29672 ) Change subject: obs: support Osmocom projects without prefix ...................................................................... obs: support Osmocom projects without prefix Allow running build_srcpkg.py with "osmo_dia2gsup" instead of "erlang/osmo_dia2gsup", because that's how it gets passed along from jobs/gerrit-verifications.yml for the binary package build verifications. Related: OS#2385 Change-Id: I09304b219e7002495bd3fcce61bc68c34d5ffcd8 --- M scripts/obs/build_srcpkg.py M scripts/obs/lib/__init__.py M scripts/obs/update_obs_project.py 3 files changed, 12 insertions(+), 7 deletions(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve msuraev: Looks good to me, but someone else must approve diff --git a/scripts/obs/build_srcpkg.py b/scripts/obs/build_srcpkg.py index 4c848b8..ad21927 100755 --- a/scripts/obs/build_srcpkg.py +++ b/scripts/obs/build_srcpkg.py @@ -37,7 +37,7 @@ lib.check_required_programs() if args.package: - lib.check_package(args.package) + args.package = lib.set_proper_package_name(args.package) lib.remove_temp() if args.meta: diff --git a/scripts/obs/lib/__init__.py b/scripts/obs/lib/__init__.py index 5979623..479c1c1 100644 --- a/scripts/obs/lib/__init__.py +++ b/scripts/obs/lib/__init__.py @@ -78,11 +78,16 @@ exit(1) -def check_package(package): +def set_proper_package_name(package): if package in lib.config.projects_osmocom: - return + return package if package in lib.config.projects_other: - return + return package + + # Add prefix to Osmocom package if missing + for package_cfg in lib.config.projects_osmocom: + if os.path.basename(package_cfg) == package: + return package_cfg print(f"ERROR: unknown package: {package}") print("See packages_osmocom and packages_other in obs/lib/config.py") diff --git a/scripts/obs/update_obs_project.py b/scripts/obs/update_obs_project.py index 43748e9..86cf40a 100755 --- a/scripts/obs/update_obs_project.py +++ b/scripts/obs/update_obs_project.py @@ -20,13 +20,13 @@ def parse_packages(packages_arg): + ret = [] if packages_arg: for package in packages_arg: - lib.check_package(package) - return packages_arg + ret += [lib.set_proper_package_name(package)] + return ret # Default to all - ret = [] ret += lib.config.projects_osmocom ret += lib.config.projects_other return ret -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29672 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I09304b219e7002495bd3fcce61bc68c34d5ffcd8 Gerrit-Change-Number: 29672 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: msuraev <msuraev(a)sysmocom.de> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: config: update note about testing packages

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/29606 ) Change subject: obs: config: update note about testing packages ...................................................................... obs: config: update note about testing packages Related: OS#2385 Change-Id: If6d772052ca64ef93536bf8729a72271f67fbecb --- M scripts/obs/lib/config.py 1 file changed, 2 insertions(+), 3 deletions(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved pespin: Looks good to me, approved diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py index d9f6bfe..4a4bf3c 100644 --- a/scripts/obs/lib/config.py +++ b/scripts/obs/lib/config.py @@ -39,9 +39,8 @@ # Osmocom projects: generated source packages will depend on a meta package, # such as osmocom-nightly, osmocom-latest or osmocom-2022q1. This meta package # prevents that packages from different feeds are mixed by accident. -# NOTE: Before adding new projects, make sure the rpm and deb build in OBS! -# Test it in your own namespace (home:youruser), see README for -# instructions and/or ask osmith for help. +# NOTE: Before adding new projects, add them to jobs/gerrit-verifications.yml +# and ensure the rpm and deb packages build successfully in jenkins. projects_osmocom = [ "erlang/osmo_dia2gsup", "libasn1c", -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29606 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: If6d772052ca64ef93536bf8729a72271f67fbecb Gerrit-Change-Number: 29606 Gerrit-PatchSet: 5 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: jobs/gerrit-verifications: write summary comment

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/29671 ) Change subject: jobs/gerrit-verifications: write summary comment ...................................................................... jobs/gerrit-verifications: write summary comment Instead of having multiple mails from jenkins per submitted gerrit patch, only have one at the end of the pipeline that writes a summary with all relevant links and votes +1/-1 for verified. This also resolves the problem we had for a long time, that the link for the build job didn't directly point to the relevant console output. Instead it pointed to a matrix job that didn't have a direct link to the relevant job. The summary looks like this: 1 failed: [build] https://jenkins.osmocom.org/jenkins/job/gerrit-osmo-bsc-nat-build/a1=defaul… 3 passed: [rpm] https://jenkins.osmocom.org/jenkins/job/gerrit-binpkgs-rpm/5/consoleFull [deb] https://jenkins.osmocom.org/jenkins/job/gerrit-binpkgs-deb/5/consoleFull [lint] https://jenkins.osmocom.org/jenkins/job/gerrit-lint/11/consoleFull Build Failed Related: OS#2385 Change-Id: Idcab969e1b5ca4e0f1383bee8f36f2d1aac4f624 --- A jobs/gerrit-pipeline-result.yml M jobs/gerrit-verifications.yml A scripts/jenkins-gerrit/pipeline_summary.py A scripts/jenkins-gerrit/pipeline_summary_send.sh 4 files changed, 280 insertions(+), 1 deletion(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve diff --git a/jobs/gerrit-pipeline-result.yml b/jobs/gerrit-pipeline-result.yml new file mode 100644 index 0000000..ac54cee --- /dev/null +++ b/jobs/gerrit-pipeline-result.yml @@ -0,0 +1,68 @@ +# This job runs at the end of the pipeline in gerrit-verififactions.yml, to +# post a list of failed/successful job links to gerrit and to vote +V/-V. + +- project: + name: gerrit-pipeline-result + jobs: + - 'gerrit-pipeline-result' + +- job: + name: 'gerrit-pipeline-result' + project-type: freestyle + node: osmocom-gerrit-debian10 || osmocom-gerrit-debian11 + retry-count: 3 # scm checkout + properties: + - build-discarder: + days-to-keep: 30 + num-to-keep: 120 + artifact-days-to-keep: -1 + artifact-num-to-keep: -1 + description: | + Result job of CI for patches sent to <a href="https://gerrit.osmocom.org">gerrit</a>. + </br></br> + Related issue: <a href="https://osmocom.org/issues/2385">OS#2385</a> + + parameters: + - string: + name: BRANCH_CI + description: | + osmo-ci.git branch + default: 'master' + - string: + name: GERRIT_BRANCH + description: set by gerrit verification pipeline job + - string: + name: GERRIT_HOST + description: set by gerrit verification pipeline job + - string: + name: GERRIT_PATCHSET_REVISION + description: set by gerrit verification pipeline job + - string: + name: GERRIT_PORT + description: set by gerrit verification pipeline job + - string: + name: GERRIT_REFSPEC + description: set by gerrit verification pipeline job + - string: + name: PIPELINE_BUILD_URL + description: set by gerrit verification pipeline job + + scm: + - git: + url: 'https://gerrit.osmocom.org/osmo-ci' + credentials-id: d5eda5e9-b59d-44ba-88d2-43473cb6e42d + branches: + - '$BRANCH_CI' + wipe-workspace: true + + builders: + - shell: 'cd scripts/jenkins-gerrit && ./pipeline_summary_send.sh' + + wrappers: + - ansicolor: + colormap: xterm + - ssh-agent-credentials: + users: + - d5eda5e9-b59d-44ba-88d2-43473cb6e42d + +# vim: expandtab tabstop=2 shiftwidth=2 diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 5235525..0e64ff3 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -458,6 +458,19 @@ echo "PIPELINE_LINT_PASSED=${{env.PIPELINE_LINT_PASSED}}" echo "PIPELINE_DEB_PASSED=${{env.PIPELINE_DEB_PASSED}}" echo "PIPELINE_RPM_PASSED=${{env.PIPELINE_RPM_PASSED}}" + + // Run the result job to get successful/failed links and add a + // comment + vote to gerrit + script {{ + build job: 'gerrit-pipeline-result', parameters: [ + string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), + string(name: "GERRIT_HOST", value: "${{env.GERRIT_HOST}}"), + string(name: "GERRIT_PATCHSET_REVISION", value: "${{env.GERRIT_PATCHSET_REVISION}}"), + string(name: "GERRIT_PORT", value: "${{env.GERRIT_PORT}}"), + string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), + string(name: "PIPELINE_BUILD_URL", value: "${{env.BUILD_URL}}"), + ] + }} }} }} // The end result is success if all started jobs were successful, @@ -509,7 +522,7 @@ failed: false unstable: false notbuilt: false - silent: false + silent: true # comment + vote is done in gerrit-pipeline-result.yml escape-quotes: false no-name-and-email: false trigger-for-unreviewed-patches: true diff --git a/scripts/jenkins-gerrit/pipeline_summary.py b/scripts/jenkins-gerrit/pipeline_summary.py new file mode 100755 index 0000000..c12ad64 --- /dev/null +++ b/scripts/jenkins-gerrit/pipeline_summary.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright 2022 sysmocom - s.f.m.c. GmbH <info(a)sysmocom.de> +import argparse +import io +import json +import re +import urllib.request + +jenkins_url = "https://jenkins.osmocom.org" +re_start_build = re.compile("Starting building: gerrit-[a-zA-Z-_]* #[0-9]*") +re_result = re.compile("^PIPELINE_[A-Z]*_PASSED=[01]$") + +def parse_args(): + parser = argparse.ArgumentParser( + description="Get a summary of failed / successful builds from the CI" + " pipeline we run for patches submitted to gerrit.") + parser.add_argument("build_url", + help="$BUILD_URL of the pipeline job, e.g." + " https://jenkins.osmocom.org/jenkins/job/gerrit-osmo-bsc-nat/17/") + parser.add_argument("-o", "--output", help="output json file") + return parser.parse_args() + + +def stage_from_job_name(job_name): + if job_name == "gerrit-pipeline-result": + # The job that runs this script. Don't include it in the summary. + return None + if job_name == "gerrit-lint": + return "lint" + if job_name == "gerrit-binpkgs-deb": + return "deb" + if job_name == "gerrit-binpkgs-rpm": + return "rpm" + if job_name.endswith("-build"): + return "build" + assert False, f"couldn't figure out stage from job_name: {job_name}" + + +def parse_pipeline(build_url): + """ Parse started jobs and result from the pipeline log. + :returns: a dict that looks like: + {"build": {"name": "gerrit-osmo-bsc-nat-build", id=7, + "passed": True, "url": "https://..."}, + "lint": {...}, + "deb": {...}, + "rpm: {...}} """ + global re_start_build + global re_result + global jenkins_url + ret = {} + + url = f"{build_url}/consoleText" + with urllib.request.urlopen(url) as response: + for line in io.TextIOWrapper(response, encoding='utf-8'): + # Parse start build lines + for match in re_start_build.findall(line): + job_name = match.split(" ")[2] + job_id = int(match.split(" ")[3].replace("#", "")) + job_url = f"{jenkins_url}/jenkins/job/{job_name}/{job_id}" + stage = stage_from_job_name(job_name) + if stage: + ret[stage] = {"url": job_url, "name": job_name, "id": job_id} + + # Parse result lines + if re_result.match(line): + stage = line.split("_")[1].lower() + passed = line.split("=")[1].rstrip() == "1" + ret[stage]["passed"] = passed + + return ret + + +def parse_build_matrix(job): + """ Parse started jobs and result from the matrix of the build job. Usually + it is only one job, but for some projects we build for multiple arches + (x86_64, arm) or build multiple times with different configure flags. + :param job: "build" dict from parse_pipeline() + :returns: a list of jobs in the matrix, looks like: + [{"passed": True, "url": "https://..."}, ...] + """ + global jenkins_url + + ret = [] + url = f"{job['url']}/consoleFull" + with urllib.request.urlopen(url) as response: + for line in io.TextIOWrapper(response, encoding='utf-8'): + if " completed with result " in line: + url = line.split("<a href='", 1)[1].split("'", 1)[0] + url = f"{jenkins_url}{url}{job['id']}" + result = line.split(" completed with result ")[1].rstrip() + passed = result == "SUCCESS" + ret += [{"passed": passed, "url": url}] + return ret + + +def jobs_for_summary(pipeline, build_matrix): + """ Sort the jobs from pipeline and build matrix into passed/failed lists. + :returns: a dict that looks like: + {"passed": [{"stage": "build", "url": "https://..."}, ...], + "failed": [...]} """ + ret = {"passed": [], "failed": []} + + # Build errors are most interesting, display them first + for job in build_matrix: + category = "passed" if job["passed"] else "failed" + ret[category] += [{"stage": "build", "url": job["url"]}] + + # Hide the build matrix job (we show the jobs started by it instead), as + # long as there is at least one failed started job when the matrix failed + matrix_failed = "build" in pipeline and not pipeline["build"]["passed"] + show_build_matrix_job = matrix_failed and not ret["failed"] + + # Add jobs from the pipeline + for stage, job in pipeline.items(): + if stage == "build" and not show_build_matrix_job: + continue + category = "passed" if job["passed"] else "failed" + ret[category] += [{"stage": stage, "url": job["url"]}] + + return ret + + +def get_jobs_list_str(jobs): + ret = "" + for job in jobs: + ret += f" [{job['stage']}] {job['url']}/consoleFull\n" + return ret + + +def get_pipeline_summary(build_url): + """ Generate a summary of failed and successful builds for gerrit. + :returns: a dict that is expected by gerrit's set-review api, e.g. + {"tag": "jenkins", + "message": "...", + "labels": {"Code-Review": -1}} """ + summary = "" + pipeline = parse_pipeline(build_url) + + build_matrix = [] + if "build" in pipeline: + build_matrix = parse_build_matrix(pipeline["build"]) + + jobs = jobs_for_summary(pipeline, build_matrix) + + if jobs["failed"]: + summary += f"{len(jobs['failed'])} failed:\n" + summary += get_jobs_list_str(jobs["failed"]) + summary += "\n" + + summary += f"{len(jobs['passed'])} passed:\n" + summary += get_jobs_list_str(jobs["passed"]) + + if "lint" in pipeline and not pipeline["lint"]["passed"]: + summary += "\n" + summary += "Please fix the linting errors. More information:\n" + summary += "https://osmocom.org/projects/cellular-infrastructure/wiki/Linting\n" + + summary += "\n" + if jobs["failed"]: + summary += "Build Failed\n" + vote = -1 + else: + summary += "Build Successful\n" + vote = 1 + + # Reference: + # https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#… + return {"tag": "jenkins", + "message": summary, + "labels": {"Verified": vote}} + + +def main(): + args = parse_args() + summary = get_pipeline_summary(args.build_url) + + print(summary["message"]) + + if args.output: + with open(args.output, "w") as handle: + json.dump(summary, handle, indent=4) + +if __name__ == "__main__": + main() diff --git a/scripts/jenkins-gerrit/pipeline_summary_send.sh b/scripts/jenkins-gerrit/pipeline_summary_send.sh new file mode 100755 index 0000000..d52ad9f --- /dev/null +++ b/scripts/jenkins-gerrit/pipeline_summary_send.sh @@ -0,0 +1,13 @@ +#!/bin/sh -ex + +./pipeline_summary.py "$PIPELINE_BUILD_URL" -o gerrit_report.json + +ssh \ + -p "$GERRIT_PORT" \ + -l jenkins \ + "$GERRIT_HOST" \ + gerrit \ + review \ + "$GERRIT_PATCHSET_REVISION" \ + --json \ + < gerrit_report.json -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29671 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Idcab969e1b5ca4e0f1383bee8f36f2d1aac4f624 Gerrit-Change-Number: 29671 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-CC: msuraev <msuraev(a)sysmocom.de> Gerrit-MessageType: merged

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs/README: add overview with build_binpkg.py

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/29605 ) Change subject: obs/README: add overview with build_binpkg.py ...................................................................... obs/README: add overview with build_binpkg.py Related: OS#2385 Change-Id: I1b741ad325e6541e6d59dfd50de66bee805e56fe --- M scripts/obs/README 1 file changed, 19 insertions(+), 0 deletions(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved pespin: Looks good to me, but someone else must approve diff --git a/scripts/obs/README b/scripts/obs/README index cd0955d..f761f1e 100644 --- a/scripts/obs/README +++ b/scripts/obs/README @@ -1,3 +1,22 @@ +Overview +======== +This directory contains the following scripts, related to building Osmocom +projects in OBS (Open Build Service) and building binary packages. Here is an +overview of the scripts, run them with -h to get a more detailed description. + +* build_srcpkg.py: build one source package for an Osmocom project +* update_obs_project.py: generate source packages and upload them to OBS +* build_binpkg.py: build rpm/deb packages for one Osmocom project + +Docker +------ +The above scripts have a -d parameter, that allows running them inside docker. +This is how we run it in jenkins to avoid installing dependencies on the host. + +Note that when building binary packages, libosmocore socket tests and possibly +others will fail unless docker has ipv6 configured: +https://osmocom.org/issues/4700#note-3 + Usage Example: Submitting source packages to Osmocom's OBS build server ======================================================================= 3 is the latest approved patch-set. No files were changed between the latest approved patch-set and the submitted one. -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29605 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I1b741ad325e6541e6d59dfd50de66bee805e56fe Gerrit-Change-Number: 29605 Gerrit-PatchSet: 5 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: obs: build_binpkg: use almalinux:8 naming

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/29686 ) Change subject: obs: build_binpkg: use almalinux:8 naming ...................................................................... obs: build_binpkg: use almalinux:8 naming Instead of using "--docker centos:8" as argument and translating it later on in the code to use "almalinux:8" as base distribution, use "--docker almalinux:8" as argument. It was brought up in code review that this makes it less confusing. Related: OS#2385 Change-Id: Id8298e8bafe065010f7bc00f1ff261aa6431ed4c --- M scripts/obs/data/build_binpkg.Dockerfile M scripts/obs/lib/config.py M scripts/obs/lib/docker.py 3 files changed, 4 insertions(+), 7 deletions(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, approved diff --git a/scripts/obs/data/build_binpkg.Dockerfile b/scripts/obs/data/build_binpkg.Dockerfile index de53478..201c7e5 100644 --- a/scripts/obs/data/build_binpkg.Dockerfile +++ b/scripts/obs/data/build_binpkg.Dockerfile @@ -14,6 +14,7 @@ # https://build.opensuse.org/projects/CentOS:CentOS-8/prjconf # For debian, make sure we don't have man pages as otherwise it takes some time # to regenerate the manuals database when installing build dependencies. +# SYS#5818: using almalinux:8 instead of centos:8 RUN case "$DISTRO" in \ debian*) \ echo "path-exclude=/usr/share/man/*" \ @@ -29,7 +30,7 @@ && \ apt-get clean \ ;; \ - centos*) \ + almalinux*) \ dnf -y install \ autoconf \ automake \ @@ -58,7 +59,7 @@ echo "deb https://downloads.osmocom.org/packages/osmocom:/master/Debian_11/ ./" \ > /etc/apt/sources.list.d/osmocom-master.list \ ;; \ - centos:8) \ + almalinux:8) \ { echo "[network_osmocom_master]"; \ echo "name=Nightly packages of the Osmocom project (CentOS_8)"; \ echo "type=rpm-md"; \ diff --git a/scripts/obs/lib/config.py b/scripts/obs/lib/config.py index 393251b..d9f6bfe 100644 --- a/scripts/obs/lib/config.py +++ b/scripts/obs/lib/config.py @@ -113,5 +113,5 @@ docker_distro_default = "debian:11" docker_distro_other = [ - "centos:8", + "almalinux:8", # instead of centos:8 (SYS#5818) ] diff --git a/scripts/obs/lib/docker.py b/scripts/obs/lib/docker.py index 136aeeb..87611e3 100644 --- a/scripts/obs/lib/docker.py +++ b/scripts/obs/lib/docker.py @@ -19,10 +19,6 @@ if image_type.endswith("_manuals"): return get_image_name(distro, image_type.replace("_manuals", "")) - # CentOS 8 is EOL (SYS#5818) - if distro == "centos:8": - return "almalinux:8" - return distro -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29686 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Id8298e8bafe065010f7bc00f1ff261aa6431ed4c Gerrit-Change-Number: 29686 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: msuraev <msuraev(a)sysmocom.de> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-MessageType: merged

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: jobs/gerrit-binpkgs: new job for deb, rpm packages

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/29604 ) Change subject: jobs/gerrit-binpkgs: new job for deb, rpm packages ...................................................................... jobs/gerrit-binpkgs: new job for deb, rpm packages Build deb and rpm packages for each patch submitted to gerrit for the projects in the list. Example: https://gerrit.osmocom.org/c/osmo-bsc-nat/+/29492 Related: OS#2385 Change-Id: I7ca8869c2e9f2e7c74a360933be12b5c2b47c2fc --- A jobs/gerrit-binpkgs.yml M jobs/gerrit-verifications.yml 2 files changed, 205 insertions(+), 9 deletions(-) Approvals: Jenkins Builder: Verified laforge: Looks good to me, but someone else must approve osmith: Looks good to me, approved diff --git a/jobs/gerrit-binpkgs.yml b/jobs/gerrit-binpkgs.yml new file mode 100644 index 0000000..5c72a41 --- /dev/null +++ b/jobs/gerrit-binpkgs.yml @@ -0,0 +1,118 @@ +# This file holds all gerrit verifications for building deb and rpm packages. +# https://jenkins.osmocom.org/jenkins/view/Jenkins-Gerrit/. +# One can simply add a gerrit job by adding project's repository to repos list. + +- project: + name: gerrit-binpkgs + jobs: + - 'gerrit-binpkgs-{type}' + type: + - 'deb' + - 'rpm' + +- job-template: + name: 'gerrit-binpkgs-{type}' + project-type: freestyle + node: osmocom-gerrit-debian10 || osmocom-gerrit-debian11 + concurrent: true + retry-count: 3 # scm checkout + properties: + - build-discarder: + days-to-keep: 30 + num-to-keep: 120 + artifact-days-to-keep: -1 + artifact-num-to-keep: -1 + description: | + {type} package job of CI for patches sent to + <a href="https://gerrit.osmocom.org">gerrit</a> + </br></br> + Related issue: <a href="https://osmocom.org/issues/2385">OS#2385</a> + + parameters: + - string: + name: BRANCH_CI + description: | + osmo-ci.git branch + default: 'origin/master' + - string: + name: GERRIT_REPO_URL + description: set by gerrit verification pipeline job + - string: + name: GERRIT_BRANCH + description: set by gerrit verification pipeline job + - string: + name: GERRIT_REFSPEC + description: set by gerrit verification pipeline job + - string: + name: PROJECT_NAME + description: project to test, e.g. 'libosmocore' (set by pipeline) + + scm: + - git: + basedir: 'code-from-gerrit' + url: '$GERRIT_REPO_URL' + credentials-id: d5eda5e9-b59d-44ba-88d2-43473cb6e42d + branches: + - $GERRIT_BRANCH + refspec: $GERRIT_REFSPEC + name: + choosing-strategy: gerrit + wipe-workspace: true + skip-tag: true + submodule: + recursive: false + - git: + basedir: 'osmo-ci' + url: 'https://gerrit.osmocom.org/osmo-ci' + credentials-id: d5eda5e9-b59d-44ba-88d2-43473cb6e42d + branches: + - '$BRANCH_CI' + wipe-workspace: true + + builders: + - shell: |- + # Skip rpm build if .spec.in file is missing + if [ "{type}" = "rpm" ] && [ -z "$(find code-from-gerrit -name '*.spec.in')" ]; then + set +x + echo + echo "No *.spec.in file found, skipping rpm build." + echo + exit 0 + fi + + # Get distro from type + case "{type}" in + deb) distro="debian:11" ;; + rpm) distro="almalinux:8" ;; + *) echo "unknown package format: '{type}'"; exit 1 ;; + esac + + # Move code from gerrit to build_srcpkg.py's git cache + cache_dir=osmo-ci/scripts/obs/_cache + mkdir -p $cache_dir + mv code-from-gerrit "$cache_dir/$PROJECT_NAME" + + # Set a known branch name + git -C "$cache_dir/$PROJECT_NAME" checkout -B "origin/gerrit" + + # Build source package + cd osmo-ci/scripts/obs/ + ./build_srcpkg.py \ + --docker \ + --feed master \ + --git-branch gerrit \ + --git-skip-fetch \ + "$PROJECT_NAME" + + # Build binary package + ./build_binpkg.py \ + --docker "$distro" \ + "$PROJECT_NAME" + wrappers: + - ansicolor: + colormap: xterm + - ssh-agent-credentials: + users: + - d5eda5e9-b59d-44ba-88d2-43473cb6e42d + +# vim: expandtab tabstop=2 shiftwidth=2 diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 135e04f..5235525 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -67,14 +67,18 @@ # Which jobs to run in the pipeline pipeline_build: true pipeline_lint: true + pipeline_binpkgs: true # in alphabetical order repos: - - asn1c + - asn1c: + pipeline_binpkgs: false + - cellmgr-ng - docker-playground: pipeline_build: false + pipeline_binpkgs: false - libasn1c - libgtpnl @@ -91,7 +95,10 @@ cmd: '{docker_run} {docker_img} {timeout_cmd} /build/contrib/jenkins_arch.sh "$arch"' - libsmpp34 - - libtelnet + + - libtelnet: + pipeline_binpkgs: false + - libusrp - openbsc: @@ -152,6 +159,7 @@ - osmo-e1-hardware: docker_img: 'registry.osmocom.org/$USER/fpga-build' cmd: 'docker pull {docker_img} && {docker_run} {docker_img} {timeout_cmd} /build/contrib/jenkins.sh' + pipeline_binpkgs: false - osmo-ggsn: a1_name: GTP @@ -173,6 +181,7 @@ repos_url: '{gerrit_url}/erlang/{repos}' gerrit_project: 'erlang/{repos}' cmd: '{docker_run} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' + pipeline_binpkgs: false - osmo-hlr - osmo-iuh @@ -242,6 +251,7 @@ repos_url: '{gerrit_url}/erlang/{repos}' gerrit_project: 'erlang/{repos}' cmd: '{docker_run} {docker_img_erlang} {timeout_cmd} /build/contrib/jenkins.sh' + pipeline_binpkgs: false - osmo-sip-connector @@ -262,22 +272,41 @@ -e WITH_MANUALS="$WITH_MANUALS" \ {docker_img} {timeout_cmd} /build/contrib/jenkins.sh - - osmocom-bb - - osmo-tetra + - osmocom-bb: + pipeline_binpkgs: false + + - osmo-tetra: + pipeline_binpkgs: false + - osmo-sysmon - osmo-remsim - simtrace2 + - osmo-opencm3-projects: docker_img: '$USER/debian-bullseye-jenkins' cmd: '{docker_run} {docker_img} {timeout_cmd} /build/contrib/jenkins.sh' - - osmo-asf4-dfu - - osmo-ccid-firmware + pipeline_binpkgs: false + + - osmo-asf4-dfu: + pipeline_binpkgs: false + + - osmo-ccid-firmware: + pipeline_binpkgs: false + - osmo-e1d - osmo-cbc - - osmo-e1-recorder - - gapk + + - osmo-e1-recorder: + pipeline_binpkgs: false + + - gapk: + pipeline_binpkgs: false + - osmo-uecups - - osmo-el2tpd + + - osmo-el2tpd: + pipeline_binpkgs: false + - osmo-smlc - osmo-gbproxy - osmo-hnodeb @@ -288,6 +317,7 @@ slave_axis: !!python/tuple [simtester] concurrent: false cmd: '{timeout_cmd} ./contrib/jenkins.sh' + pipeline_binpkgs: false - osmo-ttcn3-hacks: slave_axis: !!python/tuple [ttcn3] @@ -304,9 +334,11 @@ su build -c 'make clean'; su build -c 'make compile'; " + pipeline_binpkgs: false - dahdi-tools: pipeline_lint: false + pipeline_binpkgs: false jobs: - 'gerrit-{repos}' @@ -328,6 +360,10 @@ name: REPO_URL description: Clone URL, to be passed to jobs started in the pipeline default: '{obj:repos_url}' + - bool: + name: PIPELINE_BINPKGS + description: Enable the binpkgs job (build deb/rpm packages) + default: '{obj:pipeline_binpkgs}' dsl: | pipeline {{ agent {{ label "osmocom-gerrit" }} @@ -374,12 +410,54 @@ }} }} }} + stage("deb") {{ + when {{ + expression {{ params.PIPELINE_BINPKGS }} + }} + steps {{ + script {{ + // Run the binpkgs job for this repository and keep going on failure + catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ + env.PIPELINE_DEB_PASSED = "0" + build job: 'gerrit-binpkgs-deb', parameters: [ + string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), + string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), + string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), + string(name: "PROJECT_NAME", value: "{repos}"), + ] + env.PIPELINE_DEB_PASSED = "1" + }} + }} + }} + }} + stage("rpm") {{ + when {{ + expression {{ params.PIPELINE_BINPKGS }} + }} + steps {{ + script {{ + // Run the binpkgs job for this repository and keep going on failure + catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {{ + env.PIPELINE_RPM_PASSED = "0" + build job: 'gerrit-binpkgs-rpm', parameters: [ + string(name: "GERRIT_BRANCH", value: "${{env.GERRIT_BRANCH}}"), + string(name: "GERRIT_REFSPEC", value: "${{env.GERRIT_REFSPEC}}"), + string(name: "GERRIT_REPO_URL", value: "${{env.REPO_URL}}"), + string(name: "PROJECT_NAME", value: "{repos}"), + ] + env.PIPELINE_RPM_PASSED = "1" + }} + }} + }} + }} }} }} stage("Result") {{ steps {{ echo "PIPELINE_BUILD_PASSED=${{env.PIPELINE_BUILD_PASSED}}" echo "PIPELINE_LINT_PASSED=${{env.PIPELINE_LINT_PASSED}}" + echo "PIPELINE_DEB_PASSED=${{env.PIPELINE_DEB_PASSED}}" + echo "PIPELINE_RPM_PASSED=${{env.PIPELINE_RPM_PASSED}}" }} }} // The end result is success if all started jobs were successful, -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29604 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: I7ca8869c2e9f2e7c74a360933be12b5c2b47c2fc Gerrit-Change-Number: 29604 Gerrit-PatchSet: 5 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-CC: msuraev <msuraev(a)sysmocom.de> Gerrit-MessageType: merged

2 years, 8 months

1
0
0 0

Change in osmo-ci[master]: jobs/gerrit-verifications: add param to skip build

by osmith

osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/29669 ) Change subject: jobs/gerrit-verifications: add param to skip build ...................................................................... jobs/gerrit-verifications: add param to skip build Prepare to unify the list of projects in jobs/gerrit-verifications.yml and jobs/gerrit-lint.yml by allowing to skip the build for some repositories (e.g. docker-playground is linted but we don't run a contrib/jenkins.sh there). Change-Id: Ie6264d44120798c0ecbcaf9c63d9b95f74ae9e37 --- M jobs/gerrit-verifications.yml 1 file changed, 10 insertions(+), 0 deletions(-) Approvals: Jenkins Builder: Verified pespin: Looks good to me, but someone else must approve msuraev: Looks good to me, but someone else must approve laforge: Looks good to me, approved diff --git a/jobs/gerrit-verifications.yml b/jobs/gerrit-verifications.yml index 0a7ccac..26f7f64 100644 --- a/jobs/gerrit-verifications.yml +++ b/jobs/gerrit-verifications.yml @@ -64,6 +64,8 @@ gerrit_url: 'ssh://jenkins@gerrit.osmocom.org:29418' repos_url: '{gerrit_url}/{repos}' gerrit_project: '{repos}' + # Which jobs to run in the pipeline + pipeline_build: true # in alphabetical order repos: @@ -307,6 +309,11 @@ - job-template: name: 'gerrit-{repos}' project-type: pipeline + parameters: + - bool: + name: PIPELINE_BUILD + description: Enable the build job (runs contrib/jenkins.sh) + default: '{obj:pipeline_build}' dsl: | pipeline {{ agent {{ label "osmocom-gerrit" }} @@ -314,6 +321,9 @@ stage("Verification") {{ parallel {{ stage("Build") {{ + when {{ + expression {{ params.PIPELINE_BUILD }} + }} steps {{ script {{ // Run the build job for this repository and keep going on failure -- To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/29669 To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings Gerrit-Project: osmo-ci Gerrit-Branch: master Gerrit-Change-Id: Ie6264d44120798c0ecbcaf9c63d9b95f74ae9e37 Gerrit-Change-Number: 29669 Gerrit-PatchSet: 1 Gerrit-Owner: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: laforge <laforge(a)osmocom.org> Gerrit-Reviewer: msuraev <msuraev(a)sysmocom.de> Gerrit-Reviewer: osmith <osmith(a)sysmocom.de> Gerrit-Reviewer: pespin <pespin(a)sysmocom.de> Gerrit-MessageType: merged

2 years, 8 months

1
0
0 0

2025

2024

2023

2022

gerrit-log October 2022