<p>osmith has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.osmocom.org/10932">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">New script to verify PKG_CHECK_MODULES<br><br>Work in progress, see the Redmine issue.<br><br>Change-Id: I8f495dbe030775f66ac125e60ded95c5d7660b65<br>Relates: OS#2642<br>---<br>A scripts/dependency-check/dependency-check.py<br>A scripts/dependency-check/jenkins.sh<br>2 files changed, 199 insertions(+), 0 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/32/10932/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/scripts/dependency-check/dependency-check.py b/scripts/dependency-check/dependency-check.py</span><br><span>new file mode 100755</span><br><span>index 0000000..6e7656d</span><br><span>--- /dev/null</span><br><span>+++ b/scripts/dependency-check/dependency-check.py</span><br><span>@@ -0,0 +1,196 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/usr/bin/env python3</span><br><span style="color: hsl(120, 100%, 40%);">+from collections import OrderedDict</span><br><span style="color: hsl(120, 100%, 40%);">+import sys</span><br><span style="color: hsl(120, 100%, 40%);">+import os</span><br><span style="color: hsl(120, 100%, 40%);">+import subprocess</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def parse_error(line_num, message):</span><br><span style="color: hsl(120, 100%, 40%);">+ print("ERROR: configure.ac line " + str(line_num) + ": " + message)</span><br><span style="color: hsl(120, 100%, 40%);">+ sys.exit(1)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def parse_repository(gitdir, program, library, line_num):</span><br><span style="color: hsl(120, 100%, 40%);">+ repos = {"libosmocore": ["libosmocodec",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmocoding",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmoctrl",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmogb",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmogsm",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmosim",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmovty"],</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmo-abis": ["libosmoabis",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmotrau"],</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmo-sccp": ["libosmo-mtp",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmo-sigtran",</span><br><span style="color: hsl(120, 100%, 40%);">+ "libosmo-xua"]}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for repo, libraries in repos.items():</span><br><span style="color: hsl(120, 100%, 40%);">+ if library in libraries:</span><br><span style="color: hsl(120, 100%, 40%);">+ print(" * " + library + " (part of " + repo + ")")</span><br><span style="color: hsl(120, 100%, 40%);">+ return repo</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ print(" * " + library)</span><br><span style="color: hsl(120, 100%, 40%);">+ return library</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def parse_condition(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ # Only look at PKG_CHECK_MODULES lines, like these:</span><br><span style="color: hsl(120, 100%, 40%);">+ # PKG_CHECK_MODULES(LIBOSMOCORE, libosmocore >= 0.10.0)</span><br><span style="color: hsl(120, 100%, 40%);">+ # PKG_CHECK_MODULES(LIBSYSTEMD, libsystemd)</span><br><span style="color: hsl(120, 100%, 40%);">+ if "PKG_CHECK_MODULES" not in line:</span><br><span style="color: hsl(120, 100%, 40%);">+ return</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Extract the condition</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = line.split(",")[1].split(")")[0].strip()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Only look at Osmocom libraries</span><br><span style="color: hsl(120, 100%, 40%);">+ if ret.startswith("libosmo"):</span><br><span style="color: hsl(120, 100%, 40%);">+ return ret</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def parse_library_version(line_num, condition):</span><br><span style="color: hsl(120, 100%, 40%);">+ split = list(filter(None, condition.split(" ")))</span><br><span style="color: hsl(120, 100%, 40%);">+ if len(split) != 3:</span><br><span style="color: hsl(120, 100%, 40%);">+ parse_error(line_num, "invalid condition format, expected something"</span><br><span style="color: hsl(120, 100%, 40%);">+ " like 'libosmocore >= 0.10.0' but got: '" +</span><br><span style="color: hsl(120, 100%, 40%);">+ condition + "'")</span><br><span style="color: hsl(120, 100%, 40%);">+ library = split[0]</span><br><span style="color: hsl(120, 100%, 40%);">+ operator = split[1]</span><br><span style="color: hsl(120, 100%, 40%);">+ version = split[2]</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Right operator</span><br><span style="color: hsl(120, 100%, 40%);">+ if operator == ">=":</span><br><span style="color: hsl(120, 100%, 40%);">+ return (library, version)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Wrong operator</span><br><span style="color: hsl(120, 100%, 40%);">+ parse_error(line_num, "invalid operator, expected '>=' but got: '" +</span><br><span style="color: hsl(120, 100%, 40%);">+ operator + "'")</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def parse_configure_ac(gitdir, program):</span><br><span style="color: hsl(120, 100%, 40%);">+ # Read configure.ac</span><br><span style="color: hsl(120, 100%, 40%);">+ path = gitdir + "/" + program + "/configure.ac"</span><br><span style="color: hsl(120, 100%, 40%);">+ with open(path) as handle:</span><br><span style="color: hsl(120, 100%, 40%);">+ lines = handle.readlines()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Parse the file into ret</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = {}</span><br><span style="color: hsl(120, 100%, 40%);">+ for i in range(0, len(lines)):</span><br><span style="color: hsl(120, 100%, 40%);">+ # Parse the line</span><br><span style="color: hsl(120, 100%, 40%);">+ condition = parse_condition(lines[i])</span><br><span style="color: hsl(120, 100%, 40%);">+ if not condition:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ (library, version) = parse_library_version(i, condition)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Add to ret (with duplicate check)</span><br><span style="color: hsl(120, 100%, 40%);">+ if library in ret and version is not ret[library]:</span><br><span style="color: hsl(120, 100%, 40%);">+ parse_error(i, "found multiple PKG_CHECK_MODULES statements for " +</span><br><span style="color: hsl(120, 100%, 40%);">+ library + ", and they have different versions!")</span><br><span style="color: hsl(120, 100%, 40%);">+ repository = parse_repository(gitdir, program, library, i)</span><br><span style="color: hsl(120, 100%, 40%);">+ ret[repository] = version</span><br><span style="color: hsl(120, 100%, 40%);">+ return ret</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def clone(gitdir, program, version):</span><br><span style="color: hsl(120, 100%, 40%);">+ if not os.path.exists(gitdir + "/" + program):</span><br><span style="color: hsl(120, 100%, 40%);">+ url = "git://git.osmocom.org/" + program</span><br><span style="color: hsl(120, 100%, 40%);">+ print("Cloning git repo: " + url)</span><br><span style="color: hsl(120, 100%, 40%);">+ subprocess.run(["git", "-C", gitdir, "clone", "-q", url],</span><br><span style="color: hsl(120, 100%, 40%);">+ check=True)</span><br><span style="color: hsl(120, 100%, 40%);">+ subprocess.run(["git", "-C", gitdir + "/" + program, "checkout", version,</span><br><span style="color: hsl(120, 100%, 40%);">+ "-q"], check=True)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def dependstack_generate(gitdir, initial):</span><br><span style="color: hsl(120, 100%, 40%);">+ """ Clone an Osomocom git repository, parse its configure.ac and</span><br><span style="color: hsl(120, 100%, 40%);">+ recurse.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ :returns: ...</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ dependstack = OrderedDict()</span><br><span style="color: hsl(120, 100%, 40%);">+ parsestack = OrderedDict({initial: "master"})</span><br><span style="color: hsl(120, 100%, 40%);">+ while len(parsestack):</span><br><span style="color: hsl(120, 100%, 40%);">+ # Pop program from parsestack</span><br><span style="color: hsl(120, 100%, 40%);">+ program, version = next(iter(parsestack.items()))</span><br><span style="color: hsl(120, 100%, 40%);">+ del parsestack[program]</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Add to dependstack (or skip when it's there already)</span><br><span style="color: hsl(120, 100%, 40%);">+ if program in dependstack:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Add the program's dependencies to the parsestack</span><br><span style="color: hsl(120, 100%, 40%);">+ print("Looking at " + program + ":" + version)</span><br><span style="color: hsl(120, 100%, 40%);">+ clone(gitdir, program, version)</span><br><span style="color: hsl(120, 100%, 40%);">+ depends = parse_configure_ac(gitdir, program)</span><br><span style="color: hsl(120, 100%, 40%);">+ parsestack.update(depends)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Add the program to the dependstack</span><br><span style="color: hsl(120, 100%, 40%);">+ dependstack[program] = {"version": version, "depends": depends}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return dependstack</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def dependstack_print(dependstack):</span><br><span style="color: hsl(120, 100%, 40%);">+ print("Dependency graph:")</span><br><span style="color: hsl(120, 100%, 40%);">+ for program, data in dependstack.items():</span><br><span style="color: hsl(120, 100%, 40%);">+ version = data["version"]</span><br><span style="color: hsl(120, 100%, 40%);">+ depends = data["depends"]</span><br><span style="color: hsl(120, 100%, 40%);">+ print(" * " + program + ":" + version + " depends: " + str(depends))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def buildstack_next(gitdir, dependstack, done):</span><br><span style="color: hsl(120, 100%, 40%);">+ for program, data in dependstack.items():</span><br><span style="color: hsl(120, 100%, 40%);">+ if program in done:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ depends_done = True</span><br><span style="color: hsl(120, 100%, 40%);">+ for depend in data["depends"]:</span><br><span style="color: hsl(120, 100%, 40%);">+ if depend not in done:</span><br><span style="color: hsl(120, 100%, 40%);">+ depends_done = False</span><br><span style="color: hsl(120, 100%, 40%);">+ break</span><br><span style="color: hsl(120, 100%, 40%);">+ if depends_done:</span><br><span style="color: hsl(120, 100%, 40%);">+ return program, data["version"]</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ print("ERROR: can't figure out how to build the rest!")</span><br><span style="color: hsl(120, 100%, 40%);">+ print("Build order so far: " + str(done))</span><br><span style="color: hsl(120, 100%, 40%);">+ sys.exit(1)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def buildstack_generate(gitdir, dependstack):</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = OrderedDict()</span><br><span style="color: hsl(120, 100%, 40%);">+ count = len(dependstack.keys())</span><br><span style="color: hsl(120, 100%, 40%);">+ while len(ret) != count:</span><br><span style="color: hsl(120, 100%, 40%);">+ program, version = buildstack_next(gitdir, dependstack, ret)</span><br><span style="color: hsl(120, 100%, 40%);">+ ret[program] = version</span><br><span style="color: hsl(120, 100%, 40%);">+ return ret</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def buildstack_print(buildstack):</span><br><span style="color: hsl(120, 100%, 40%);">+ print("Build order:")</span><br><span style="color: hsl(120, 100%, 40%);">+ for program, version in buildstack.items():</span><br><span style="color: hsl(120, 100%, 40%);">+ print(" * " + program + ":" + version)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def buildstack_build(gitdir, buildstack):</span><br><span style="color: hsl(120, 100%, 40%);">+ for program, version in buildstack.items():</span><br><span style="color: hsl(120, 100%, 40%);">+ print("Building " + program + ":" + version)</span><br><span style="color: hsl(120, 100%, 40%);">+ os.chdir(gitdir + "/" + program)</span><br><span style="color: hsl(120, 100%, 40%);">+ commands = [["autoreconf", "-fi"],</span><br><span style="color: hsl(120, 100%, 40%);">+ ["make"],</span><br><span style="color: hsl(120, 100%, 40%);">+ ["make", "check"], # needed?</span><br><span style="color: hsl(120, 100%, 40%);">+ ["sudo", "make", "install"],</span><br><span style="color: hsl(120, 100%, 40%);">+ ["sudo", "ldconfig"]]</span><br><span style="color: hsl(120, 100%, 40%);">+ for command in commands:</span><br><span style="color: hsl(120, 100%, 40%);">+ print(" + " + " ".join(command))</span><br><span style="color: hsl(120, 100%, 40%);">+ subprocess.run(command, check=True)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def main():</span><br><span style="color: hsl(120, 100%, 40%);">+ gitdir = "/home/user/code/"</span><br><span style="color: hsl(120, 100%, 40%);">+ dependstack = dependstack_generate(gitdir, "osmo-bts")</span><br><span style="color: hsl(120, 100%, 40%);">+ dependstack_print(dependstack)</span><br><span style="color: hsl(120, 100%, 40%);">+ buildstack = buildstack_generate(gitdir, dependstack)</span><br><span style="color: hsl(120, 100%, 40%);">+ buildstack_print(buildstack)</span><br><span style="color: hsl(120, 100%, 40%);">+ buildstack_build(gitdir, buildstack)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+main()</span><br><span>diff --git a/scripts/dependency-check/jenkins.sh b/scripts/dependency-check/jenkins.sh</span><br><span>new file mode 100755</span><br><span>index 0000000..57f06f3</span><br><span>--- /dev/null</span><br><span>+++ b/scripts/dependency-check/jenkins.sh</span><br><span>@@ -0,0 +1,3 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/bin/sh -ex</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+echo "TODO"</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/10932">change 10932</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/10932"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-ci </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>
<div style="display:none"> Gerrit-Change-Id: I8f495dbe030775f66ac125e60ded95c5d7660b65 </div>
<div style="display:none"> Gerrit-Change-Number: 10932 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: osmith <osmith@sysmocom.de> </div>