osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40842?usp=email )
Change subject: testenv: pass targets to osmo-dev's gen_makefile
......................................................................
testenv: pass targets to osmo-dev's gen_makefile
Pass the targets we are about to build to osmo-dev's gen_makefile
script. This has two advantages:
1) Check early if a target isn't known by the current osmo-dev version,
displaying a useful error to the user instead of just failing during
make later on:
ERROR: filter_projects_deps_targets: can't find project osmo-new-project in projects_deps!
[testenv] gen_makefile.py from osmo-dev failed!
[testenv] Your osmo-dev.git clone might be outdated, try:
[testenv] $ git -C /home/user/code/osmo-dev pull
2) Parsing the generated Makefile when running "make" after generating
it doesn't cause a noticable delay anymore. On my machine (with many
Osmocom git repos cloned in osmo-dev's src dir) it went from several
seconds to instantly.
Depends: osmo-dev I89bb60e94dd03606dbba5a3609d5e1a95993af5b
Change-Id: Ib2d8802b305f145d27aab3c1cc3129709b93d93d
---
M _testenv/testenv.py
M _testenv/testenv/osmo_dev.py
2 files changed, 4 insertions(+), 12 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/42/40842/1
diff --git a/_testenv/testenv.py b/_testenv/testenv.py
index ebc625e..9e33e37 100755
--- a/_testenv/testenv.py
+++ b/_testenv/testenv.py
@@ -45,9 +45,6 @@
testenv.podman.start()
testenv.podman.check_titan_version()
- if not testenv.args.binary_repo:
- testenv.osmo_dev.init()
-
testenv.testsuite.init()
testenv.testsuite.build()
@@ -55,6 +52,7 @@
if not testenv.args.binary_repo:
for cfg_name, cfg in testenv.testenv_cfg.cfgs.items():
testenv.testenv_cfg.set_current(cfg_name)
+ testenv.osmo_dev.init(cfg)
testenv.osmo_dev.make(cfg)
# Run the components + testsuite
diff --git a/_testenv/testenv/osmo_dev.py b/_testenv/testenv/osmo_dev.py
index 8f9667e..0b6bdf0 100644
--- a/_testenv/testenv/osmo_dev.py
+++ b/_testenv/testenv/osmo_dev.py
@@ -7,8 +7,6 @@
import testenv
import testenv.cmd
-init_done = False
-
def get_osmo_dev_dir():
# Users may have used osmo-dev to clone osmo-ttcn3-hacks:
@@ -66,12 +64,7 @@
return ret
-def init():
- global init_done
-
- if init_done:
- return
-
+def init(cfg):
extra_opts = []
if testenv.args.asan:
extra_opts += ["sanitize.opts"]
@@ -91,6 +84,8 @@
"--src-dir",
testenv.src_dir,
"--autoreconf-in-src-copy",
+ "--targets",
+ ",".join(get_targets(cfg)),
"default.opts",
"ccache.opts",
"gtp_linux.opts",
@@ -111,7 +106,6 @@
logging.critical("Your osmo-dev.git clone might be outdated, try:")
logging.critical(f"$ git -C {shlex.quote(cwd)} pull")
sys.exit(1)
- init_done = True
def make(cfg):
--
To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/40842?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Change-Id: Ib2d8802b305f145d27aab3c1cc3129709b93d93d
Gerrit-Change-Number: 40842
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-dev/+/40837?usp=email )
Change subject: gen_makefile: turn projects_deps into a dict
......................................................................
gen_makefile: turn projects_deps into a dict
Let read_projects_deps() return a dictionary instead of a list of
tuples. This allows looking up entries in the dictionary by project
name, which I'm making use of in a future patch.
While using a dictionary here makes more sense, I assume this was
implemented as list of tuples because with earlier (long EOL) python3
versions, the order of entries in the dictionary were not stable. From
https://docs.python.org/3/library/stdtypes.html#dict:
> Changed in version 3.7: Dictionary order is guaranteed to be insertion
> order. This behavior was an implementation detail of CPython from 3.6.
Change-Id: I74c53f4d6dda791c5d9e14f2274260f0e8cbbad2
---
M gen_makefile.py
1 file changed, 9 insertions(+), 9 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-dev refs/changes/37/40837/1
diff --git a/gen_makefile.py b/gen_makefile.py
index e06dbd2..c49a27a 100755
--- a/gen_makefile.py
+++ b/gen_makefile.py
@@ -144,15 +144,15 @@
self.extend(k, v)
def read_projects_deps(path):
- 'Read deps config and return tuples of (project_name, which-other-to-build-first).'
- l = []
+ 'Read deps config and return a dict of {project_name: which-other-to-build-first, …}.'
+ ret = {}
for line in open(path):
line = line.strip()
if not line or line.startswith('#'):
continue
tokens = line.split()
- l.append((tokens[0], tokens[1:]))
- return l
+ ret[tokens[0]] = tokens[1:]
+ return ret
def read_projects_dict(path):
'Read urls/buildsystems config and return dict {project_name: url, …}.'
@@ -170,7 +170,7 @@
'Read config opts file and return tuples of (project_name, config-opts).'
if not path:
return {}
- return dict(read_projects_deps(path))
+ return read_projects_deps(path)
def gen_makefile_clone(proj, src, src_proj, update_src_copy_cmd):
if proj == "osmocom-bb_layer23":
@@ -646,17 +646,17 @@
"""
# convenience target: clone all repositories first
-content += 'clone: \\\n\t' + ' \\\n\t'.join([ '.make.%s.clone' % p for p, d in projects_deps ]) + '\n\n'
+content += 'clone: \\\n\t' + ' \\\n\t'.join([ '.make.%s.clone' % p for p, d in projects_deps.items() ]) + '\n\n'
# convenience target: clean all
-content += 'clean: \\\n\t' + ' \\\n\t'.join([ '%s-clean' % p for p, d in projects_deps ]) + '\n\n'
+content += 'clean: \\\n\t' + ' \\\n\t'.join([ '%s-clean' % p for p, d in projects_deps.items() ]) + '\n\n'
# now the actual useful build rules
content += 'all: clone all-install\n\n'
-content += 'all-install: \\\n\t' + ' \\\n\t'.join([ '.make.%s.install' % p for p, d in projects_deps ]) + '\n\n'
+content += 'all-install: \\\n\t' + ' \\\n\t'.join([ '.make.%s.install' % p for p, d in projects_deps.items() ]) + '\n\n'
-for proj, deps in projects_deps:
+for proj, deps in projects_deps.items():
all_config_opts = []
all_config_opts.extend(configure_opts.get('ALL') or [])
all_config_opts.extend(configure_opts.get(proj) or [])
--
To view, visit https://gerrit.osmocom.org/c/osmo-dev/+/40837?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: osmo-dev
Gerrit-Branch: master
Gerrit-Change-Id: I74c53f4d6dda791c5d9e14f2274260f0e8cbbad2
Gerrit-Change-Number: 40837
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-dev/+/40839?usp=email )
Change subject: gen_makefile: add --targets
......................................................................
gen_makefile: add --targets
Add an option to specify the targets to include in the Makefile, instead
of always including all of them. This option will be used by testenv to
check early if there are any unknown targets, and to make running the
generated Makefile faster.
Running the Makefile is currently slowed down when having a lot of
projects cloned in osmo-dev's src dir, because regardless of the target
to be built, make is looking for all source files for all projects while
parsing the Makefile. For each project we have:
libosmocore_configure_files := $(shell find -L …/src/libosmocore …)
libosmocore_files := $(shell find -L …/src/libosmocore …)
Changing ":=" to "=" doesn't make it faster either, make still executes
the find commands while parsing target lines such as:
.make.libosmocore.build: .make.libosmocore.configure $(libosmocore_files)
Related: osmo-ttcn3-hacks Ib2d8802b305f145d27aab3c1cc3129709b93d93d
Change-Id: I89bb60e94dd03606dbba5a3609d5e1a95993af5b
---
M gen_makefile.py
M tests/test_gen_makefile.py
2 files changed, 51 insertions(+), 0 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-dev refs/changes/39/40839/1
diff --git a/gen_makefile.py b/gen_makefile.py
index 5eb28ef..4730c59 100755
--- a/gen_makefile.py
+++ b/gen_makefile.py
@@ -153,6 +153,9 @@
parser.add_argument('-A', '--autoreconf-in-src-copy', action='store_true',
help="run autoreconf in a copy of the source dir, avoids 'run make distclean' errors")
+parser.add_argument('--targets',
+ help="comma separated list of high-level targets to build instead of all targets")
+
args = parser.parse_args()
class listdict(dict):
@@ -215,6 +218,40 @@
ret += f"{short}: {' '.join(full)}\n"
return ret
+def filter_projects_deps_targets():
+ if not args.targets:
+ return projects_deps
+
+ ret = {}
+ for target in args.targets.split(","):
+ # .make.osmocom-bb.clone -> osmocom-bb
+ if target.startswith(".make."):
+ target = target.split(".")[2]
+
+ current_targets = [target]
+ if target in convenience_targets:
+ current_targets = convenience_targets[target]
+
+ for target in current_targets:
+ # Add target + all dependencies to ret
+ queue = [target]
+ while queue:
+ project = queue.pop()
+ if project not in projects_deps:
+ print()
+ print(f"ERROR: filter_projects_deps_targets: can't find project {project} in projects_deps!")
+ print()
+ sys.exit(1)
+
+ deps = projects_deps[project]
+ ret[project] = deps
+
+ for dep in deps:
+ if dep not in ret:
+ queue += [dep]
+
+ return ret
+
def gen_makefile_clone(proj, src, src_proj, update_src_copy_cmd):
if proj == "osmocom-bb_layer23":
return f'''
@@ -555,6 +592,7 @@
'''
projects_deps = read_projects_deps(all_deps_file)
+projects_deps = filter_projects_deps_targets()
projects_urls = read_projects_dict(all_urls_file)
projects_buildsystems = read_projects_dict(all_buildsystems_file)
configure_opts = listdict()
@@ -631,6 +669,8 @@
content += " --build-debug \\\n"
if args.autoreconf_in_src_copy:
content += " --autoreconf-in-src-copy \\\n"
+if args.targets:
+ content += f" --targets={shlex.quote(args.targets)} \\\n"
content += " $(NULL)\n"
if args.autoreconf_in_src_copy:
diff --git a/tests/test_gen_makefile.py b/tests/test_gen_makefile.py
index 2ae0667..6232003 100644
--- a/tests/test_gen_makefile.py
+++ b/tests/test_gen_makefile.py
@@ -64,3 +64,14 @@
run_cmd(["./gen_makefile.py", "-m", tmp_path, "--autoreconf-in-src-copy"], cwd=osmo_dev_path)
run_cmd(["make", ".make.open5gs.configure"], cwd=tmp_path)
run_make_regen_2x(tmp_path)
+
+
+def test_gen_makefile_with_targets_arg(tmp_path):
+ run_cmd(["./gen_makefile.py", "-m", tmp_path, "--targets", "trxcon,osmo-mgw"], cwd=osmo_dev_path)
+ run_cmd("grep -q '^osmocom-bb_trxcon_files :=' Makefile", cwd=tmp_path, shell=True)
+ run_cmd("grep -q '^osmo-mgw_files :=' Makefile", cwd=tmp_path, shell=True)
+ run_cmd("grep -q '^libosmocore_files :=' Makefile", cwd=tmp_path, shell=True)
+ run_cmd("! grep -q '^osmo-bts_files :=' Makefile", cwd=tmp_path, shell=True)
+ run_cmd("! grep -q '^osmo-s1gw_files :=' Makefile", cwd=tmp_path, shell=True)
+ run_cmd("! grep -q '^open5gs_files :=' Makefile", cwd=tmp_path, shell=True)
+ run_make_regen_2x(tmp_path)
--
To view, visit https://gerrit.osmocom.org/c/osmo-dev/+/40839?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: osmo-dev
Gerrit-Branch: master
Gerrit-Change-Id: I89bb60e94dd03606dbba5a3609d5e1a95993af5b
Gerrit-Change-Number: 40839
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Paul Bichis has restored this change. ( https://gerrit.osmocom.org/c/libosmo-abis/+/40833?usp=email )
Change subject: Create rpm make targets and add libosmo-abis.spec.in
......................................................................
Restored
--
To view, visit https://gerrit.osmocom.org/c/libosmo-abis/+/40833?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: restore
Gerrit-Project: libosmo-abis
Gerrit-Branch: master
Gerrit-Change-Id: I86c2c621887d1d9352f3ddfa1270c1379cf50fd9
Gerrit-Change-Number: 40833
Gerrit-PatchSet: 2
Gerrit-Owner: Paul Bichis <paul(a)afk.ro>
Gerrit-Reviewer: Jenkins Builder
Paul Bichis has abandoned this change. ( https://gerrit.osmocom.org/c/libosmo-abis/+/40833?usp=email )
Change subject: Create rpm make targets and add libosmo-abis.spec.in
......................................................................
Abandoned
--
To view, visit https://gerrit.osmocom.org/c/libosmo-abis/+/40833?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: abandon
Gerrit-Project: libosmo-abis
Gerrit-Branch: master
Gerrit-Change-Id: I86c2c621887d1d9352f3ddfa1270c1379cf50fd9
Gerrit-Change-Number: 40833
Gerrit-PatchSet: 2
Gerrit-Owner: Paul Bichis <paul(a)afk.ro>
Gerrit-Reviewer: Jenkins Builder
Paul Bichis has abandoned this change. ( https://gerrit.osmocom.org/c/libosmo-abis/+/40832?usp=email )
Change subject: Create rpm make targets and add libosmo-abis.spec.in
......................................................................
Abandoned
--
To view, visit https://gerrit.osmocom.org/c/libosmo-abis/+/40832?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: abandon
Gerrit-Project: libosmo-abis
Gerrit-Branch: master
Gerrit-Change-Id: I199d1168538b5220571c42b1fc35c7f5f9f3e4b4
Gerrit-Change-Number: 40832
Gerrit-PatchSet: 2
Gerrit-Owner: Paul Bichis <paul(a)afk.ro>
Gerrit-Reviewer: Jenkins Builder