<p>pespin has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/17850">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">check_dependencies: Import modules dynamically and find related debian packages<br><br>This way we don't need to manually add new imports here or drop unusued<br>ones. It also makes sure local imports in all our py files is correct.<br>For instance, running the script already caught an issue which is added<br>to this patch (osmo_ms_driver/__main__.py).<br><br>This new version of the script also allows specifying subsets of<br>features to skip when checking for dependencies. This way, for instance<br>somebody not willing to use a sispm powersupply can stil check all the<br>needed dependencies are fine.<br><br>This new tool will make it easier to slowly make some dependencies only<br>used by some object test classes optional (for instance, python-smpplib<br>if user doesn't want to run an ESME node).<br><br>Change-Id: I29ddf8971837754abd930d847bd1036e8e510de6<br>---<br>M check_dependencies.py<br>M src/osmo_ms_driver/__main__.py<br>2 files changed, 125 insertions(+), 28 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/50/17850/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/check_dependencies.py b/check_dependencies.py</span><br><span>index c3b1d64..3efbaf9 100755</span><br><span>--- a/check_dependencies.py</span><br><span>+++ b/check_dependencies.py</span><br><span>@@ -3,31 +3,128 @@</span><br><span> # just import all python3 modules used by osmo-gsm-tester to make sure they are</span><br><span> # installed.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-from inspect import getframeinfo, stack</span><br><span style="color: hsl(0, 100%, 40%);">-from mako.lookup import TemplateLookup</span><br><span style="color: hsl(0, 100%, 40%);">-from mako.template import Template</span><br><span style="color: hsl(0, 100%, 40%);">-import argparse</span><br><span style="color: hsl(0, 100%, 40%);">-import contextlib</span><br><span style="color: hsl(0, 100%, 40%);">-import copy</span><br><span style="color: hsl(0, 100%, 40%);">-import difflib</span><br><span style="color: hsl(0, 100%, 40%);">-import fcntl</span><br><span style="color: hsl(0, 100%, 40%);">-import inspect</span><br><span style="color: hsl(0, 100%, 40%);">-import io</span><br><span style="color: hsl(0, 100%, 40%);">-import os</span><br><span style="color: hsl(0, 100%, 40%);">-import pprint</span><br><span style="color: hsl(0, 100%, 40%);">-import re</span><br><span style="color: hsl(0, 100%, 40%);">-import subprocess</span><br><span style="color: hsl(0, 100%, 40%);">-import sys</span><br><span style="color: hsl(0, 100%, 40%);">-import tempfile</span><br><span style="color: hsl(0, 100%, 40%);">-import time</span><br><span style="color: hsl(0, 100%, 40%);">-import traceback</span><br><span style="color: hsl(0, 100%, 40%);">-import yaml</span><br><span style="color: hsl(0, 100%, 40%);">-import pydbus</span><br><span style="color: hsl(0, 100%, 40%);">-import sqlite3</span><br><span style="color: hsl(0, 100%, 40%);">-import sispm</span><br><span style="color: hsl(0, 100%, 40%);">-import smpplib</span><br><span style="color: hsl(0, 100%, 40%);">-import urllib.request</span><br><span style="color: hsl(0, 100%, 40%);">-import xml.etree.ElementTree</span><br><span style="color: hsl(0, 100%, 40%);">-import numpy</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-print('dependencies ok')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+import os</span><br><span style="color: hsl(120, 100%, 40%);">+import sys</span><br><span style="color: hsl(120, 100%, 40%);">+import argparse</span><br><span style="color: hsl(120, 100%, 40%);">+import pprint</span><br><span style="color: hsl(120, 100%, 40%);">+import subprocess</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+feature_module_map = {</span><br><span style="color: hsl(120, 100%, 40%);">+ 'powersupply_intellinet' : ['powersupply_intellinet'],</span><br><span style="color: hsl(120, 100%, 40%);">+ 'powersupply_sispm' : ['powersupply_sispm'],</span><br><span style="color: hsl(120, 100%, 40%);">+ 'rfemu_amarisoftctrl': ['rfemu_amarisoftctrl'],</span><br><span style="color: hsl(120, 100%, 40%);">+ 'rfemu_minicircuits': ['rfemu_minicircuits'],</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def skip_features_to_skip_modules(skip_features):</span><br><span style="color: hsl(120, 100%, 40%);">+ skip_obj_modules = []</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for skip_feature in skip_features:</span><br><span style="color: hsl(120, 100%, 40%);">+ if skip_feature not in feature_module_map:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('feature %s doesn\'t exist!' % skip_feature)</span><br><span style="color: hsl(120, 100%, 40%);">+ for skip_module in feature_module_map[skip_feature]:</span><br><span style="color: hsl(120, 100%, 40%);">+ skip_obj_modules.append(skip_module)</span><br><span style="color: hsl(120, 100%, 40%);">+ return skip_obj_modules</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def import_runtime_dependencies():</span><br><span style="color: hsl(120, 100%, 40%);">+ # we don't have any right now, but in the future if we import a module during runtime (eg inside a function), then we need to place it here:</span><br><span style="color: hsl(120, 100%, 40%);">+ # import foobar</span><br><span style="color: hsl(120, 100%, 40%);">+ pass</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def import_all_py_in_dir(rel_path, skip_modules=[]):</span><br><span style="color: hsl(120, 100%, 40%);">+ selfdir = os.path.dirname(os.path.abspath(__file__))</span><br><span style="color: hsl(120, 100%, 40%);">+ dir = os.path.join(selfdir, rel_path)</span><br><span style="color: hsl(120, 100%, 40%);">+ print('importing files in directory %s' % dir)</span><br><span style="color: hsl(120, 100%, 40%);">+ for entry in os.listdir(dir):</span><br><span style="color: hsl(120, 100%, 40%);">+ full_entry = os.path.join(selfdir, rel_path, entry)</span><br><span style="color: hsl(120, 100%, 40%);">+ if not os.path.isfile(full_entry):</span><br><span style="color: hsl(120, 100%, 40%);">+ if args.verbose:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('skipping entry %s' % full_entry)</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ if not full_entry.endswith('.py'):</span><br><span style="color: hsl(120, 100%, 40%);">+ if args.verbose:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('skipping file %s' % full_entry)</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ modulename = entry[:-3]</span><br><span style="color: hsl(120, 100%, 40%);">+ if modulename in skip_modules:</span><br><span style="color: hsl(120, 100%, 40%);">+ if args.verbose:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('skipping module %s' % modulename)</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ modulepath = rel_path.replace('/', '.') + '.' + modulename</span><br><span style="color: hsl(120, 100%, 40%);">+ print('importing %s' % modulepath)</span><br><span style="color: hsl(120, 100%, 40%);">+ __import__(modulepath, globals(), locals())</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def get_module_names():</span><br><span style="color: hsl(120, 100%, 40%);">+ all_modules=sys.modules.items()</span><br><span style="color: hsl(120, 100%, 40%);">+ all_modules_filtered = {}</span><br><span style="color: hsl(120, 100%, 40%);">+ for mname, m in all_modules:</span><br><span style="color: hsl(120, 100%, 40%);">+ if not hasattr(m, '__file__'):</span><br><span style="color: hsl(120, 100%, 40%);">+ continue # skip built-in modules</span><br><span style="color: hsl(120, 100%, 40%);">+ if mname.startswith('_'):</span><br><span style="color: hsl(120, 100%, 40%);">+ continue # skip internal modules</span><br><span style="color: hsl(120, 100%, 40%);">+ if mname.startswith('src.osmo_') or 'osmo_gsm_tester' in mname or 'osmo_ms_driver' in mname:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue # skip our own local modules</span><br><span style="color: hsl(120, 100%, 40%);">+ mname = mname.split('.')[0] # store only main module</span><br><span style="color: hsl(120, 100%, 40%);">+ if m not in all_modules_filtered.values():</span><br><span style="color: hsl(120, 100%, 40%);">+ all_modules_filtered[mname] = m</span><br><span style="color: hsl(120, 100%, 40%);">+ return all_modules_filtered</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def print_deb_packages(modules):</span><br><span style="color: hsl(120, 100%, 40%);">+ packages_deb = []</span><br><span style="color: hsl(120, 100%, 40%);">+ modules_err = []</span><br><span style="color: hsl(120, 100%, 40%);">+ for mname, m in modules.items():</span><br><span style="color: hsl(120, 100%, 40%);">+ proc = subprocess.Popen(["dpkg", "-S", m.__file__], stdout=subprocess.PIPE, stderr=subprocess.PIPE)</span><br><span style="color: hsl(120, 100%, 40%);">+ outs, errs = proc.communicate()</span><br><span style="color: hsl(120, 100%, 40%);">+ if args.verbose:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('out: %s, err: %s' %(outs, errs))</span><br><span style="color: hsl(120, 100%, 40%);">+ if len(errs): # error -> package not found (installed through pip?)</span><br><span style="color: hsl(120, 100%, 40%);">+ modules_err.append((mname, errs.decode('utf-8')))</span><br><span style="color: hsl(120, 100%, 40%);">+ elif len(outs):</span><br><span style="color: hsl(120, 100%, 40%);">+ outs = outs.decode('utf-8')</span><br><span style="color: hsl(120, 100%, 40%);">+ outs = outs.split()[0].rstrip(':') # first part is debian package name</span><br><span style="color: hsl(120, 100%, 40%);">+ if not outs in packages_deb:</span><br><span style="color: hsl(120, 100%, 40%);">+ packages_deb.append(outs)</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('WARNING: dpkg returns empty!')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ print('Debian packages:')</span><br><span style="color: hsl(120, 100%, 40%);">+ for pkgname in packages_deb:</span><br><span style="color: hsl(120, 100%, 40%);">+ print("\t" + pkgname)</span><br><span style="color: hsl(120, 100%, 40%);">+ print()</span><br><span style="color: hsl(120, 100%, 40%);">+ print('Modules without debian package (pip or setuptools?):')</span><br><span style="color: hsl(120, 100%, 40%);">+ for mname, err in modules_err:</span><br><span style="color: hsl(120, 100%, 40%);">+ print("\t" + mname.ljust(20) + " [" + err.rstrip() +"]")</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+parser = argparse.ArgumentParser(epilog=__doc__, formatter_class=argparse.RawTextHelpFormatter)</span><br><span style="color: hsl(120, 100%, 40%);">+parser.add_argument('-s', '--skip-feature', dest='skip_features', choices=feature_module_map.keys(), action='append',</span><br><span style="color: hsl(120, 100%, 40%);">+ help='''All osmo-gsm-tester features not used by the user running the script''')</span><br><span style="color: hsl(120, 100%, 40%);">+parser.add_argument('-p', '--distro-packages', dest='distro_packages', action='store_true',</span><br><span style="color: hsl(120, 100%, 40%);">+ help='Print distro packages installing modules')</span><br><span style="color: hsl(120, 100%, 40%);">+parser.add_argument('-v', '--verbose', dest='verbose', action='store_true',</span><br><span style="color: hsl(120, 100%, 40%);">+ help='Print a lot more information')</span><br><span style="color: hsl(120, 100%, 40%);">+args = parser.parse_args()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+skip_obj_modules = skip_features_to_skip_modules(list(args.skip_features or []))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+print('Skip checking modules: %r' % skip_obj_modules)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# We need to add it for cross-references between osmo_ms_driver and osmo_gsm_tester to work:</span><br><span style="color: hsl(120, 100%, 40%);">+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), 'src/'))</span><br><span style="color: hsl(120, 100%, 40%);">+import_all_py_in_dir('src/osmo_ms_driver')</span><br><span style="color: hsl(120, 100%, 40%);">+import_all_py_in_dir('src/osmo_gsm_tester/core')</span><br><span style="color: hsl(120, 100%, 40%);">+import_all_py_in_dir('src/osmo_gsm_tester/obj', skip_obj_modules)</span><br><span style="color: hsl(120, 100%, 40%);">+import_all_py_in_dir('src/osmo_gsm_tester')</span><br><span style="color: hsl(120, 100%, 40%);">+import_runtime_dependencies()</span><br><span style="color: hsl(120, 100%, 40%);">+print('Importing dependencies ok, all installed')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+print('Retreiving list of imported modules...')</span><br><span style="color: hsl(120, 100%, 40%);">+modules = get_module_names()</span><br><span style="color: hsl(120, 100%, 40%);">+if args.verbose:</span><br><span style="color: hsl(120, 100%, 40%);">+ for mname, m in modules.items():</span><br><span style="color: hsl(120, 100%, 40%);">+ print('%s --> %s' %(mname, m.__file__))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if args.distro_packages:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('Generating distro package list from imported module list...')</span><br><span style="color: hsl(120, 100%, 40%);">+ print_deb_packages(modules)</span><br><span>diff --git a/src/osmo_ms_driver/__main__.py b/src/osmo_ms_driver/__main__.py</span><br><span>index a4276d9..f84be88 100644</span><br><span>--- a/src/osmo_ms_driver/__main__.py</span><br><span>+++ b/src/osmo_ms_driver/__main__.py</span><br><span>@@ -23,7 +23,7 @@</span><br><span> from .starter import BinaryOptions, MobileTestStarter</span><br><span> from .test_support import imsi_ki_gen</span><br><span> from osmo_gsm_tester.core import log, util</span><br><span style="color: hsl(0, 100%, 40%);">-from osmo_gsm_tester import ms_osmo_mobile</span><br><span style="color: hsl(120, 100%, 40%);">+from osmo_gsm_tester.obj import ms_osmo_mobile</span><br><span> </span><br><span> # System modules</span><br><span> from datetime import timedelta</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/17850">change 17850</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/17850"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I29ddf8971837754abd930d847bd1036e8e510de6 </div>
<div style="display:none"> Gerrit-Change-Number: 17850 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>