Change in osmo-gsm-tester[master]: Cmdline arg -c sets main configuration file (old paths.conf) instead ...

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/gerrit-log@lists.osmocom.org/.

pespin gerrit-no-reply at lists.osmocom.org
Mon May 11 17:46:01 UTC 2020


pespin has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18204 )


Change subject: Cmdline arg -c sets main configuration file (old paths.conf) instead of dir containing it
......................................................................

Cmdline arg -c sets main configuration file (old paths.conf) instead of dir containing it

It has been notified that current configuration system is difficult to
understand and to use, so it has been envisioned to refactor it a bit.
The idea is that the user passes a -c path/to/main.conf file, which in
turn contains whatever osmo-gsm-tester main settings supports (basically
what old paths.conf used to be, plus some files harcoded to the same -c
directory are now configurable through the main configuration file).

TODO:
* Update documentation
* Update sample config files (doc/examples).
* Trial loaded from main.conf if not overriden by cmdline

Change-Id: Ieca65b71b543c44cfcec8e83efd0fe053c432e55
---
M contrib/jenkins-run.sh
M selftest/resource_test/resource_test.py
M selftest/scenario_test/scenario_test.py
M selftest/suite_test/suite_test.py
M src/osmo-gsm-tester.py
M src/osmo_gsm_tester/core/config.py
M src/osmo_gsm_tester/core/resource.py
R sysmocom/main.conf
M sysmocom/ttcn3/jenkins-run.sh
R sysmocom/ttcn3/main.conf
10 files changed, 108 insertions(+), 80 deletions(-)



  git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/04/18204/1

diff --git a/contrib/jenkins-run.sh b/contrib/jenkins-run.sh
index 89ff13f..1c31a9d 100755
--- a/contrib/jenkins-run.sh
+++ b/contrib/jenkins-run.sh
@@ -2,7 +2,7 @@
 set -e -x
 base="$PWD"
 SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P) # this file's directory
-OSMO_GSM_TESTER_CONF=${OSMO_GSM_TESTER_CONF:-${SCRIPT_DIR}/../sysmocom}
+OSMO_GSM_TESTER_CONF=${OSMO_GSM_TESTER_CONF:-${SCRIPT_DIR}/../sysmocom/main.conf}
 
 time_start="$(date '+%F %T')"
 
diff --git a/selftest/resource_test/resource_test.py b/selftest/resource_test/resource_test.py
index b74ba2a..f18aa73 100755
--- a/selftest/resource_test/resource_test.py
+++ b/selftest/resource_test/resource_test.py
@@ -13,7 +13,7 @@
 workdir = util.get_tempdir()
 
 # override config locations to make sure we use only the test conf
-config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'conf')
+config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'conf', 'paths.conf')
 
 log.get_process_id = lambda: '123-1490837279'
 
diff --git a/selftest/scenario_test/scenario_test.py b/selftest/scenario_test/scenario_test.py
index f5f42f7..15f8983 100755
--- a/selftest/scenario_test/scenario_test.py
+++ b/selftest/scenario_test/scenario_test.py
@@ -18,7 +18,7 @@
     'foobar' : schema.BOOL_STR,
     }
 
-config.override_conf = os.path.join(os.path.dirname(sys.argv[0]))
+config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'paths.conf')
 
 def print_scenario(sc):
     # we use copy() to be able to get the dictionary in super class of Scenario:
diff --git a/selftest/suite_test/suite_test.py b/selftest/suite_test/suite_test.py
index a096027..4b32439 100755
--- a/selftest/suite_test/suite_test.py
+++ b/selftest/suite_test/suite_test.py
@@ -11,7 +11,7 @@
 from osmo_gsm_tester.core import suite
 from osmo_gsm_tester.core.schema import generate_schemas, get_all_schema
 
-config.override_conf = os.path.join(os.path.dirname(sys.argv[0]))
+config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'paths.conf')
 
 example_trial_dir = os.path.join('test_trial_tmp')
 
diff --git a/src/osmo-gsm-tester.py b/src/osmo-gsm-tester.py
index fb5574b..204b1c7 100755
--- a/src/osmo-gsm-tester.py
+++ b/src/osmo-gsm-tester.py
@@ -24,9 +24,9 @@
 
 Examples:
 
-./osmo-gsm-tester.py -c doc/examples/2g_osmocom/ ~/my_trial_package/ -s osmo_trx
-./osmo-gsm-tester.py -c doc/examples/2g_osmocom/ ~/my_trial_package/ -s sms_tests:dyn_ts+eu_band+bts_sysmo
-./osmo-gsm-tester.py -c sysmocom/ ~/my_trial_package/ -s sms_tests/mo_mt_sms:bts_trx
+./osmo-gsm-tester.py -c doc/examples/2g_osmocom/main.conf ~/my_trial_package/ -s osmo_trx
+./osmo-gsm-tester.py -c doc/examples/2g_osmocom/main.conf ~/my_trial_package/ -s sms_tests:dyn_ts+eu_band+bts_sysmo
+./osmo-gsm-tester.py -c sysmocom/main.conf ~/my_trial_package/ -s sms_tests/mo_mt_sms:bts_trx
 
 (The names for test suites and scenarios used in these examples must be defined
 by the osmo-gsm-tester configuration.)
@@ -54,8 +54,9 @@
 A test run thus needs to define:
 * A trial package containing built binaries
 * A set of test suites, each with its combinations of scenarios
-* A configuration directory specifying sets of resources, default configurations
-  and paths on where to find suites, scenarios, etc.
+* A main configuration file specifying paths to other files containing sets of
+  resources, default configurations and paths on where to find suites,
+  scenarios, etc.
 
 If no combination of suites and scenarios is provided, the default list of
 suites will be run as defined in the osmo-gsm-tester configuration.
@@ -101,8 +102,8 @@
     # is easiest to maintain.
     parser.add_argument('-V', '--version', action='store_true',
             help='Show version')
-    parser.add_argument('-c', '--conf-dir', dest='conf_dir',
-            help='''Specify configuration directory path (containing paths.conf)''')
+    parser.add_argument('-c', '--conf-path', dest='conf_path',
+            help='''Specify main configuration file path''')
     parser.add_argument('trial_package',
             help='Directory containing binaries to test')
     parser.add_argument('-s', '--suite-scenario', dest='suite_scenario', action='append',
@@ -140,18 +141,18 @@
         log.style_change(trace=True)
     if args.source:
         log.style_change(src=True)
-    if args.conf_dir:
-        config.override_conf = args.conf_dir
+    if args.conf_path:
+        config.override_conf = args.conf_path
 
     combination_strs = list(args.suite_scenario or [])
 
     if not combination_strs:
-        combination_strs = config.read_config_file(config.DEFAULT_SUITES_CONF, if_missing_return=[])
+        combination_strs = config.read_config_file(config.CFG_DEFAULT_SUITES_CONF, if_missing_return=[])
 
         if combination_strs:
             print('Running default suites:\n  ' + ('\n  '.join(combination_strs)))
         else:
-            print('No default suites configured (%r)' % config.DEFAULT_SUITES_CONF)
+            print('Failed to load default suites (%r)' % config.get_main_config_value(config.DEFAULT_SUITES_CONF, fail_if_missing=False))
 
 
     if not combination_strs:
diff --git a/src/osmo_gsm_tester/core/config.py b/src/osmo_gsm_tester/core/config.py
index 88e522d..991e724 100644
--- a/src/osmo_gsm_tester/core/config.py
+++ b/src/osmo_gsm_tester/core/config.py
@@ -53,6 +53,7 @@
 import yaml
 import os
 import copy
+import pprint
 
 from . import log, util, template
 from . import schema
@@ -60,94 +61,122 @@
 
 override_conf = None
 
-DEFAULT_CONFIG_LOCATIONS = [
-    '.',
-    os.path.join(os.getenv('HOME'), '.config', 'osmo-gsm-tester'),
-    '/usr/local/etc/osmo-gsm-tester',
-    '/etc/osmo-gsm-tester'
-    ]
-
-PATHS_CONF = 'paths.conf'
-DEFAULT_SUITES_CONF = 'default-suites.conf'
-DEFAULTS_CONF = 'defaults.conf'
-RESOURCES_CONF = 'resources.conf'
-
-PATH_STATE_DIR = 'state_dir'
-PATH_SUITES_DIR = 'suites_dir'
-PATH_SCENARIOS_DIR = 'scenarios_dir'
-PATHS_SCHEMA = {
-        PATH_STATE_DIR: schema.STR,
-        PATH_SUITES_DIR: schema.STR,
-        PATH_SCENARIOS_DIR: schema.STR,
+CFG_STATE_DIR = 'state_dir'
+CFG_SUITES_DIR = 'suites_dir'
+CFG_SCENARIOS_DIR = 'scenarios_dir'
+CFG_DEFAULT_SUITES_CONF = 'default_suites_conf_path'
+CFG_DEFAULTS_CONF = 'defaults_conf_path'
+CFG_RESOURCES_CONF = 'resource_conf_path'
+MAIN_CONFIG_SCHEMA = {
+        CFG_STATE_DIR: schema.STR,
+        CFG_SUITES_DIR: schema.STR,
+        CFG_SCENARIOS_DIR: schema.STR,
+        CFG_DEFAULT_SUITES_CONF: schema.STR,
+        CFG_DEFAULTS_CONF: schema.STR,
+        CFG_RESOURCES_CONF: schema.STR,
     }
 
-PATHS_TEMPDIR_STR = '$TEMPDIR'
+DF_CFG_STATE_DIR = '/var/tmp/osmo-gsm-tester/state/'
+DF_CFG_SUITES_DIR = './suites'
+DF_CFG_SCENARIOS_DIR = './scenarios'
+DF_CFG_DEFAULT_SUITES_CONF = './default-suites.conf'
+DF_CFG_DEFAULTS_CONF = './defaults.conf'
+DF_CFG_RESOURCES_CONF = './resources.conf'
 
-PATHS = None
+DEFAULT_CONFIG_FILENAME = 'main.conf'
 
-def _get_config_file(basename, fail_if_missing=True):
+DEFAULT_CONFIG_LOCATIONS = [
+    '.',
+    os.path.join(os.getenv('HOME'), '.config', 'osmo-gsm-tester', DEFAULT_CONFIG_FILENAME),
+    os.path.join('/usr/local/etc/osmo-gsm-tester', DEFAULT_CONFIG_FILENAME),
+    os.path.join('/etc/osmo-gsm-tester', DEFAULT_CONFIG_FILENAME)
+    ]
+
+MAIN_CONFIG = None
+MAIN_CONFIG_PATH = None
+
+def _find_main_config_path():
     if override_conf:
         locations = [ override_conf ]
     elif os.getenv('OSMO_GSM_TESTER_CONF'):
         ENV_CONF = os.getenv('OSMO_GSM_TESTER_CONF')
-        log.err('Using environment variable OSMO_GSM_TESTER_CONF=%s is deprecated. Rather use -c command line argument!' % ENV_CONF)
-        locations = [ ENV_CONF ]
+        log.err('Using environment variable OSMO_GSM_TESTER_CONF=%s(/paths.conf) is deprecated. Rather use -c command line argument!' % ENV_CONF)
+        locations = [ ENV_CONF + 'paths.conf' ] # directory is expected in OSMO_GSM_TESTER_CONF, bakcward compatibility
     else:
         locations = DEFAULT_CONFIG_LOCATIONS
 
     for l in locations:
         real_l = os.path.realpath(l)
-        p = os.path.realpath(os.path.join(real_l, basename))
-        if os.path.isfile(p):
-            log.dbg('Found config file', basename, 'as', p, 'in', l, 'which is', real_l, _category=log.C_CNF)
-            return (p, real_l)
-    if not fail_if_missing:
-        return None, None
-    raise RuntimeError('configuration file not found: %r in %r' % (basename,
-        [os.path.abspath(p) for p in locations]))
+        if os.path.isfile(real_l):
+            log.dbg('Found main configuration file in ', l, 'which is', real_l, _category=log.C_CNF)
+            return real_l
+    raise RuntimeError('Main configuration file not found in %r' % ([l for l in locations]))
 
-def get_config_file(basename, fail_if_missing=True):
-    path, found_in = _get_config_file(basename, fail_if_missing)
+def _get_main_config_path():
+    global MAIN_CONFIG_PATH
+    if MAIN_CONFIG_PATH is None:
+        MAIN_CONFIG_PATH = _find_main_config_path()
+    return MAIN_CONFIG_PATH
+
+def main_config_path_to_abspath(path):
+    'Relative files in main config are relative towards the config file, not towards $CWD'
+    if not path.startswith(os.pathsep):
+        return os.path.realpath(os.path.join(os.path.dirname(_get_main_config_path()), path))
     return path
 
-def read_config_file(basename, validation_schema=None, if_missing_return=False):
+def _get_main_config():
+    global MAIN_CONFIG
+    if MAIN_CONFIG is None:
+        cfg = read(_get_main_config_path(), MAIN_CONFIG_SCHEMA)
+        MAIN_CONFIG = {
+            CFG_STATE_DIR: DF_CFG_STATE_DIR,
+            CFG_SUITES_DIR: DF_CFG_SUITES_DIR,
+            CFG_SCENARIOS_DIR: DF_CFG_SCENARIOS_DIR,
+            CFG_DEFAULT_SUITES_CONF: DF_CFG_DEFAULT_SUITES_CONF,
+            CFG_DEFAULTS_CONF: DF_CFG_DEFAULTS_CONF,
+            CFG_RESOURCES_CONF: DF_CFG_RESOURCES_CONF,
+            }
+        overlay(MAIN_CONFIG, cfg)
+        for key, path in sorted(MAIN_CONFIG.items()):
+             MAIN_CONFIG[key] = main_config_path_to_abspath(path)
+        log.dbg('MAIN CONFIG:\n' + pprint.pformat(MAIN_CONFIG), _category=log.C_CNF)
+    return MAIN_CONFIG
+
+def get_main_config_value(cfg_name, fail_if_missing=True):
+    cfg = _get_main_config()
+    f = cfg.get(cfg_name, None)
+    if f is None and fail_if_missing:
+        raise RuntimeError('Missing configuration %s' % (cfg_name))
+    return f
+
+def read_config_file(cfg_name, validation_schema=None, if_missing_return=False):
+    '''Read content of config file cfg_name (referring to key in main config).
+    If "if_missing_return" is different than False, then instead of failing it will return whatever it is stored in that arg
+    '''
     fail_if_missing = True
     if if_missing_return is not False:
         fail_if_missing = False
-    path = get_config_file(basename, fail_if_missing=fail_if_missing)
+    path = get_main_config_value(cfg_name, fail_if_missing=fail_if_missing)
     if path is None:
         return if_missing_return
     return read(path, validation_schema=validation_schema, if_missing_return=if_missing_return)
 
-def get_configured_path(label, allow_unset=False):
-    global PATHS
-
-    if PATHS is None:
-        paths_file, found_in = _get_config_file(PATHS_CONF)
-        PATHS = read(paths_file, PATHS_SCHEMA)
-        # sorted for deterministic regression test results
-        for key, path in sorted(PATHS.items()):
-            if not path.startswith(os.pathsep):
-                PATHS[key] = os.path.realpath(os.path.join(found_in, path))
-                log.dbg(paths_file + ': relative path', path, 'is', PATHS[key], _category=log.C_CNF)
-    p = PATHS.get(label)
-    if p is None and not allow_unset:
-        raise RuntimeError('missing configuration in %s: %r' % (PATHS_CONF, label))
-
-    log.dbg('Found path', label, 'as', p, _category=log.C_CNF)
-    if p.startswith(PATHS_TEMPDIR_STR):
-        p = os.path.join(get_tempdir(), p[len(PATHS_TEMPDIR_STR):])
-        log.dbg('Path', label, 'contained', PATHS_TEMPDIR_STR, 'and becomes', p, _category=log.C_CNF)
-    return p
-
 def get_state_dir():
-    return Dir(get_configured_path(PATH_STATE_DIR))
+    return Dir(get_main_config_value(CFG_STATE_DIR))
 
 def get_suites_dir():
-    return Dir(get_configured_path(PATH_SUITES_DIR))
+    return Dir(get_main_config_value(CFG_SUITES_DIR))
 
 def get_scenarios_dir():
-    return Dir(get_configured_path(PATH_SCENARIOS_DIR))
+    return Dir(get_main_config_value(CFG_SCENARIOS_DIR))
+
+DEFAULTS_CONF = None
+def get_defaults(for_kind):
+    global DEFAULTS_CONF
+    if DEFAULTS_CONF is None:
+        DEFAULTS_CONF = read_config_file(CFG_DEFAULTS_CONF, if_missing_return={})
+    defaults = DEFAULTS_CONF.get(for_kind, {})
+    return copy.deepcopy(defaults)
 
 def read(path, validation_schema=None, if_missing_return=False):
     log.ctx(path)
@@ -191,10 +220,6 @@
     config = yaml.safe_load(_tostr(_standardize_item(config)))
     return config
 
-def get_defaults(for_kind):
-    defaults = read_config_file(DEFAULTS_CONF, if_missing_return={})
-    return defaults.get(for_kind, {})
-
 def overlay(dest, src):
     if is_dict(dest):
         if not is_dict(src):
diff --git a/src/osmo_gsm_tester/core/resource.py b/src/osmo_gsm_tester/core/resource.py
index a1a8ea5..af05a51 100644
--- a/src/osmo_gsm_tester/core/resource.py
+++ b/src/osmo_gsm_tester/core/resource.py
@@ -48,7 +48,7 @@
     _registered_exit_handler = False
 
     def __init__(self):
-        self.config_path = config.get_config_file(config.RESOURCES_CONF)
+        self.config_path = config.get_main_config_value(config.CFG_RESOURCES_CONF)
         self.state_dir = config.get_state_dir()
         super().__init__(log.C_CNF, conf=self.config_path, state=self.state_dir.path)
         self.read_conf()
diff --git a/sysmocom/paths.conf b/sysmocom/main.conf
similarity index 100%
rename from sysmocom/paths.conf
rename to sysmocom/main.conf
diff --git a/sysmocom/ttcn3/jenkins-run.sh b/sysmocom/ttcn3/jenkins-run.sh
index c744606..2bc687d 100755
--- a/sysmocom/ttcn3/jenkins-run.sh
+++ b/sysmocom/ttcn3/jenkins-run.sh
@@ -1,6 +1,8 @@
 #!/bin/sh
 set -e -x
 base="$PWD"
+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P) # this file's directory
+OSMO_GSM_TESTER_CONF=${OSMO_GSM_TESTER_CONF:-${SCRIPT_DIR}/main.conf}
 
 time_start="$(date '+%F %T')"
 
diff --git a/sysmocom/ttcn3/paths.conf b/sysmocom/ttcn3/main.conf
similarity index 100%
rename from sysmocom/ttcn3/paths.conf
rename to sysmocom/ttcn3/main.conf

-- 
To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18204
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings

Gerrit-Project: osmo-gsm-tester
Gerrit-Branch: master
Gerrit-Change-Id: Ieca65b71b543c44cfcec8e83efd0fe053c432e55
Gerrit-Change-Number: 18204
Gerrit-PatchSet: 1
Gerrit-Owner: pespin <pespin at sysmocom.de>
Gerrit-MessageType: newchange
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/gerrit-log/attachments/20200511/776e0ab1/attachment.htm>


More information about the gerrit-log mailing list