<p>pespin <strong>submitted</strong> this change.</p><p><a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18204">View Change</a></p><div style="white-space:pre-wrap">Approvals:
pespin: Looks good to me, approved
Jenkins Builder: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">Cmdline arg -c sets main configuration file (old paths.conf) instead of dir containing it<br><br>It has been notified that current configuration system is difficult to<br>understand and to use, so it has been envisioned to refactor it a bit.<br>The idea is that the user passes a -c path/to/main.conf file, which in<br>turn contains whatever osmo-gsm-tester main settings supports (basically<br>what old paths.conf used to be, plus some files harcoded to the same -c<br>directory are now configurable through the main configuration file).<br><br>Change-Id: Ieca65b71b543c44cfcec8e83efd0fe053c432e55<br>---<br>M contrib/jenkins-run.sh<br>M doc/examples/2g_osmocom/README.md<br>A doc/examples/2g_osmocom/main.conf<br>D doc/examples/2g_osmocom/paths.conf<br>M doc/examples/4g_srsLTE/README.md<br>A doc/examples/4g_srsLTE/main.conf<br>D doc/examples/4g_srsLTE/paths.conf<br>M doc/manuals/chapters/config.adoc<br>M selftest/resource_test/resource_test.ok<br>M selftest/resource_test/resource_test.py<br>M selftest/scenario_test/scenario_test.ok<br>M selftest/scenario_test/scenario_test.py<br>M selftest/suite_test/suite_test.ok<br>M selftest/suite_test/suite_test.py<br>M src/osmo-gsm-tester.py<br>M src/osmo_gsm_tester/core/config.py<br>M src/osmo_gsm_tester/core/resource.py<br>M sysmocom/README.md<br>A sysmocom/main.conf<br>D sysmocom/paths.conf<br>D sysmocom/ttcn3/defaults.conf<br>M sysmocom/ttcn3/jenkins-run.sh<br>A sysmocom/ttcn3/main.conf<br>D sysmocom/ttcn3/paths.conf<br>24 files changed, 290 insertions(+), 212 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/contrib/jenkins-run.sh b/contrib/jenkins-run.sh</span><br><span>index 89ff13f..1c31a9d 100755</span><br><span>--- a/contrib/jenkins-run.sh</span><br><span>+++ b/contrib/jenkins-run.sh</span><br><span>@@ -2,7 +2,7 @@</span><br><span> set -e -x</span><br><span> base="$PWD"</span><br><span> SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P) # this file's directory</span><br><span style="color: hsl(0, 100%, 40%);">-OSMO_GSM_TESTER_CONF=${OSMO_GSM_TESTER_CONF:-${SCRIPT_DIR}/../sysmocom}</span><br><span style="color: hsl(120, 100%, 40%);">+OSMO_GSM_TESTER_CONF=${OSMO_GSM_TESTER_CONF:-${SCRIPT_DIR}/../sysmocom/main.conf}</span><br><span> </span><br><span> time_start="$(date '+%F %T')"</span><br><span> </span><br><span>diff --git a/doc/examples/2g_osmocom/README.md b/doc/examples/2g_osmocom/README.md</span><br><span>index 47b0737..b85ee1f 100644</span><br><span>--- a/doc/examples/2g_osmocom/README.md</span><br><span>+++ b/doc/examples/2g_osmocom/README.md</span><br><span>@@ -1,13 +1,14 @@</span><br><span> This a sample 2G test suite configured and ready to use.</span><br><span> The only thing missing is a trial dir containing binaries.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-You can point osmo-gsm-tester.py at this config using the '-c $DIR' command line</span><br><span style="color: hsl(0, 100%, 40%);">-argument, where DIR is the directory path where this README file resides.</span><br><span style="color: hsl(120, 100%, 40%);">+You can point osmo-gsm-tester.py at this config using the '-c $DIR/main.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+command line argument, where DIR is the directory path where this README file</span><br><span style="color: hsl(120, 100%, 40%);">+resides.</span><br><span> </span><br><span> If you have your trial with binary tar archives in ~/my_trial</span><br><span> you can run the suite for example like this:</span><br><span> ```</span><br><span style="color: hsl(0, 100%, 40%);">-osmo-gsm-tester.py -c $DIR ~/my_trial</span><br><span style="color: hsl(120, 100%, 40%);">+osmo-gsm-tester.py -c $DIR/main.conf ~/my_trial</span><br><span> ```</span><br><span> </span><br><span> Alternatively you can setup this example as default config for your user by</span><br><span>@@ -19,4 +20,4 @@</span><br><span> </span><br><span> A ./state dir will be created to store the current osmo-gsm-tester state. If</span><br><span> you prefer not to write to $DIR, set up an own configuration pointing at a</span><br><span style="color: hsl(0, 100%, 40%);">-different path (see paths.conf: 'state_dir').</span><br><span style="color: hsl(120, 100%, 40%);">+different path (see main.conf: 'state_dir').</span><br><span>diff --git a/doc/examples/2g_osmocom/main.conf b/doc/examples/2g_osmocom/main.conf</span><br><span>new file mode 100644</span><br><span>index 0000000..b810519</span><br><span>--- /dev/null</span><br><span>+++ b/doc/examples/2g_osmocom/main.conf</span><br><span>@@ -0,0 +1,6 @@</span><br><span style="color: hsl(120, 100%, 40%);">+state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span style="color: hsl(120, 100%, 40%);">+suites_dir: './suites'</span><br><span style="color: hsl(120, 100%, 40%);">+scenarios_dir: './scenarios'</span><br><span style="color: hsl(120, 100%, 40%);">+default_suites_conf_path: './default-suites.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+defaults_conf_path: './defaults.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+resource_conf_path: './resources.conf'</span><br><span>diff --git a/doc/examples/2g_osmocom/paths.conf b/doc/examples/2g_osmocom/paths.conf</span><br><span>deleted file mode 100644</span><br><span>index 27c5818..0000000</span><br><span>--- a/doc/examples/2g_osmocom/paths.conf</span><br><span>+++ /dev/null</span><br><span>@@ -1,3 +0,0 @@</span><br><span style="color: hsl(0, 100%, 40%);">-state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span style="color: hsl(0, 100%, 40%);">-suites_dir: './suites'</span><br><span style="color: hsl(0, 100%, 40%);">-scenarios_dir: './scenarios'</span><br><span>diff --git a/doc/examples/4g_srsLTE/README.md b/doc/examples/4g_srsLTE/README.md</span><br><span>index b577035..09af755 100644</span><br><span>--- a/doc/examples/4g_srsLTE/README.md</span><br><span>+++ b/doc/examples/4g_srsLTE/README.md</span><br><span>@@ -1,13 +1,14 @@</span><br><span> This a sample 4G test suite configured and ready to use srsLTE stack.</span><br><span> The only thing missing is a trial dir containing binaries.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-You can point osmo-gsm-tester.py at this config using the '-c $DIR' command line</span><br><span style="color: hsl(0, 100%, 40%);">-argument, where DIR is the directory path where this README file resides.</span><br><span style="color: hsl(120, 100%, 40%);">+You can point osmo-gsm-tester.py at this config using the '-c $DIR/main.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+command line argument, where DIR is the directory path where this README file</span><br><span style="color: hsl(120, 100%, 40%);">+resides.</span><br><span> </span><br><span> If you have your trial with binary tar archives in ~/my_trial</span><br><span> you can run the suite for example like this:</span><br><span> ```</span><br><span style="color: hsl(0, 100%, 40%);">-osmo-gsm-tester.py -c $DIR ~/my_trial</span><br><span style="color: hsl(120, 100%, 40%);">+osmo-gsm-tester.py -c $DIR/main.conf ~/my_trial</span><br><span> ```</span><br><span> </span><br><span> Alternatively you can setup this example as default config for your user by</span><br><span>@@ -19,4 +20,4 @@</span><br><span> </span><br><span> A ./state dir will be created to store the current osmo-gsm-tester state. If</span><br><span> you prefer not to write to $DIR, set up an own configuration pointing at a</span><br><span style="color: hsl(0, 100%, 40%);">-different path (see paths.conf: 'state_dir').</span><br><span style="color: hsl(120, 100%, 40%);">+different path (see main.conf: 'state_dir').</span><br><span>diff --git a/doc/examples/4g_srsLTE/main.conf b/doc/examples/4g_srsLTE/main.conf</span><br><span>new file mode 100644</span><br><span>index 0000000..b810519</span><br><span>--- /dev/null</span><br><span>+++ b/doc/examples/4g_srsLTE/main.conf</span><br><span>@@ -0,0 +1,6 @@</span><br><span style="color: hsl(120, 100%, 40%);">+state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span style="color: hsl(120, 100%, 40%);">+suites_dir: './suites'</span><br><span style="color: hsl(120, 100%, 40%);">+scenarios_dir: './scenarios'</span><br><span style="color: hsl(120, 100%, 40%);">+default_suites_conf_path: './default-suites.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+defaults_conf_path: './defaults.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+resource_conf_path: './resources.conf'</span><br><span>diff --git a/doc/examples/4g_srsLTE/paths.conf b/doc/examples/4g_srsLTE/paths.conf</span><br><span>deleted file mode 100644</span><br><span>index 27c5818..0000000</span><br><span>--- a/doc/examples/4g_srsLTE/paths.conf</span><br><span>+++ /dev/null</span><br><span>@@ -1,3 +0,0 @@</span><br><span style="color: hsl(0, 100%, 40%);">-state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span style="color: hsl(0, 100%, 40%);">-suites_dir: './suites'</span><br><span style="color: hsl(0, 100%, 40%);">-scenarios_dir: './scenarios'</span><br><span>diff --git a/doc/manuals/chapters/config.adoc b/doc/manuals/chapters/config.adoc</span><br><span>index 3f4bde1..c302cd6 100644</span><br><span>--- a/doc/manuals/chapters/config.adoc</span><br><span>+++ b/doc/manuals/chapters/config.adoc</span><br><span>@@ -4,22 +4,29 @@</span><br><span> </span><br><span> All configuration attributes in {app-name} are stored and provided as YAML</span><br><span> files, which are handled internally mostly as sets of dictionaries, lists and</span><br><span style="color: hsl(0, 100%, 40%);">-scalars. Each of these configurations have a known format, which is called</span><br><span style="color: hsl(0, 100%, 40%);">-'schema'. Each provided configuration is validated against its 'schema' at parse</span><br><span style="color: hsl(0, 100%, 40%);">-time. Hence, 'schemas' can be seen as a namespace containing a structured tree</span><br><span style="color: hsl(0, 100%, 40%);">-of configuration attributes. Each attribute has a schema type assigned which</span><br><span style="color: hsl(0, 100%, 40%);">-constrains the type of value it can hold.</span><br><span style="color: hsl(120, 100%, 40%);">+scalars. Each of these configurations have a known format (set of keys and</span><br><span style="color: hsl(120, 100%, 40%);">+values), which is called 'schema'. Each provided configuration is validated</span><br><span style="color: hsl(120, 100%, 40%);">+against its 'schema' at parse time. Hence, 'schemas' can be seen as a namespace</span><br><span style="color: hsl(120, 100%, 40%);">+containing a structured tree of configuration attributes. Each attribute has a</span><br><span style="color: hsl(120, 100%, 40%);">+schema type assigned which constrains the type of value it can hold.</span><br><span> </span><br><span> There are several well-known schemas used across {app-name}, and they are</span><br><span> described in following sub-sections.</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+[[schema_main_cfg]]</span><br><span style="color: hsl(120, 100%, 40%);">+==== Schema 'main config'</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+This schema defines all the attributes available in {app-name} the main</span><br><span style="color: hsl(120, 100%, 40%);">+configuration file <<main_conf,main.conf>>, and it is used to validate it.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> [[schema_resources]]</span><br><span> ==== Schema 'resources'</span><br><span> </span><br><span> This schema defines all the attributes which can be assigned to</span><br><span> a _resource_, and it is used to validate the <<resources_conf,resources.conf>></span><br><span> file. Hence, the <<resources_conf,resources.conf>> contains a list of elements</span><br><span style="color: hsl(0, 100%, 40%);">-for each resource type.</span><br><span style="color: hsl(120, 100%, 40%);">+for each resource type. This schema is also used and extended by the</span><br><span style="color: hsl(120, 100%, 40%);">+<<schema_want,'want' schema>>.</span><br><span> </span><br><span> It is important to understand that the content in this schema refers to a list of</span><br><span> resources for each resource class. Since a list is ordered by definition, it</span><br><span>@@ -34,9 +41,10 @@</span><br><span> types are currently being treated as unordered sets, which mean combination of</span><br><span> filters or modifiers apply differently. In the future, it may be possible to</span><br><span> have both behaviors for scalar/simple types by using also the YAML 'set' type in</span><br><span style="color: hsl(0, 100%, 40%);">-{app-handle}.</span><br><span style="color: hsl(120, 100%, 40%);">+{app-name}.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-//TODO: update this list and use a table for each resource type</span><br><span style="color: hsl(120, 100%, 40%);">+//TODO: update this list and use a table for each resource type in its own object section</span><br><span style="color: hsl(120, 100%, 40%);">+////</span><br><span> These kinds of resources and their attributes are known:</span><br><span> </span><br><span> 'ip_address'::</span><br><span>@@ -127,6 +135,7 @@</span><br><span> - 'gprs'</span><br><span> - 'voice'</span><br><span> - 'ussd'</span><br><span style="color: hsl(120, 100%, 40%);">+////</span><br><span> </span><br><span> [[schema_want]]</span><br><span> ==== Schema 'want'</span><br><span>@@ -152,78 +161,109 @@</span><br><span> type: osmo-bts-sysmo</span><br><span> ----</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-[[schema_conf]]</span><br><span style="color: hsl(0, 100%, 40%);">-==== Schema 'conf'</span><br><span style="color: hsl(120, 100%, 40%);">+[[schema_config]]</span><br><span style="color: hsl(120, 100%, 40%);">+==== Schema 'config'</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-This schema is used by <<suite_conf,suite.conf>> and <<scenario_conf,scenario.conf>> files. It contains 3 main element sections:::</span><br><span style="color: hsl(0, 100%, 40%);">-[[schema_conf_sec_resources]]</span><br><span style="color: hsl(0, 100%, 40%);">-- Section 'resources': Contains a set of elements validated with <<schema_resources,resources>></span><br><span style="color: hsl(120, 100%, 40%);">+This schema defines all the attributes which can be used by object classes or</span><br><span style="color: hsl(120, 100%, 40%);">+tests during test execution. The main difference between this schema and the</span><br><span style="color: hsl(120, 100%, 40%);">+<<schema_resources,resources>> schema is that the former contains configuration</span><br><span style="color: hsl(120, 100%, 40%);">+to be applied globally for all objects being used, while the later applies</span><br><span style="color: hsl(120, 100%, 40%);">+attributes to a specific object in the list of allocated resources. This schema</span><br><span style="color: hsl(120, 100%, 40%);">+hence allows setting attributes for objects which are not allocated as resources</span><br><span style="color: hsl(120, 100%, 40%);">+and hence not directly accessible through scenarios, like a BSC or an iperf3</span><br><span style="color: hsl(120, 100%, 40%);">+client.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+This schema is built dynamically at runtime from content registered by:</span><br><span style="color: hsl(120, 100%, 40%);">+- object classes registering their own attributes</span><br><span style="color: hsl(120, 100%, 40%);">+- test suite registering their own attributes through <<suite_conf,suite.conf>></span><br><span style="color: hsl(120, 100%, 40%);">+ and tests being able to later retrieve them through 'testenv' API.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+[[schema_all]]</span><br><span style="color: hsl(120, 100%, 40%);">+==== Schema 'all'</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+This schema is basically an aggregated namespace for <<schema_want,want>> schema</span><br><span style="color: hsl(120, 100%, 40%);">+and <<schema_config,config>> schema, and is the one used by</span><br><span style="color: hsl(120, 100%, 40%);">+<<suite_conf,suite.conf>> and <<scenario_conf,scenario.conf>> files. It contains</span><br><span style="color: hsl(120, 100%, 40%);">+these main element sections:::</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+[[schema_all_sec_resources]]</span><br><span style="color: hsl(120, 100%, 40%);">+- Section 'resources': Contains a set of elements validated with <<schema_want,want>></span><br><span> schema. In <<suite_conf,suite.conf>> it is used to construct the list of</span><br><span> requested resources. In <<scenario_conf,scenario.conf>>, it is used to inject</span><br><span> attributes to the initial <<suite_conf,suite.conf>> _resources_ section and</span><br><span> hence further restrain it.</span><br><span style="color: hsl(0, 100%, 40%);">-[[schema_conf_sec_modifiers]]</span><br><span style="color: hsl(120, 100%, 40%);">+[[schema_all_sec_modifiers]]</span><br><span> - Section 'modifiers': Both in <<suite_conf,suite.conf>> and</span><br><span> <<scenario_conf,scenario.conf>>, values presented in here are injected into</span><br><span style="color: hsl(0, 100%, 40%);">- the content of the <<schema_conf_sec_resources,resources section>> after</span><br><span style="color: hsl(120, 100%, 40%);">+ the content of the <<schema_all_sec_resources,resources section>> after</span><br><span> _resource_ allocation, hereby overwriting attributes passed to the object</span><br><span> class instance managing the specific _resource_ (matches by resource type and</span><br><span> list position). Since it is combined with the content of</span><br><span style="color: hsl(0, 100%, 40%);">- <<schema_conf_sec_resources,resources section>>, it is clear that the</span><br><span style="color: hsl(0, 100%, 40%);">- <<schema_resources,resources schema>> is used to validate this content.</span><br><span style="color: hsl(0, 100%, 40%);">-[[schema_conf_sec_config]]</span><br><span style="color: hsl(0, 100%, 40%);">-- Section 'config': Contains configuration attributes for {app-name} classes which are</span><br><span style="color: hsl(0, 100%, 40%);">- not _resources_, and hence cannot be configured with <<schema_modifiers,modifiers>>.</span><br><span style="color: hsl(0, 100%, 40%);">- They can overwrite values provided in the <<defaults_conf,defaults.conf>> file.</span><br><span style="color: hsl(120, 100%, 40%);">+ <<schema_all_sec_resources,resources section>>, it is clear that the</span><br><span style="color: hsl(120, 100%, 40%);">+ <<schema_want,want schema>> is used to validate this content.</span><br><span style="color: hsl(120, 100%, 40%);">+[[schema_all_sec_config]]</span><br><span style="color: hsl(120, 100%, 40%);">+- Section 'config': Contains configuration attributes for {app-name} object</span><br><span style="color: hsl(120, 100%, 40%);">+ classes which are not _resources_, and hence cannot be configured with</span><br><span style="color: hsl(120, 100%, 40%);">+ <<schema_all_sec_modifiers,modifiers>>. They can overwrite values provided in the</span><br><span style="color: hsl(120, 100%, 40%);">+ <<defaults_conf,defaults.conf>> file. Content in this section follows the</span><br><span style="color: hsl(120, 100%, 40%);">+ <<schema_config,config>> schema.</span><br><span> </span><br><span> //TODO: defaults.timeout should be change in code to be config.test_timeout or similar</span><br><span> //TODO: 'config' should be split into its own schema and validate defaults.conf</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-[[config_paths]]</span><br><span style="color: hsl(0, 100%, 40%);">-=== Config Paths</span><br><span style="color: hsl(120, 100%, 40%);">+[[config]]</span><br><span style="color: hsl(120, 100%, 40%);">+=== Configuration files and directories</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-The osmo-gsm-tester looks for configuration files in various standard</span><br><span style="color: hsl(0, 100%, 40%);">-directories in this order:</span><br><span style="color: hsl(120, 100%, 40%);">+Find in below sub-sections all user-defined files and directories used by</span><br><span style="color: hsl(120, 100%, 40%);">+{app-name} to run tests on a given setup.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-- '.' (Current Working Directory)</span><br><span style="color: hsl(0, 100%, 40%);">-- '$HOME/.config/osmo-gsm-tester/'</span><br><span style="color: hsl(0, 100%, 40%);">-- '/usr/local/etc/osmo-gsm-tester/'</span><br><span style="color: hsl(0, 100%, 40%);">-- '/etc/osmo-gsm-tester/'</span><br><span style="color: hsl(120, 100%, 40%);">+[[config_main]]</span><br><span style="color: hsl(120, 100%, 40%);">+==== 'main.conf'</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-The config location can also be set through '-c' command line argument, which</span><br><span style="color: hsl(120, 100%, 40%);">+The main configuration file is basically a placeholder for {app-name} to find</span><br><span style="color: hsl(120, 100%, 40%);">+paths to all other files and directories used to operate and run tests.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+{app-name} looks for the main configuration file in various standard paths in</span><br><span style="color: hsl(120, 100%, 40%);">+this order:</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+- './main.conf' (Current Working Directory)</span><br><span style="color: hsl(120, 100%, 40%);">+- '$HOME/.config/osmo-gsm-tester/main.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+- '/usr/local/etc/osmo-gsm-tester/main.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+- '/etc/osmo-gsm-tester/main.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+The config file location can also be set through '-c' command line argument, which</span><br><span> then overrides the above locations.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-The osmo-gsm-tester expects to find the following configuration files in a</span><br><span style="color: hsl(0, 100%, 40%);">-configuration directory:</span><br><span style="color: hsl(120, 100%, 40%);">+{app-name} expects to find the following configuration settings in 'main.conf':</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-- <<paths_conf,paths.conf>></span><br><span style="color: hsl(0, 100%, 40%);">-- <<resource_conf,resources.conf>></span><br><span style="color: hsl(0, 100%, 40%);">-- <<default_suites_conf,default-suites.conf>> (optional)</span><br><span style="color: hsl(0, 100%, 40%);">-- <<defaults_conf,defaults.conf>> (optional)</span><br><span style="color: hsl(120, 100%, 40%);">+- 'state_dir': Path to <<state_dir,state_dir>> directory</span><br><span style="color: hsl(120, 100%, 40%);">+- 'suites_dir': Path to <<suites_dir,suites_dir>> directory</span><br><span style="color: hsl(120, 100%, 40%);">+- 'scenarios_dir': Path to <<scenarios_dir,scenarios_dir>> directory (optional)</span><br><span style="color: hsl(120, 100%, 40%);">+- 'default_suites_conf_path': Path to <<default_suites_conf,default-suites.conf>> file (optional)</span><br><span style="color: hsl(120, 100%, 40%);">+- 'defaults_conf_path': Path to <<defaults_conf,defaults.conf>> file (optional)</span><br><span style="color: hsl(120, 100%, 40%);">+- 'resource_conf_path': Path to <<resource_conf,resources.conf>> file (optional)</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-These are described in detail in the following sections.</span><br><span style="color: hsl(120, 100%, 40%);">+These are described in detail in the following sections. If no value is provided</span><br><span style="color: hsl(120, 100%, 40%);">+for a given setting, sane default paths are used: For 'state_dir',</span><br><span style="color: hsl(120, 100%, 40%);">+'/var/tmp/osmo-gsm-tester/state/' is used. All other files and directories are</span><br><span style="color: hsl(120, 100%, 40%);">+expected, by default, to be in the same directory as <<config_main,main.conf>></span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-[[paths_conf]]</span><br><span style="color: hsl(0, 100%, 40%);">-==== 'paths.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+IMPORTANT: Relative paths provided in 'main.conf' are parsed as being relative</span><br><span style="color: hsl(120, 100%, 40%);">+to the directory of that 'main.conf' file itself, and not relative to the CWD</span><br><span style="color: hsl(120, 100%, 40%);">+of the {app-name} process parsing it.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-The 'paths.conf' file defines where to store the global state (of reserved</span><br><span style="color: hsl(0, 100%, 40%);">-resources) and where to find suite and scenario definitions.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-Any relative paths found in a 'paths.conf' file are interpreted as relative to</span><br><span style="color: hsl(0, 100%, 40%);">-the directory of that 'paths.conf' file.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-There's not yet any well-known schema to validate this file contents since it</span><br><span style="color: hsl(0, 100%, 40%);">-has only 3 attributes.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-.Sample paths.conf file:</span><br><span style="color: hsl(120, 100%, 40%);">+.Sample main.conf file:</span><br><span> ----</span><br><span> state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span> suites_dir: '/usr/local/src/osmo-gsm-tester/suites'</span><br><span> scenarios_dir: './scenarios'</span><br><span style="color: hsl(120, 100%, 40%);">+default_suites_conf_path: './default-suites.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+defaults_conf_path: './defaults.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+resource_conf_path: './resources.conf'</span><br><span> ----</span><br><span> </span><br><span> [[state_dir]]</span><br><span style="color: hsl(0, 100%, 40%);">-===== 'state_dir'</span><br><span style="color: hsl(120, 100%, 40%);">+==== 'state_dir'</span><br><span> </span><br><span> It contains global or system-wide state for osmo-gsm-tester. In a typical state</span><br><span> dir you can find the following files:</span><br><span>@@ -256,7 +296,7 @@</span><br><span> resource allocation may not work as expected.</span><br><span> </span><br><span> [[suites_dir]]</span><br><span style="color: hsl(0, 100%, 40%);">-===== 'suites_dir'</span><br><span style="color: hsl(120, 100%, 40%);">+==== 'suites_dir'</span><br><span> </span><br><span> Suites contain a set of tests which are designed to be run together to test a</span><br><span> set of features given a specific set of resources. As a result, resources are</span><br><span>@@ -359,7 +399,7 @@</span><br><span> ----</span><br><span> </span><br><span> [[scenarios_dir]]</span><br><span style="color: hsl(0, 100%, 40%);">-===== 'scenarios_dir'</span><br><span style="color: hsl(120, 100%, 40%);">+==== 'scenarios_dir'</span><br><span> </span><br><span> This dir contains scenario configuration files.</span><br><span> </span><br><span>@@ -503,7 +543,7 @@</span><br><span> available (yet).</span><br><span> </span><br><span> [[default_suites_conf]]</span><br><span style="color: hsl(0, 100%, 40%);">-==== 'default-suites.conf' (optional)</span><br><span style="color: hsl(120, 100%, 40%);">+==== 'default-suites.conf'</span><br><span> </span><br><span> The 'default-suites.conf' file contains a YAML list of 'suite:scenario+scenario+...'</span><br><span> combination strings as defined by the 'osmo-gsm-tester.py -s' commandline</span><br><span>@@ -512,10 +552,10 @@</span><br><span> combinations is run in sequence.</span><br><span> </span><br><span> A suite name must match the name of a directory in the</span><br><span style="color: hsl(0, 100%, 40%);">-<<suites_dir,suites_dir/>> as defined by <<paths_conf,paths.conf>>.</span><br><span style="color: hsl(120, 100%, 40%);">+<<suites_dir,suites_dir/>> as defined by <<main_conf,main.conf>>.</span><br><span> </span><br><span> A scenario name must match the name of a configuration file in the</span><br><span style="color: hsl(0, 100%, 40%);">-<<scenarios_dir,scnearios_dir/>> as defined by <<paths_conf,paths.conf>></span><br><span style="color: hsl(120, 100%, 40%);">+<<scenarios_dir,scnearios_dir/>> as defined by <<main_conf,main.conf>></span><br><span> (optionally without the '.conf' suffix).</span><br><span> </span><br><span> .Sample 'default-suites.conf' file:</span><br><span>@@ -530,7 +570,7 @@</span><br><span> - voice:trx+dyn_ts</span><br><span> ----</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-==== 'defaults.conf' (optional)</span><br><span style="color: hsl(120, 100%, 40%);">+==== 'defaults.conf'</span><br><span> </span><br><span> In {app-name} object instances requested by the test and created by the suite</span><br><span> relate to a specific allocated resource. That's not always the case, and even if</span><br><span>@@ -613,16 +653,17 @@</span><br><span> All {app-name} related configuration for that environment is publicly available</span><br><span> in 'osmo-gsm-tester.git' itself:</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-- <<paths_conf,paths.conf>>: Available Available under 'example/', with its paths</span><br><span style="color: hsl(0, 100%, 40%);">- already configured to take required bits from inside the git repository.</span><br><span style="color: hsl(0, 100%, 40%);">-- <<suite_dir,suites_dir>>: Available under 'example/suites/'</span><br><span style="color: hsl(0, 100%, 40%);">-- <<scenarios_dir,scenarios_dir>>: Available under 'example/scenarios/'</span><br><span style="color: hsl(0, 100%, 40%);">-- <<resource_conf,resources.conf>>: Available under 'example/' as</span><br><span style="color: hsl(120, 100%, 40%);">+- <<main_conf,main.conf>>: Available Available under 'sysmocom/', with its paths</span><br><span style="color: hsl(120, 100%, 40%);">+ already configured to take required bits from inside the git repository directory.</span><br><span style="color: hsl(120, 100%, 40%);">+- <<suite_dir,suites_dir>>: Available under 'sysmocom/suites/'</span><br><span style="color: hsl(120, 100%, 40%);">+- <<scenarios_dir,scenarios_dir>>: Available under 'sysmocom/scenarios/'</span><br><span style="color: hsl(120, 100%, 40%);">+- <<resource_conf,resources.conf>>: Available under 'sysmocom/' as</span><br><span> 'resources.conf.prod' for Production setup and as 'resources.conf.rnd' for the</span><br><span> RnD setup. One must use a symbolic link to have it available as</span><br><span> 'resources.conf'.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-//TODO: resources.conf file path should be modifiable through paths.conf!</span><br><span style="color: hsl(120, 100%, 40%);">+There are also small sample setups under the 'doc/examples/' directory to</span><br><span style="color: hsl(120, 100%, 40%);">+showcase how to set up different types of networks.</span><br><span> </span><br><span> ==== Typical Invocations</span><br><span> </span><br><span>@@ -633,25 +674,34 @@</span><br><span> Examples for launching test trials:</span><br><span> </span><br><span> - Run the default suites (see <<default_suites_conf,default_suites.conf>>) on a</span><br><span style="color: hsl(0, 100%, 40%);">- given set of binaries:</span><br><span style="color: hsl(120, 100%, 40%);">+ given set of binaries from 'path/to/my-trial' with <<main_conf,main.conf>></span><br><span style="color: hsl(120, 100%, 40%);">+ available under a standard path:</span><br><span> </span><br><span> ----</span><br><span> osmo-gsm-tester.py path/to/my-trial</span><br><span> ----</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-- Run an explicit choice of 'suite:scenario' combinations:</span><br><span style="color: hsl(120, 100%, 40%);">+- Same as above, but run an explicit choice of 'suite:scenario' combinations:</span><br><span> </span><br><span> ----</span><br><span> osmo-gsm-tester.py path/to/my-trial -s sms:sysmo -s sms:trx -s sms:nanobts</span><br><span> ----</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-- Run one 'suite:scenario1+scenario2' combination, setting log level to 'debug'</span><br><span style="color: hsl(0, 100%, 40%);">- and enabling logging of full python tracebacks, and also only run just the</span><br><span style="color: hsl(0, 100%, 40%);">- 'mo_mt_sms.py' test from the suite, e.g. to investigate a test failure:</span><br><span style="color: hsl(120, 100%, 40%);">+- Same as above, but run one 'suite:scenario1+scenario2' combination, setting</span><br><span style="color: hsl(120, 100%, 40%);">+ log level to 'debug' and enabling logging of full python tracebacks, and also</span><br><span style="color: hsl(120, 100%, 40%);">+ only run just the 'mo_mt_sms.py' test from the suite, e.g. to investigate a</span><br><span style="color: hsl(120, 100%, 40%);">+ test failure:</span><br><span> </span><br><span> ----</span><br><span> osmo-gsm-tester.py path/to/my-trial -s sms:sysmo+foobar -l dbg -T -t mo_mt</span><br><span> ----</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+- Same as above, but tell {app-name} to read the 'main.conf' in specific</span><br><span style="color: hsl(120, 100%, 40%);">+ directory 'path/to/my/main.conf':</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+----</span><br><span style="color: hsl(120, 100%, 40%);">+osmo-gsm-tester.py -c path/to/my/main.conf path/to/my-trial -s sms:sysmo+foobar -l dbg -T -t mo_mt</span><br><span style="color: hsl(120, 100%, 40%);">+----</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> A test script may also be run step-by-step in a python debugger, see</span><br><span> <<debugging>>.</span><br><span>diff --git a/selftest/resource_test/resource_test.ok b/selftest/resource_test/resource_test.ok</span><br><span>index bbdbc5b..8a0b52f 100644</span><br><span>--- a/selftest/resource_test/resource_test.ok</span><br><span>+++ b/selftest/resource_test/resource_test.ok</span><br><span>@@ -9,12 +9,14 @@</span><br><span> ok, caused exception RuntimeError: Refusing to drop a list of resources from itself. This is probably a bug where a list of Resources() should have been copied but is passed as-is. use Resources.clear() instead.</span><br><span> - test removing a Resources list from one with the same list in it</span><br><span> - test resources config and state dir:</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found config file paths.conf as [PATH]/selftest/resource_test/conf/paths.conf in [PATH]/selftest/resource_test/conf which is [PATH]/selftest/resource_test/conf</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: [PATH]/selftest/resource_test/conf/paths.conf: relative path ./test_work/state_dir is [PATH]/selftest/resource_test/conf/test_work/state_dir</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: [PATH]/selftest/resource_test/conf/paths.conf: relative path ./suite_test is [PATH]/selftest/resource_test/conf/suite_test</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path state_dir as [PATH]/selftest/resource_test/conf/test_work/state_dir</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found config file resources.conf as [PATH]/selftest/resource_test/conf/resources.conf in [PATH]/selftest/resource_test/conf which is [PATH]/selftest/resource_test/conf</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found path state_dir as [PATH]/selftest/resource_test/conf/test_work/state_dir</span><br><span style="color: hsl(120, 100%, 40%);">+cnf -: DBG: Found main configuration file in [PATH]/selftest/resource_test/conf/paths.conf which is [PATH]/selftest/resource_test/conf/paths.conf</span><br><span style="color: hsl(120, 100%, 40%);">+cnf -: DBG: MAIN CONFIG:</span><br><span style="color: hsl(120, 100%, 40%);">+{'default_suites_conf_path': '[PATH]/selftest/resource_test/conf/default-suites.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'defaults_conf_path': '[PATH]/selftest/resource_test/conf/defaults.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'resource_conf_path': '[PATH]/selftest/resource_test/conf/resources.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'scenarios_dir': '[PATH]/selftest/resource_test/conf/scenarios',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'state_dir': '[PATH]/selftest/resource_test/conf/test_work/state_dir',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'suites_dir': '[PATH]/selftest/resource_test/conf/suite_test'}</span><br><span> *** all resources:</span><br><span> {'arfcn': [{'_hash': 'e620569450f8259b3f0212ec19c285dd07df063c',</span><br><span> 'arfcn': '512',</span><br><span>diff --git a/selftest/resource_test/resource_test.py b/selftest/resource_test/resource_test.py</span><br><span>index b74ba2a..f18aa73 100755</span><br><span>--- a/selftest/resource_test/resource_test.py</span><br><span>+++ b/selftest/resource_test/resource_test.py</span><br><span>@@ -13,7 +13,7 @@</span><br><span> workdir = util.get_tempdir()</span><br><span> </span><br><span> # override config locations to make sure we use only the test conf</span><br><span style="color: hsl(0, 100%, 40%);">-config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'conf')</span><br><span style="color: hsl(120, 100%, 40%);">+config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'conf', 'paths.conf')</span><br><span> </span><br><span> log.get_process_id = lambda: '123-1490837279'</span><br><span> </span><br><span>diff --git a/selftest/scenario_test/scenario_test.ok b/selftest/scenario_test/scenario_test.ok</span><br><span>index 7fe2049..e37ef57 100644</span><br><span>--- a/selftest/scenario_test/scenario_test.ok</span><br><span>+++ b/selftest/scenario_test/scenario_test.ok</span><br><span>@@ -1,32 +1,28 @@</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found config file paths.conf as [PATH]/selftest/scenario_test/paths.conf in [PATH]/selftest/scenario_test which is [PATH]/selftest/scenario_test</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: [PATH]/selftest/scenario_test/paths.conf: relative path . is [PATH]/selftest/scenario_test</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: [PATH]/selftest/scenario_test/paths.conf: relative path ./test_work/state_dir is [PATH]/selftest/scenario_test/test_work/state_dir</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: [PATH]/selftest/scenario_test/paths.conf: relative path . is [PATH]/selftest/scenario_test</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span style="color: hsl(120, 100%, 40%);">+cnf -: DBG: Found main configuration file in [PATH]/selftest/scenario_test/paths.conf which is [PATH]/selftest/scenario_test/paths.conf</span><br><span style="color: hsl(120, 100%, 40%);">+cnf -: DBG: MAIN CONFIG:</span><br><span style="color: hsl(120, 100%, 40%);">+{'default_suites_conf_path': '[PATH]/selftest/scenario_test/default-suites.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'defaults_conf_path': '[PATH]/selftest/scenario_test/defaults.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'resource_conf_path': '[PATH]/selftest/scenario_test/resources.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'scenarios_dir': '[PATH]/selftest/scenario_test',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'state_dir': '[PATH]/selftest/scenario_test/test_work/state_dir',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'suites_dir': '[PATH]/selftest/scenario_test'}</span><br><span> scenario_case_01.conf</span><br><span> {'anotherlist': ['4', '0'],</span><br><span> 'foobar': 'True',</span><br><span> 'somelist': [{'somelistitem': 'firststring'},</span><br><span> {'somelistitem': 'secondstring'},</span><br><span> {'somelistitem': 'thirdstring'}]}</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> scenario_case_01.conf</span><br><span> {'anotherlist': ['4', '0'],</span><br><span> 'foobar': 'True',</span><br><span> 'somelist': [{'somelistitem': 'firststring'},</span><br><span> {'somelistitem': 'secondstring'},</span><br><span> {'somelistitem': 'thirdstring'}]}</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> OK: expected RuntimeError: No such scenario file: '[PATH]/selftest/scenario_test/scenario_case_01@.conf' (nor scenario_case_01@.conf)</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> OK: expected ValueError</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> OK: expected ValueError</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> OK: expected RuntimeError: No such scenario file: '[PATH]/selftest/scenario_test/scenario_case_03.conf'</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> OK: expected RuntimeError: No such scenario file: '[PATH]/selftest/scenario_test/scenario_case_03.conf'</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> tst scenario_case_03@heyho,1,yes.conf: DBG: {param_dict={param1='heyho', param2='1', param3='yes'}}</span><br><span> scenario_case_03@heyho,1,yes.conf</span><br><span> {'anotherlist': ['1', '0'],</span><br><span>@@ -34,7 +30,6 @@</span><br><span> 'somelist': [{'somelistitem': 'firststring'},</span><br><span> {'somelistitem': 'heyho'},</span><br><span> {'somelistitem': 'thirdstring'}]}</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> tst scenario_case_03@heyho,1,yes.conf: DBG: {param_dict={param1='heyho', param2='1', param3='yes'}}</span><br><span> scenario_case_03@heyho,1,yes.conf</span><br><span> {'anotherlist': ['1', '0'],</span><br><span>@@ -42,17 +37,13 @@</span><br><span> 'somelist': [{'somelistitem': 'firststring'},</span><br><span> {'somelistitem': 'heyho'},</span><br><span> {'somelistitem': 'thirdstring'}]}</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> tst scenario_case_03@heyho,1.conf: DBG: {param_dict={param1='heyho', param2='1'}}</span><br><span> OK: expected NameError: Undefined</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> tst scenario_case_03@heyho,1.conf: DBG: {param_dict={param1='heyho', param2='1'}}</span><br><span> OK: expected NameError: Undefined</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> tst scenario_case_03@specific.conf: DBG: {param_dict={param1='specific'}}</span><br><span> scenario_case_03@specific.conf</span><br><span> {'somelist': [{'somelistitem': 'specific'}]}</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path scenarios_dir as [PATH]/selftest/scenario_test</span><br><span> tst scenario_case_03@specific.conf: DBG: {param_dict={param1='specific'}}</span><br><span> scenario_case_03@specific.conf</span><br><span> {'somelist': [{'somelistitem': 'specific'}]}</span><br><span>diff --git a/selftest/scenario_test/scenario_test.py b/selftest/scenario_test/scenario_test.py</span><br><span>index f5f42f7..15f8983 100755</span><br><span>--- a/selftest/scenario_test/scenario_test.py</span><br><span>+++ b/selftest/scenario_test/scenario_test.py</span><br><span>@@ -18,7 +18,7 @@</span><br><span> 'foobar' : schema.BOOL_STR,</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-config.override_conf = os.path.join(os.path.dirname(sys.argv[0]))</span><br><span style="color: hsl(120, 100%, 40%);">+config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'paths.conf')</span><br><span> </span><br><span> def print_scenario(sc):</span><br><span> # we use copy() to be able to get the dictionary in super class of Scenario:</span><br><span>diff --git a/selftest/suite_test/suite_test.ok b/selftest/suite_test/suite_test.ok</span><br><span>index fa38605..fb26a06 100644</span><br><span>--- a/selftest/suite_test/suite_test.ok</span><br><span>+++ b/selftest/suite_test/suite_test.ok</span><br><span>@@ -1,15 +1,17 @@</span><br><span> - non-existing suite dir</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found config file paths.conf as [PATH]/selftest/suite_test/paths.conf in [PATH]/selftest/suite_test which is [PATH]/selftest/suite_test</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: [PATH]/selftest/suite_test/paths.conf: relative path ./test_work/state_dir is [PATH]/selftest/suite_test/test_work/state_dir</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: [PATH]/selftest/suite_test/paths.conf: relative path . is [PATH]/selftest/suite_test</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path suites_dir as [PATH]/selftest/suite_test</span><br><span style="color: hsl(120, 100%, 40%);">+cnf -: DBG: Found main configuration file in [PATH]/selftest/suite_test/paths.conf which is [PATH]/selftest/suite_test/paths.conf</span><br><span style="color: hsl(120, 100%, 40%);">+cnf -: DBG: MAIN CONFIG:</span><br><span style="color: hsl(120, 100%, 40%);">+{'default_suites_conf_path': '[PATH]/selftest/suite_test/default-suites.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'defaults_conf_path': '[PATH]/selftest/suite_test/defaults.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'resource_conf_path': '[PATH]/selftest/suite_test/resources.conf',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'scenarios_dir': '[PATH]/selftest/suite_test/scenarios',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'state_dir': '[PATH]/selftest/suite_test/test_work/state_dir',</span><br><span style="color: hsl(120, 100%, 40%);">+ 'suites_dir': '[PATH]/selftest/suite_test'}</span><br><span> --- -: ERR: RuntimeError: Suite not found: 'does_not_exist' in [PATH]/selftest/suite_test</span><br><span> - no suite.conf</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path suites_dir as [PATH]/selftest/suite_test</span><br><span> cnf empty_dir: DBG: reading suite.conf</span><br><span> cnf [PATH]/selftest/suite_test/empty_dir/suite.conf: ERR: FileNotFoundError: [Errno 2] No such file or directory: '[PATH]/selftest/suite_test/empty_dir/suite.conf' [empty_dir↪[PATH]/selftest/suite_test/empty_dir/suite.conf]</span><br><span> - valid suite dir</span><br><span style="color: hsl(0, 100%, 40%);">-cnf -: DBG: Found path suites_dir as [PATH]/selftest/suite_test</span><br><span> cnf test_suite: DBG: reading suite.conf</span><br><span> defaults:</span><br><span> timeout: 60s</span><br><span>@@ -25,8 +27,6 @@</span><br><span> - times: '2'</span><br><span> </span><br><span> - run hello world test</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found config file resources.conf as [PATH]/selftest/suite_test/resources.conf in [PATH]/selftest/suite_test which is [PATH]/selftest/suite_test</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found path state_dir as [PATH]/selftest/suite_test/test_work/state_dir</span><br><span> </span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite</span><br><span>@@ -179,8 +179,6 @@</span><br><span> FAIL: test_fail_raise.py (N.N sec) ExpectedFail: This failure is expected</span><br><span> skip: test_suite_params.py</span><br><span> - test with half empty scenario</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found config file resources.conf as [PATH]/selftest/suite_test/resources.conf in [PATH]/selftest/suite_test which is [PATH]/selftest/suite_test [config.py:[LINENR]]</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found path state_dir as [PATH]/selftest/suite_test/test_work/state_dir [config.py:[LINENR]]</span><br><span> </span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite</span><br><span>@@ -267,8 +265,6 @@</span><br><span> skip: test_fail_raise.py</span><br><span> skip: test_suite_params.py</span><br><span> - test with scenario</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found config file resources.conf as [PATH]/selftest/suite_test/resources.conf in [PATH]/selftest/suite_test which is [PATH]/selftest/suite_test [config.py:[LINENR]]</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found path state_dir as [PATH]/selftest/suite_test/test_work/state_dir [config.py:[LINENR]]</span><br><span> </span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite</span><br><span>@@ -355,8 +351,6 @@</span><br><span> skip: test_fail_raise.py</span><br><span> skip: test_suite_params.py</span><br><span> - test with scenario and modifiers</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found config file resources.conf as [PATH]/selftest/suite_test/resources.conf in [PATH]/selftest/suite_test which is [PATH]/selftest/suite_test [config.py:[LINENR]]</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found path state_dir as [PATH]/selftest/suite_test/test_work/state_dir [config.py:[LINENR]]</span><br><span> tst test_suite: reserving resources in [PATH]/selftest/suite_test/test_work/state_dir ... [suite.py:[LINENR]]</span><br><span> tst test_suite: DBG: {combining='resources'} [suite.py:[LINENR]]</span><br><span> tst {combining_scenarios='resources'}: DBG: {definition_conf={bts=[{'label': 'sysmoCell 5000'}, {'label': 'sysmoCell 5000'}, {'type': 'sysmo'}], ip_address=[{}], modem=[{}, {}]}} [test_suite↪{combining_scenarios='resources'}] [suite.py:[LINENR]]</span><br><span>@@ -489,8 +483,6 @@</span><br><span> skip: test_fail_raise.py</span><br><span> skip: test_suite_params.py</span><br><span> - test with suite-specific config</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found config file resources.conf as [PATH]/selftest/suite_test/resources.conf in [PATH]/selftest/suite_test which is [PATH]/selftest/suite_test [config.py:[LINENR]]</span><br><span style="color: hsl(0, 100%, 40%);">-cnf ResourcesPool: DBG: Found path state_dir as [PATH]/selftest/suite_test/test_work/state_dir [config.py:[LINENR]]</span><br><span> tst test_suite: reserving resources in [PATH]/selftest/suite_test/test_work/state_dir ... [suite.py:[LINENR]]</span><br><span> tst test_suite: DBG: {combining='resources'} [suite.py:[LINENR]]</span><br><span> tst {combining_scenarios='resources'}: DBG: {definition_conf={bts=[{'label': 'sysmoCell 5000'}, {'label': 'sysmoCell 5000'}, {'type': 'sysmo'}], ip_address=[{}], modem=[{}, {}]}} [test_suite↪{combining_scenarios='resources'}] [suite.py:[LINENR]]</span><br><span>diff --git a/selftest/suite_test/suite_test.py b/selftest/suite_test/suite_test.py</span><br><span>index a096027..4b32439 100755</span><br><span>--- a/selftest/suite_test/suite_test.py</span><br><span>+++ b/selftest/suite_test/suite_test.py</span><br><span>@@ -11,7 +11,7 @@</span><br><span> from osmo_gsm_tester.core import suite</span><br><span> from osmo_gsm_tester.core.schema import generate_schemas, get_all_schema</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-config.override_conf = os.path.join(os.path.dirname(sys.argv[0]))</span><br><span style="color: hsl(120, 100%, 40%);">+config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'paths.conf')</span><br><span> </span><br><span> example_trial_dir = os.path.join('test_trial_tmp')</span><br><span> </span><br><span>diff --git a/src/osmo-gsm-tester.py b/src/osmo-gsm-tester.py</span><br><span>index fb5574b..204b1c7 100755</span><br><span>--- a/src/osmo-gsm-tester.py</span><br><span>+++ b/src/osmo-gsm-tester.py</span><br><span>@@ -24,9 +24,9 @@</span><br><span> </span><br><span> Examples:</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-./osmo-gsm-tester.py -c doc/examples/2g_osmocom/ ~/my_trial_package/ -s osmo_trx</span><br><span style="color: hsl(0, 100%, 40%);">-./osmo-gsm-tester.py -c doc/examples/2g_osmocom/ ~/my_trial_package/ -s sms_tests:dyn_ts+eu_band+bts_sysmo</span><br><span style="color: hsl(0, 100%, 40%);">-./osmo-gsm-tester.py -c sysmocom/ ~/my_trial_package/ -s sms_tests/mo_mt_sms:bts_trx</span><br><span style="color: hsl(120, 100%, 40%);">+./osmo-gsm-tester.py -c doc/examples/2g_osmocom/main.conf ~/my_trial_package/ -s osmo_trx</span><br><span style="color: hsl(120, 100%, 40%);">+./osmo-gsm-tester.py -c doc/examples/2g_osmocom/main.conf ~/my_trial_package/ -s sms_tests:dyn_ts+eu_band+bts_sysmo</span><br><span style="color: hsl(120, 100%, 40%);">+./osmo-gsm-tester.py -c sysmocom/main.conf ~/my_trial_package/ -s sms_tests/mo_mt_sms:bts_trx</span><br><span> </span><br><span> (The names for test suites and scenarios used in these examples must be defined</span><br><span> by the osmo-gsm-tester configuration.)</span><br><span>@@ -54,8 +54,9 @@</span><br><span> A test run thus needs to define:</span><br><span> * A trial package containing built binaries</span><br><span> * A set of test suites, each with its combinations of scenarios</span><br><span style="color: hsl(0, 100%, 40%);">-* A configuration directory specifying sets of resources, default configurations</span><br><span style="color: hsl(0, 100%, 40%);">- and paths on where to find suites, scenarios, etc.</span><br><span style="color: hsl(120, 100%, 40%);">+* A main configuration file specifying paths to other files containing sets of</span><br><span style="color: hsl(120, 100%, 40%);">+ resources, default configurations and paths on where to find suites,</span><br><span style="color: hsl(120, 100%, 40%);">+ scenarios, etc.</span><br><span> </span><br><span> If no combination of suites and scenarios is provided, the default list of</span><br><span> suites will be run as defined in the osmo-gsm-tester configuration.</span><br><span>@@ -101,8 +102,8 @@</span><br><span> # is easiest to maintain.</span><br><span> parser.add_argument('-V', '--version', action='store_true',</span><br><span> help='Show version')</span><br><span style="color: hsl(0, 100%, 40%);">- parser.add_argument('-c', '--conf-dir', dest='conf_dir',</span><br><span style="color: hsl(0, 100%, 40%);">- help='''Specify configuration directory path (containing paths.conf)''')</span><br><span style="color: hsl(120, 100%, 40%);">+ parser.add_argument('-c', '--conf-path', dest='conf_path',</span><br><span style="color: hsl(120, 100%, 40%);">+ help='''Specify main configuration file path''')</span><br><span> parser.add_argument('trial_package',</span><br><span> help='Directory containing binaries to test')</span><br><span> parser.add_argument('-s', '--suite-scenario', dest='suite_scenario', action='append',</span><br><span>@@ -140,18 +141,18 @@</span><br><span> log.style_change(trace=True)</span><br><span> if args.source:</span><br><span> log.style_change(src=True)</span><br><span style="color: hsl(0, 100%, 40%);">- if args.conf_dir:</span><br><span style="color: hsl(0, 100%, 40%);">- config.override_conf = args.conf_dir</span><br><span style="color: hsl(120, 100%, 40%);">+ if args.conf_path:</span><br><span style="color: hsl(120, 100%, 40%);">+ config.override_conf = args.conf_path</span><br><span> </span><br><span> combination_strs = list(args.suite_scenario or [])</span><br><span> </span><br><span> if not combination_strs:</span><br><span style="color: hsl(0, 100%, 40%);">- combination_strs = config.read_config_file(config.DEFAULT_SUITES_CONF, if_missing_return=[])</span><br><span style="color: hsl(120, 100%, 40%);">+ combination_strs = config.read_config_file(config.CFG_DEFAULT_SUITES_CONF, if_missing_return=[])</span><br><span> </span><br><span> if combination_strs:</span><br><span> print('Running default suites:\n ' + ('\n '.join(combination_strs)))</span><br><span> else:</span><br><span style="color: hsl(0, 100%, 40%);">- print('No default suites configured (%r)' % config.DEFAULT_SUITES_CONF)</span><br><span style="color: hsl(120, 100%, 40%);">+ print('Failed to load default suites (%r)' % config.get_main_config_value(config.DEFAULT_SUITES_CONF, fail_if_missing=False))</span><br><span> </span><br><span> </span><br><span> if not combination_strs:</span><br><span>diff --git a/src/osmo_gsm_tester/core/config.py b/src/osmo_gsm_tester/core/config.py</span><br><span>index 88e522d..ea16e33 100644</span><br><span>--- a/src/osmo_gsm_tester/core/config.py</span><br><span>+++ b/src/osmo_gsm_tester/core/config.py</span><br><span>@@ -1,8 +1,9 @@</span><br><span> # osmo_gsm_tester: read and manage config files and global config</span><br><span> #</span><br><span style="color: hsl(0, 100%, 40%);">-# Copyright (C) 2016-2017 by sysmocom - s.f.m.c. GmbH</span><br><span style="color: hsl(120, 100%, 40%);">+# Copyright (C) 2016-2020 by sysmocom - s.f.m.c. GmbH</span><br><span> #</span><br><span> # Author: Neels Hofmeyr <neels@hofmeyr.de></span><br><span style="color: hsl(120, 100%, 40%);">+# Author: Pau Espin Pedrol <pespin@sysmocom.de></span><br><span> #</span><br><span> # This program is free software: you can redistribute it and/or modify</span><br><span> # it under the terms of the GNU General Public License as</span><br><span>@@ -53,6 +54,7 @@</span><br><span> import yaml</span><br><span> import os</span><br><span> import copy</span><br><span style="color: hsl(120, 100%, 40%);">+import pprint</span><br><span> </span><br><span> from . import log, util, template</span><br><span> from . import schema</span><br><span>@@ -60,94 +62,122 @@</span><br><span> </span><br><span> override_conf = None</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-DEFAULT_CONFIG_LOCATIONS = [</span><br><span style="color: hsl(0, 100%, 40%);">- '.',</span><br><span style="color: hsl(0, 100%, 40%);">- os.path.join(os.getenv('HOME'), '.config', 'osmo-gsm-tester'),</span><br><span style="color: hsl(0, 100%, 40%);">- '/usr/local/etc/osmo-gsm-tester',</span><br><span style="color: hsl(0, 100%, 40%);">- '/etc/osmo-gsm-tester'</span><br><span style="color: hsl(0, 100%, 40%);">- ]</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-PATHS_CONF = 'paths.conf'</span><br><span style="color: hsl(0, 100%, 40%);">-DEFAULT_SUITES_CONF = 'default-suites.conf'</span><br><span style="color: hsl(0, 100%, 40%);">-DEFAULTS_CONF = 'defaults.conf'</span><br><span style="color: hsl(0, 100%, 40%);">-RESOURCES_CONF = 'resources.conf'</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-PATH_STATE_DIR = 'state_dir'</span><br><span style="color: hsl(0, 100%, 40%);">-PATH_SUITES_DIR = 'suites_dir'</span><br><span style="color: hsl(0, 100%, 40%);">-PATH_SCENARIOS_DIR = 'scenarios_dir'</span><br><span style="color: hsl(0, 100%, 40%);">-PATHS_SCHEMA = {</span><br><span style="color: hsl(0, 100%, 40%);">- PATH_STATE_DIR: schema.STR,</span><br><span style="color: hsl(0, 100%, 40%);">- PATH_SUITES_DIR: schema.STR,</span><br><span style="color: hsl(0, 100%, 40%);">- PATH_SCENARIOS_DIR: schema.STR,</span><br><span style="color: hsl(120, 100%, 40%);">+CFG_STATE_DIR = 'state_dir'</span><br><span style="color: hsl(120, 100%, 40%);">+CFG_SUITES_DIR = 'suites_dir'</span><br><span style="color: hsl(120, 100%, 40%);">+CFG_SCENARIOS_DIR = 'scenarios_dir'</span><br><span style="color: hsl(120, 100%, 40%);">+CFG_DEFAULT_SUITES_CONF = 'default_suites_conf_path'</span><br><span style="color: hsl(120, 100%, 40%);">+CFG_DEFAULTS_CONF = 'defaults_conf_path'</span><br><span style="color: hsl(120, 100%, 40%);">+CFG_RESOURCES_CONF = 'resource_conf_path'</span><br><span style="color: hsl(120, 100%, 40%);">+MAIN_CONFIG_SCHEMA = {</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_STATE_DIR: schema.STR,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_SUITES_DIR: schema.STR,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_SCENARIOS_DIR: schema.STR,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_DEFAULT_SUITES_CONF: schema.STR,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_DEFAULTS_CONF: schema.STR,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_RESOURCES_CONF: schema.STR,</span><br><span> }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-PATHS_TEMPDIR_STR = '$TEMPDIR'</span><br><span style="color: hsl(120, 100%, 40%);">+DF_CFG_STATE_DIR = '/var/tmp/osmo-gsm-tester/state/'</span><br><span style="color: hsl(120, 100%, 40%);">+DF_CFG_SUITES_DIR = './suites'</span><br><span style="color: hsl(120, 100%, 40%);">+DF_CFG_SCENARIOS_DIR = './scenarios'</span><br><span style="color: hsl(120, 100%, 40%);">+DF_CFG_DEFAULT_SUITES_CONF = './default-suites.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+DF_CFG_DEFAULTS_CONF = './defaults.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+DF_CFG_RESOURCES_CONF = './resources.conf'</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-PATHS = None</span><br><span style="color: hsl(120, 100%, 40%);">+DEFAULT_CONFIG_FILENAME = 'main.conf'</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-def _get_config_file(basename, fail_if_missing=True):</span><br><span style="color: hsl(120, 100%, 40%);">+DEFAULT_CONFIG_LOCATIONS = [</span><br><span style="color: hsl(120, 100%, 40%);">+ '.',</span><br><span style="color: hsl(120, 100%, 40%);">+ os.path.join(os.getenv('HOME'), '.config', 'osmo-gsm-tester', DEFAULT_CONFIG_FILENAME),</span><br><span style="color: hsl(120, 100%, 40%);">+ os.path.join('/usr/local/etc/osmo-gsm-tester', DEFAULT_CONFIG_FILENAME),</span><br><span style="color: hsl(120, 100%, 40%);">+ os.path.join('/etc/osmo-gsm-tester', DEFAULT_CONFIG_FILENAME)</span><br><span style="color: hsl(120, 100%, 40%);">+ ]</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+MAIN_CONFIG = None</span><br><span style="color: hsl(120, 100%, 40%);">+MAIN_CONFIG_PATH = None</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def _find_main_config_path():</span><br><span> if override_conf:</span><br><span> locations = [ override_conf ]</span><br><span> elif os.getenv('OSMO_GSM_TESTER_CONF'):</span><br><span> ENV_CONF = os.getenv('OSMO_GSM_TESTER_CONF')</span><br><span style="color: hsl(0, 100%, 40%);">- log.err('Using environment variable OSMO_GSM_TESTER_CONF=%s is deprecated. Rather use -c command line argument!' % ENV_CONF)</span><br><span style="color: hsl(0, 100%, 40%);">- locations = [ ENV_CONF ]</span><br><span style="color: hsl(120, 100%, 40%);">+ log.err('Using environment variable OSMO_GSM_TESTER_CONF=%s(/paths.conf) is deprecated. Rather use -c command line argument!' % ENV_CONF)</span><br><span style="color: hsl(120, 100%, 40%);">+ locations = [ ENV_CONF + 'paths.conf' ] # directory is expected in OSMO_GSM_TESTER_CONF, bakcward compatibility</span><br><span> else:</span><br><span> locations = DEFAULT_CONFIG_LOCATIONS</span><br><span> </span><br><span> for l in locations:</span><br><span> real_l = os.path.realpath(l)</span><br><span style="color: hsl(0, 100%, 40%);">- p = os.path.realpath(os.path.join(real_l, basename))</span><br><span style="color: hsl(0, 100%, 40%);">- if os.path.isfile(p):</span><br><span style="color: hsl(0, 100%, 40%);">- log.dbg('Found config file', basename, 'as', p, 'in', l, 'which is', real_l, _category=log.C_CNF)</span><br><span style="color: hsl(0, 100%, 40%);">- return (p, real_l)</span><br><span style="color: hsl(0, 100%, 40%);">- if not fail_if_missing:</span><br><span style="color: hsl(0, 100%, 40%);">- return None, None</span><br><span style="color: hsl(0, 100%, 40%);">- raise RuntimeError('configuration file not found: %r in %r' % (basename,</span><br><span style="color: hsl(0, 100%, 40%);">- [os.path.abspath(p) for p in locations]))</span><br><span style="color: hsl(120, 100%, 40%);">+ if os.path.isfile(real_l):</span><br><span style="color: hsl(120, 100%, 40%);">+ log.dbg('Found main configuration file in ', l, 'which is', real_l, _category=log.C_CNF)</span><br><span style="color: hsl(120, 100%, 40%);">+ return real_l</span><br><span style="color: hsl(120, 100%, 40%);">+ raise RuntimeError('Main configuration file not found in %r' % ([l for l in locations]))</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-def get_config_file(basename, fail_if_missing=True):</span><br><span style="color: hsl(0, 100%, 40%);">- path, found_in = _get_config_file(basename, fail_if_missing)</span><br><span style="color: hsl(120, 100%, 40%);">+def _get_main_config_path():</span><br><span style="color: hsl(120, 100%, 40%);">+ global MAIN_CONFIG_PATH</span><br><span style="color: hsl(120, 100%, 40%);">+ if MAIN_CONFIG_PATH is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ MAIN_CONFIG_PATH = _find_main_config_path()</span><br><span style="color: hsl(120, 100%, 40%);">+ return MAIN_CONFIG_PATH</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def main_config_path_to_abspath(path):</span><br><span style="color: hsl(120, 100%, 40%);">+ 'Relative files in main config are relative towards the config file, not towards $CWD'</span><br><span style="color: hsl(120, 100%, 40%);">+ if not path.startswith(os.pathsep):</span><br><span style="color: hsl(120, 100%, 40%);">+ return os.path.realpath(os.path.join(os.path.dirname(_get_main_config_path()), path))</span><br><span> return path</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-def read_config_file(basename, validation_schema=None, if_missing_return=False):</span><br><span style="color: hsl(120, 100%, 40%);">+def _get_main_config():</span><br><span style="color: hsl(120, 100%, 40%);">+ global MAIN_CONFIG</span><br><span style="color: hsl(120, 100%, 40%);">+ if MAIN_CONFIG is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ cfg = read(_get_main_config_path(), MAIN_CONFIG_SCHEMA)</span><br><span style="color: hsl(120, 100%, 40%);">+ MAIN_CONFIG = {</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_STATE_DIR: DF_CFG_STATE_DIR,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_SUITES_DIR: DF_CFG_SUITES_DIR,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_SCENARIOS_DIR: DF_CFG_SCENARIOS_DIR,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_DEFAULT_SUITES_CONF: DF_CFG_DEFAULT_SUITES_CONF,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_DEFAULTS_CONF: DF_CFG_DEFAULTS_CONF,</span><br><span style="color: hsl(120, 100%, 40%);">+ CFG_RESOURCES_CONF: DF_CFG_RESOURCES_CONF,</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ overlay(MAIN_CONFIG, cfg)</span><br><span style="color: hsl(120, 100%, 40%);">+ for key, path in sorted(MAIN_CONFIG.items()):</span><br><span style="color: hsl(120, 100%, 40%);">+ MAIN_CONFIG[key] = main_config_path_to_abspath(path)</span><br><span style="color: hsl(120, 100%, 40%);">+ log.dbg('MAIN CONFIG:\n' + pprint.pformat(MAIN_CONFIG), _category=log.C_CNF)</span><br><span style="color: hsl(120, 100%, 40%);">+ return MAIN_CONFIG</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def get_main_config_value(cfg_name, fail_if_missing=True):</span><br><span style="color: hsl(120, 100%, 40%);">+ cfg = _get_main_config()</span><br><span style="color: hsl(120, 100%, 40%);">+ f = cfg.get(cfg_name, None)</span><br><span style="color: hsl(120, 100%, 40%);">+ if f is None and fail_if_missing:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise RuntimeError('Missing configuration %s' % (cfg_name))</span><br><span style="color: hsl(120, 100%, 40%);">+ return f</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def read_config_file(cfg_name, validation_schema=None, if_missing_return=False):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''Read content of config file cfg_name (referring to key in main config).</span><br><span style="color: hsl(120, 100%, 40%);">+ If "if_missing_return" is different than False, then instead of failing it will return whatever it is stored in that arg</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span> fail_if_missing = True</span><br><span> if if_missing_return is not False:</span><br><span> fail_if_missing = False</span><br><span style="color: hsl(0, 100%, 40%);">- path = get_config_file(basename, fail_if_missing=fail_if_missing)</span><br><span style="color: hsl(120, 100%, 40%);">+ path = get_main_config_value(cfg_name, fail_if_missing=fail_if_missing)</span><br><span> if path is None:</span><br><span> return if_missing_return</span><br><span> return read(path, validation_schema=validation_schema, if_missing_return=if_missing_return)</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-def get_configured_path(label, allow_unset=False):</span><br><span style="color: hsl(0, 100%, 40%);">- global PATHS</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if PATHS is None:</span><br><span style="color: hsl(0, 100%, 40%);">- paths_file, found_in = _get_config_file(PATHS_CONF)</span><br><span style="color: hsl(0, 100%, 40%);">- PATHS = read(paths_file, PATHS_SCHEMA)</span><br><span style="color: hsl(0, 100%, 40%);">- # sorted for deterministic regression test results</span><br><span style="color: hsl(0, 100%, 40%);">- for key, path in sorted(PATHS.items()):</span><br><span style="color: hsl(0, 100%, 40%);">- if not path.startswith(os.pathsep):</span><br><span style="color: hsl(0, 100%, 40%);">- PATHS[key] = os.path.realpath(os.path.join(found_in, path))</span><br><span style="color: hsl(0, 100%, 40%);">- log.dbg(paths_file + ': relative path', path, 'is', PATHS[key], _category=log.C_CNF)</span><br><span style="color: hsl(0, 100%, 40%);">- p = PATHS.get(label)</span><br><span style="color: hsl(0, 100%, 40%);">- if p is None and not allow_unset:</span><br><span style="color: hsl(0, 100%, 40%);">- raise RuntimeError('missing configuration in %s: %r' % (PATHS_CONF, label))</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- log.dbg('Found path', label, 'as', p, _category=log.C_CNF)</span><br><span style="color: hsl(0, 100%, 40%);">- if p.startswith(PATHS_TEMPDIR_STR):</span><br><span style="color: hsl(0, 100%, 40%);">- p = os.path.join(get_tempdir(), p[len(PATHS_TEMPDIR_STR):])</span><br><span style="color: hsl(0, 100%, 40%);">- log.dbg('Path', label, 'contained', PATHS_TEMPDIR_STR, 'and becomes', p, _category=log.C_CNF)</span><br><span style="color: hsl(0, 100%, 40%);">- return p</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span> def get_state_dir():</span><br><span style="color: hsl(0, 100%, 40%);">- return Dir(get_configured_path(PATH_STATE_DIR))</span><br><span style="color: hsl(120, 100%, 40%);">+ return Dir(get_main_config_value(CFG_STATE_DIR))</span><br><span> </span><br><span> def get_suites_dir():</span><br><span style="color: hsl(0, 100%, 40%);">- return Dir(get_configured_path(PATH_SUITES_DIR))</span><br><span style="color: hsl(120, 100%, 40%);">+ return Dir(get_main_config_value(CFG_SUITES_DIR))</span><br><span> </span><br><span> def get_scenarios_dir():</span><br><span style="color: hsl(0, 100%, 40%);">- return Dir(get_configured_path(PATH_SCENARIOS_DIR))</span><br><span style="color: hsl(120, 100%, 40%);">+ return Dir(get_main_config_value(CFG_SCENARIOS_DIR))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+DEFAULTS_CONF = None</span><br><span style="color: hsl(120, 100%, 40%);">+def get_defaults(for_kind):</span><br><span style="color: hsl(120, 100%, 40%);">+ global DEFAULTS_CONF</span><br><span style="color: hsl(120, 100%, 40%);">+ if DEFAULTS_CONF is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ DEFAULTS_CONF = read_config_file(CFG_DEFAULTS_CONF, if_missing_return={})</span><br><span style="color: hsl(120, 100%, 40%);">+ defaults = DEFAULTS_CONF.get(for_kind, {})</span><br><span style="color: hsl(120, 100%, 40%);">+ return copy.deepcopy(defaults)</span><br><span> </span><br><span> def read(path, validation_schema=None, if_missing_return=False):</span><br><span> log.ctx(path)</span><br><span>@@ -191,10 +221,6 @@</span><br><span> config = yaml.safe_load(_tostr(_standardize_item(config)))</span><br><span> return config</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-def get_defaults(for_kind):</span><br><span style="color: hsl(0, 100%, 40%);">- defaults = read_config_file(DEFAULTS_CONF, if_missing_return={})</span><br><span style="color: hsl(0, 100%, 40%);">- return defaults.get(for_kind, {})</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span> def overlay(dest, src):</span><br><span> if is_dict(dest):</span><br><span> if not is_dict(src):</span><br><span>diff --git a/src/osmo_gsm_tester/core/resource.py b/src/osmo_gsm_tester/core/resource.py</span><br><span>index a1a8ea5..af05a51 100644</span><br><span>--- a/src/osmo_gsm_tester/core/resource.py</span><br><span>+++ b/src/osmo_gsm_tester/core/resource.py</span><br><span>@@ -48,7 +48,7 @@</span><br><span> _registered_exit_handler = False</span><br><span> </span><br><span> def __init__(self):</span><br><span style="color: hsl(0, 100%, 40%);">- self.config_path = config.get_config_file(config.RESOURCES_CONF)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.config_path = config.get_main_config_value(config.CFG_RESOURCES_CONF)</span><br><span> self.state_dir = config.get_state_dir()</span><br><span> super().__init__(log.C_CNF, conf=self.config_path, state=self.state_dir.path)</span><br><span> self.read_conf()</span><br><span>diff --git a/sysmocom/README.md b/sysmocom/README.md</span><br><span>index 1b311a0..ffe289b 100644</span><br><span>--- a/sysmocom/README.md</span><br><span>+++ b/sysmocom/README.md</span><br><span>@@ -1,13 +1,14 @@</span><br><span> This a real 2G test suite configured and ready to use.</span><br><span> The only thing missing is a trial dir containing binaries.</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-You can point osmo-gsm-tester.py at this config using the '-c $DIR' command line</span><br><span style="color: hsl(0, 100%, 40%);">-argument, where DIR is the directory path where this README file resides.</span><br><span style="color: hsl(120, 100%, 40%);">+You can point osmo-gsm-tester.py at this config using the '-c $DIR/main.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+command line argument, where DIR is the directory path where this README file</span><br><span style="color: hsl(120, 100%, 40%);">+resides.</span><br><span> </span><br><span> If you have your trial with binary tar archives in ~/my_trial</span><br><span> you can run the suite for example like this:</span><br><span> ```</span><br><span style="color: hsl(0, 100%, 40%);">-osmo-gsm-tester.py -c $DIR ~/my_trial</span><br><span style="color: hsl(120, 100%, 40%);">+osmo-gsm-tester.py -c $DIR/main.conf ~/my_trial</span><br><span> ```</span><br><span> </span><br><span> Alternatively you can setup this example as default config for your user by</span><br><span>@@ -19,4 +20,4 @@</span><br><span> </span><br><span> A ./state dir will be created to store the current osmo-gsm-tester state. If</span><br><span> you prefer not to write to $DIR, set up an own configuration pointing at a</span><br><span style="color: hsl(0, 100%, 40%);">-different path (see paths.conf: 'state_dir').</span><br><span style="color: hsl(120, 100%, 40%);">+different path (see main.conf: 'state_dir').</span><br><span>diff --git a/sysmocom/main.conf b/sysmocom/main.conf</span><br><span>new file mode 100644</span><br><span>index 0000000..b810519</span><br><span>--- /dev/null</span><br><span>+++ b/sysmocom/main.conf</span><br><span>@@ -0,0 +1,6 @@</span><br><span style="color: hsl(120, 100%, 40%);">+state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span style="color: hsl(120, 100%, 40%);">+suites_dir: './suites'</span><br><span style="color: hsl(120, 100%, 40%);">+scenarios_dir: './scenarios'</span><br><span style="color: hsl(120, 100%, 40%);">+default_suites_conf_path: './default-suites.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+defaults_conf_path: './defaults.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+resource_conf_path: './resources.conf'</span><br><span>diff --git a/sysmocom/paths.conf b/sysmocom/paths.conf</span><br><span>deleted file mode 100644</span><br><span>index 27c5818..0000000</span><br><span>--- a/sysmocom/paths.conf</span><br><span>+++ /dev/null</span><br><span>@@ -1,3 +0,0 @@</span><br><span style="color: hsl(0, 100%, 40%);">-state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span style="color: hsl(0, 100%, 40%);">-suites_dir: './suites'</span><br><span style="color: hsl(0, 100%, 40%);">-scenarios_dir: './scenarios'</span><br><span>diff --git a/sysmocom/ttcn3/defaults.conf b/sysmocom/ttcn3/defaults.conf</span><br><span>deleted file mode 120000</span><br><span>index 1e7775f..0000000</span><br><span>--- a/sysmocom/ttcn3/defaults.conf</span><br><span>+++ /dev/null</span><br><span>@@ -1 +0,0 @@</span><br><span style="color: hsl(0, 100%, 40%);">-../defaults.conf</span><br><span>\ No newline at end of file</span><br><span>diff --git a/sysmocom/ttcn3/jenkins-run.sh b/sysmocom/ttcn3/jenkins-run.sh</span><br><span>index c744606..2bc687d 100755</span><br><span>--- a/sysmocom/ttcn3/jenkins-run.sh</span><br><span>+++ b/sysmocom/ttcn3/jenkins-run.sh</span><br><span>@@ -1,6 +1,8 @@</span><br><span> #!/bin/sh</span><br><span> set -e -x</span><br><span> base="$PWD"</span><br><span style="color: hsl(120, 100%, 40%);">+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P) # this file's directory</span><br><span style="color: hsl(120, 100%, 40%);">+OSMO_GSM_TESTER_CONF=${OSMO_GSM_TESTER_CONF:-${SCRIPT_DIR}/main.conf}</span><br><span> </span><br><span> time_start="$(date '+%F %T')"</span><br><span> </span><br><span>diff --git a/sysmocom/ttcn3/main.conf b/sysmocom/ttcn3/main.conf</span><br><span>new file mode 100644</span><br><span>index 0000000..9f38b4e</span><br><span>--- /dev/null</span><br><span>+++ b/sysmocom/ttcn3/main.conf</span><br><span>@@ -0,0 +1,6 @@</span><br><span style="color: hsl(120, 100%, 40%);">+state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span style="color: hsl(120, 100%, 40%);">+suites_dir: './suites'</span><br><span style="color: hsl(120, 100%, 40%);">+scenarios_dir: './scenarios'</span><br><span style="color: hsl(120, 100%, 40%);">+default_suites_conf_path: './default-suites.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+defaults_conf_path: '../defaults.conf'</span><br><span style="color: hsl(120, 100%, 40%);">+resource_conf_path: './resources.conf'</span><br><span>diff --git a/sysmocom/ttcn3/paths.conf b/sysmocom/ttcn3/paths.conf</span><br><span>deleted file mode 100644</span><br><span>index 27c5818..0000000</span><br><span>--- a/sysmocom/ttcn3/paths.conf</span><br><span>+++ /dev/null</span><br><span>@@ -1,3 +0,0 @@</span><br><span style="color: hsl(0, 100%, 40%);">-state_dir: '/var/tmp/osmo-gsm-tester/state'</span><br><span style="color: hsl(0, 100%, 40%);">-suites_dir: './suites'</span><br><span style="color: hsl(0, 100%, 40%);">-scenarios_dir: './scenarios'</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18204">change 18204</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18204"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: Ieca65b71b543c44cfcec8e83efd0fe053c432e55 </div>
<div style="display:none"> Gerrit-Change-Number: 18204 </div>
<div style="display:none"> Gerrit-PatchSet: 4 </div>
<div style="display:none"> Gerrit-Owner: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder </div>
<div style="display:none"> Gerrit-Reviewer: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>