<p>pespin has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18832">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">Add per-test KPI support<br><br>tests can now use 'tenv.test().set_kpis(some_dict)' to set any kind of<br>data as KPIs, which will be presented in the junit report.<br><br>The representation of KPIs in the xml file doesn't follow the junit<br>format, mainly because it has no support for per-test properties.<br><br>Change-Id: I00e976f65a202e82d440bf33708f06c8ce2643e2<br>---<br>M selftest/report_test/expected_junit_output.xml<br>M selftest/report_test/report_test.ok<br>M selftest/report_test/report_test.py<br>M src/osmo_gsm_tester/core/report.py<br>M src/osmo_gsm_tester/core/test.py<br>5 files changed, 98 insertions(+), 3 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/32/18832/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/selftest/report_test/expected_junit_output.xml b/selftest/report_test/expected_junit_output.xml</span><br><span>index 5de0edf..9f6185c 100644</span><br><span>--- a/selftest/report_test/expected_junit_output.xml</span><br><span>+++ b/selftest/report_test/expected_junit_output.xml</span><br><span>@@ -1,4 +1,4 @@</span><br><span style="color: hsl(0, 100%, 40%);">-<testsuites errors="2" failures="1" name="trial" tests="10" time="102"></span><br><span style="color: hsl(120, 100%, 40%);">+<testsuites errors="2" failures="2" name="trial" tests="12" time="122"></span><br><span> <testsuite disabled="0" errors="0" failures="0" hostname="localhost" id="0" name="suiteA" skipped="0" tests="2"></span><br><span> <testcase classname="suiteA" name="suiteA-0" time="30"></span><br><span> <system-out>test log file not available</system-out></span><br><span>@@ -59,4 +59,34 @@</span><br><span> <property name="ref:orange" value="abcd"></property></span><br><span> </properties></span><br><span> </testsuite></span><br><span style="color: hsl(0, 100%, 40%);">-</testsuites></span><br><span style="color: hsl(120, 100%, 40%);">+ <testsuite disabled="0" errors="0" failures="1" hostname="localhost" id="4" name="suiteE" skipped="0" tests="2"></span><br><span style="color: hsl(120, 100%, 40%);">+ <testcase classname="suiteE" name="suiteE-0" time="12"></span><br><span style="color: hsl(120, 100%, 40%);">+ <failure type="fake_fail_type">fake_fail_message</failure></span><br><span style="color: hsl(120, 100%, 40%);">+ <system-err>system stderr fake content</system-err></span><br><span style="color: hsl(120, 100%, 40%);">+ <kpis></span><br><span style="color: hsl(120, 100%, 40%);">+ <kpi_node name="ueA"></span><br><span style="color: hsl(120, 100%, 40%);">+ <property name="kpiA" value="30"></property></span><br><span style="color: hsl(120, 100%, 40%);">+ <property name="kpiB" value="foobar"></property></span><br><span style="color: hsl(120, 100%, 40%);">+ <kpi_node name="yet-another-level"></span><br><span style="color: hsl(120, 100%, 40%);">+ <property name="foo" value="bar"></property></span><br><span style="color: hsl(120, 100%, 40%);">+ </kpi_node></span><br><span style="color: hsl(120, 100%, 40%);">+ </kpi_node></span><br><span style="color: hsl(120, 100%, 40%);">+ <kpi_node name="enbD"></span><br><span style="color: hsl(120, 100%, 40%);">+ <property name="foobar-boolean" value="True"></property></span><br><span style="color: hsl(120, 100%, 40%);">+ </kpi_node></span><br><span style="color: hsl(120, 100%, 40%);">+ <property name="somekpi" value="someval"></property></span><br><span style="color: hsl(120, 100%, 40%);">+ </kpis></span><br><span style="color: hsl(120, 100%, 40%);">+ <system-out>test log file not available</system-out></span><br><span style="color: hsl(120, 100%, 40%);">+ </testcase></span><br><span style="color: hsl(120, 100%, 40%);">+ <testcase classname="suiteE" name="suiteE-1" time="10"></span><br><span style="color: hsl(120, 100%, 40%);">+ <kpis></span><br><span style="color: hsl(120, 100%, 40%);">+ <property name="abcd" value="abcdval"></property></span><br><span style="color: hsl(120, 100%, 40%);">+ </kpis></span><br><span style="color: hsl(120, 100%, 40%);">+ <system-out>test log file not available</system-out></span><br><span style="color: hsl(120, 100%, 40%);">+ </testcase></span><br><span style="color: hsl(120, 100%, 40%);">+ <properties></span><br><span style="color: hsl(120, 100%, 40%);">+ <property name="ref:foobar/potato" value="1234"></property></span><br><span style="color: hsl(120, 100%, 40%);">+ <property name="ref:orange" value="abcd"></property></span><br><span style="color: hsl(120, 100%, 40%);">+ </properties></span><br><span style="color: hsl(120, 100%, 40%);">+ </testsuite></span><br><span style="color: hsl(120, 100%, 40%);">+</testsuites></span><br><span>\ No newline at end of file</span><br><span>diff --git a/selftest/report_test/report_test.ok b/selftest/report_test/report_test.ok</span><br><span>index 87092ea..442d343 100644</span><br><span>--- a/selftest/report_test/report_test.ok</span><br><span>+++ b/selftest/report_test/report_test.ok</span><br><span>@@ -15,3 +15,5 @@</span><br><span> tst {combining_scenarios='config'}: DBG: {definition_conf={}} [suiteC↪{combining_scenarios='config'}]</span><br><span> tst suiteD: DBG: {combining='config'}</span><br><span> tst {combining_scenarios='config'}: DBG: {definition_conf={}} [suiteD↪{combining_scenarios='config'}]</span><br><span style="color: hsl(120, 100%, 40%);">+tst suiteE: DBG: {combining='config'}</span><br><span style="color: hsl(120, 100%, 40%);">+tst {combining_scenarios='config'}: DBG: {definition_conf={}} [suiteE↪{combining_scenarios='config'}]</span><br><span>diff --git a/selftest/report_test/report_test.py b/selftest/report_test/report_test.py</span><br><span>index 57e3a89..5888bf3 100755</span><br><span>--- a/selftest/report_test/report_test.py</span><br><span>+++ b/selftest/report_test/report_test.py</span><br><span>@@ -39,11 +39,13 @@</span><br><span> self.suite_dir = util.Dir(example_trial_dir).new_child('suitedef' + name)</span><br><span> </span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-def fake_run_test(test_obj, status, duration, sysout=None):</span><br><span style="color: hsl(120, 100%, 40%);">+def fake_run_test(test_obj, status, duration, sysout=None, kpis=None):</span><br><span> test_obj.status = status</span><br><span> test_obj.duration = duration</span><br><span> if sysout is not None:</span><br><span> test_obj.set_report_stdout(sysout)</span><br><span style="color: hsl(120, 100%, 40%);">+ if kpis is not None:</span><br><span style="color: hsl(120, 100%, 40%);">+ test_obj.set_kpis(kpis)</span><br><span> if status == test.Test.FAIL:</span><br><span> test_obj.fail_type = 'fake_fail_type'</span><br><span> test_obj.fail_message = 'fake_fail_message'</span><br><span>@@ -92,6 +94,14 @@</span><br><span> fake_run_test(s.tests[1], test.Test.PASS, 10)</span><br><span> fake_run_suite(s, 20)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+# Test adding KPIs</span><br><span style="color: hsl(120, 100%, 40%);">+s_def = FakeSuiteDefinition('suiteE', 2)</span><br><span style="color: hsl(120, 100%, 40%);">+s = suite.SuiteRun(trial, s_def.name(), s_def)</span><br><span style="color: hsl(120, 100%, 40%);">+trial.suites.append(s)</span><br><span style="color: hsl(120, 100%, 40%);">+fake_run_test(s.tests[0], test.Test.FAIL, 12, kpis={'ueA': {'kpiA': 30, 'kpiB': 'foobar', 'yet-another-level': {'foo': 'bar'}}, 'enbD': {'foobar-boolean': True }, 'somekpi': 'someval'})</span><br><span style="color: hsl(120, 100%, 40%);">+fake_run_test(s.tests[1], test.Test.PASS, 10, kpis={'abcd': 'abcdval'})</span><br><span style="color: hsl(120, 100%, 40%);">+fake_run_suite(s, 20)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> element = report.trial_to_junit(trial)</span><br><span> </span><br><span> def indent(elem, level=0):</span><br><span>@@ -126,6 +136,9 @@</span><br><span> with open(exp_path, 'r') as f:</span><br><span> exp = f.read().rstrip()</span><br><span> udiff(exp, got, exp_path)</span><br><span style="color: hsl(120, 100%, 40%);">+ # Uncomment to update exp_path:</span><br><span style="color: hsl(120, 100%, 40%);">+ #with open(exp_path, 'w') as f:</span><br><span style="color: hsl(120, 100%, 40%);">+ # f.write(got)</span><br><span> </span><br><span> #deleting generated tmp trial dir:</span><br><span> shutil.rmtree(example_trial_dir, ignore_errors=True)</span><br><span>diff --git a/src/osmo_gsm_tester/core/report.py b/src/osmo_gsm_tester/core/report.py</span><br><span>index 5014bf5..d2c68c5 100644</span><br><span>--- a/src/osmo_gsm_tester/core/report.py</span><br><span>+++ b/src/osmo_gsm_tester/core/report.py</span><br><span>@@ -53,6 +53,46 @@</span><br><span> prop.set('name', 'ref:' + key)</span><br><span> prop.set('value', val)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+def dict_to_junit(parent, d):</span><br><span style="color: hsl(120, 100%, 40%);">+ for key, val in d.items():</span><br><span style="color: hsl(120, 100%, 40%);">+ if isinstance(val, dict):</span><br><span style="color: hsl(120, 100%, 40%);">+ node = et.SubElement(parent, 'kpi_node')</span><br><span style="color: hsl(120, 100%, 40%);">+ node.set('name', key)</span><br><span style="color: hsl(120, 100%, 40%);">+ dict_to_junit(node, val)</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ if isinstance(val, (tuple, list)):</span><br><span style="color: hsl(120, 100%, 40%);">+ node = et.SubElement(parent, 'kpi_node')</span><br><span style="color: hsl(120, 100%, 40%);">+ node.set('name', key)</span><br><span style="color: hsl(120, 100%, 40%);">+ list_to_junit(node, val)</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ # scalar:</span><br><span style="color: hsl(120, 100%, 40%);">+ node = et.SubElement(parent, 'property')</span><br><span style="color: hsl(120, 100%, 40%);">+ node.set('name', key)</span><br><span style="color: hsl(120, 100%, 40%);">+ node.set('value', str(val))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def list_to_junit(parent, li):</span><br><span style="color: hsl(120, 100%, 40%);">+ for i in range(len(li)):</span><br><span style="color: hsl(120, 100%, 40%);">+ if isinstance(li[i], dict):</span><br><span style="color: hsl(120, 100%, 40%);">+ node = et.SubElement(parent, 'kpi_node')</span><br><span style="color: hsl(120, 100%, 40%);">+ node.set('name', str(i))</span><br><span style="color: hsl(120, 100%, 40%);">+ dict_to_junit(node, li[i])</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ if isinstance(val, (tuple, list)):</span><br><span style="color: hsl(120, 100%, 40%);">+ node = et.SubElement(parent, 'kpi_node')</span><br><span style="color: hsl(120, 100%, 40%);">+ node.set('name', str(i))</span><br><span style="color: hsl(120, 100%, 40%);">+ list_to_junit(node, li[i])</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ # scalar:</span><br><span style="color: hsl(120, 100%, 40%);">+ node = et.SubElement(parent, 'property')</span><br><span style="color: hsl(120, 100%, 40%);">+ node.set('name', str(i))</span><br><span style="color: hsl(120, 100%, 40%);">+ node.set('value', str(li[i]))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def kpis_to_junit(parent, kpis):</span><br><span style="color: hsl(120, 100%, 40%);">+ if not kpis:</span><br><span style="color: hsl(120, 100%, 40%);">+ return</span><br><span style="color: hsl(120, 100%, 40%);">+ assert isinstance(kpis, dict)</span><br><span style="color: hsl(120, 100%, 40%);">+ knode = et.SubElement(parent, 'kpis')</span><br><span style="color: hsl(120, 100%, 40%);">+ dict_to_junit(knode, kpis)</span><br><span> </span><br><span> def trial_to_junit_write(trial, junit_path):</span><br><span> elements = et.ElementTree(element=trial_to_junit(trial))</span><br><span>@@ -118,6 +158,7 @@</span><br><span> elif t.status != test.Test.PASS:</span><br><span> error = et.SubElement(testcase, 'error')</span><br><span> error.text = 'could not run'</span><br><span style="color: hsl(120, 100%, 40%);">+ kpis_to_junit(testcase, t.kpis())</span><br><span> sout = et.SubElement(testcase, 'system-out')</span><br><span> sout.text = escape_xml_invalid_characters(t.report_stdout())</span><br><span> return testcase</span><br><span>diff --git a/src/osmo_gsm_tester/core/test.py b/src/osmo_gsm_tester/core/test.py</span><br><span>index c6d88e6..dfbd169 100644</span><br><span>--- a/src/osmo_gsm_tester/core/test.py</span><br><span>+++ b/src/osmo_gsm_tester/core/test.py</span><br><span>@@ -49,6 +49,7 @@</span><br><span> self.fail_message = None</span><br><span> self.log_targets = []</span><br><span> self._report_stdout = None</span><br><span style="color: hsl(120, 100%, 40%);">+ self._kpis = None</span><br><span> self.timeout = int(config_test_specific['timeout']) if 'timeout' in config_test_specific else None</span><br><span> </span><br><span> def module_name(self):</span><br><span>@@ -139,6 +140,14 @@</span><br><span> def config_test_specific(self):</span><br><span> return self._config_test_specific</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def set_kpis(self, kpis):</span><br><span style="color: hsl(120, 100%, 40%);">+ if not isinstance(kpis, dict):</span><br><span style="color: hsl(120, 100%, 40%);">+ raise log.Error('Expected dictionary in toplevel kpis')</span><br><span style="color: hsl(120, 100%, 40%);">+ self._kpis = kpis</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def kpis(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ return self._kpis</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> def set_report_stdout(self, text):</span><br><span> 'Overwrite stdout text stored in report from inside a test'</span><br><span> self._report_stdout = text</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18832">change 18832</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/18832"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I00e976f65a202e82d440bf33708f06c8ce2643e2 </div>
<div style="display:none"> Gerrit-Change-Number: 18832 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>