<p>neels has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/21516">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">add test.report_fragment()<br><br>Allow enriching the junit output with arbitrary subtasks within a test.<br><br>The current aim is, for handover tests, to not just show that a test<br>failed, but to show exactly which steps worked and which didn't, e.g.:<br><br> handover.py__01_bts0_started PASSED<br> handover.py__02.1_ms0_attach PASSED<br> handover.py__02.2_ms1_attach PASSED<br> handover.py__02.3_subscribed_in_msc PASSED<br> handover.py__03_call_established PASSED<br> handover.py__04.1_bts1_started FAILED<br><br>In this case it is immediately obvious from looking at the jenkins<br>results analyzer that bts1 is the cause of the test failure, and it is<br>visible which parts of the test are flaky, over time.<br><br>First user Will be the upcoming handover_2G suite, in<br>I0b2671304165a1aaae2b386af46fbd8b098e3bd8.<br><br>Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59<br>---<br>M selftest/suite_test/suite_test.ok.ign<br>M selftest/suite_test/suite_test.py<br>A selftest/suite_test/suitedirA/test_suite/test_report_fragment.py<br>M src/osmo_gsm_tester/core/report.py<br>M src/osmo_gsm_tester/core/test.py<br>5 files changed, 102 insertions(+), 5 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/16/21516/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/selftest/suite_test/suite_test.ok.ign b/selftest/suite_test/suite_test.ok.ign</span><br><span>index 460da92..7fb57c6 100644</span><br><span>--- a/selftest/suite_test/suite_test.ok.ign</span><br><span>+++ b/selftest/suite_test/suite_test.ok.ign</span><br><span>@@ -4,3 +4,8 @@</span><br><span> {combining_scenarios='resources', scenario='foo'}:.* {combining_scenarios='resources', scenario='foo'}: [RESOURCE_DICT]</span><br><span> test_suite-[0-9]*-[0-9]* test_suite-[ID_NUM]-[ID_NUM]</span><br><span> suiteC-[0-9]*-[0-9]* suiteC-[ID_NUM]-[ID_NUM]</span><br><span style="color: hsl(120, 100%, 40%);">+line [0-9]+ line [LINENR]</span><br><span style="color: hsl(120, 100%, 40%);">+[0-9][0-9]:[0-9][0-9]:[0-9][0-9]\.[0-9]+ [TIMESTAMP]</span><br><span style="color: hsl(120, 100%, 40%);">+time="[0-9]+" time="[VAL]"</span><br><span style="color: hsl(120, 100%, 40%);">+ File "[^"]*" File "[FILE]"</span><br><span style="color: hsl(120, 100%, 40%);">+timestamp="[^"]+" timestamp="[TIMESTAMP]"</span><br><span>diff --git a/selftest/suite_test/suite_test.py b/selftest/suite_test/suite_test.py</span><br><span>index 9708037..75eb712 100755</span><br><span>--- a/selftest/suite_test/suite_test.py</span><br><span>+++ b/selftest/suite_test/suite_test.py</span><br><span>@@ -11,6 +11,8 @@</span><br><span> from osmo_gsm_tester.core import suite</span><br><span> from osmo_gsm_tester.core.schema import generate_schemas, get_all_schema</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+import xml.etree.ElementTree as et</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'paths.conf')</span><br><span> </span><br><span> example_trial_dir = os.path.join('test_trial_tmp')</span><br><span>@@ -51,6 +53,11 @@</span><br><span> results = s.run_tests('hello_world.py')</span><br><span> print(report.suite_to_text(s))</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+print('- run report fragment test')</span><br><span style="color: hsl(120, 100%, 40%);">+results = s.run_tests('test_report_fragment.py')</span><br><span style="color: hsl(120, 100%, 40%);">+print(report.suite_to_text(s))</span><br><span style="color: hsl(120, 100%, 40%);">+print('\njunit XML:\n' + et.tostring(report.suite_to_junit(s)).decode('utf-8') + '\n\n')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> log.style_change(src=True)</span><br><span> #log.style_change(trace=True)</span><br><span> print('\n- a test with an error')</span><br><span>diff --git a/selftest/suite_test/suitedirA/test_suite/test_report_fragment.py b/selftest/suite_test/suitedirA/test_suite/test_report_fragment.py</span><br><span>new file mode 100644</span><br><span>index 0000000..06ff37d</span><br><span>--- /dev/null</span><br><span>+++ b/selftest/suite_test/suitedirA/test_suite/test_report_fragment.py</span><br><span>@@ -0,0 +1,11 @@</span><br><span style="color: hsl(120, 100%, 40%);">+from osmo_gsm_tester.testenv import *</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+with test.report_fragment('fragment1'):</span><br><span style="color: hsl(120, 100%, 40%);">+ print('a step in the first fragment')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+with test.report_fragment('fragment2'):</span><br><span style="color: hsl(120, 100%, 40%);">+ print('a step in the second fragment')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+with test.report_fragment('fragment3'):</span><br><span style="color: hsl(120, 100%, 40%);">+ print('a step in the third fragment')</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('failure in the third fragment')</span><br><span>diff --git a/src/osmo_gsm_tester/core/report.py b/src/osmo_gsm_tester/core/report.py</span><br><span>index c3390fe..35327ce 100644</span><br><span>--- a/src/osmo_gsm_tester/core/report.py</span><br><span>+++ b/src/osmo_gsm_tester/core/report.py</span><br><span>@@ -132,14 +132,37 @@</span><br><span> testsuite.set('time', str(math.ceil(suite.duration)))</span><br><span> testsuite.set('tests', str(len(suite.tests)))</span><br><span> passed, skipped, failed, errors = suite.count_test_results()</span><br><span style="color: hsl(0, 100%, 40%);">- testsuite.set('errors', str(errors))</span><br><span style="color: hsl(0, 100%, 40%);">- testsuite.set('failures', str(failed))</span><br><span style="color: hsl(0, 100%, 40%);">- testsuite.set('skipped', str(skipped))</span><br><span style="color: hsl(0, 100%, 40%);">- testsuite.set('disabled', str(skipped))</span><br><span> for suite_test in suite.tests:</span><br><span> testcase = test_to_junit(suite_test)</span><br><span> testcase.set('classname', suite.name())</span><br><span> testsuite.append(testcase)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for report_fragment in suite_test.report_fragments:</span><br><span style="color: hsl(120, 100%, 40%);">+ full_name = '%s__%s' % (suite_test.name(), report_fragment.name)</span><br><span style="color: hsl(120, 100%, 40%);">+ el = et.Element('testcase')</span><br><span style="color: hsl(120, 100%, 40%);">+ el.set('name', full_name)</span><br><span style="color: hsl(120, 100%, 40%);">+ el.set('time', str(math.ceil(report_fragment.duration)))</span><br><span style="color: hsl(120, 100%, 40%);">+ if report_fragment.result == test.Test.SKIP:</span><br><span style="color: hsl(120, 100%, 40%);">+ et.SubElement(el, 'skipped')</span><br><span style="color: hsl(120, 100%, 40%);">+ skipped += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ elif report_fragment.result == test.Test.FAIL:</span><br><span style="color: hsl(120, 100%, 40%);">+ failure = et.SubElement(el, 'failure')</span><br><span style="color: hsl(120, 100%, 40%);">+ failure.set('type', suite_test.fail_type or 'failure')</span><br><span style="color: hsl(120, 100%, 40%);">+ failed += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ elif report_fragment.result != test.Test.PASS:</span><br><span style="color: hsl(120, 100%, 40%);">+ error = et.SubElement(el, 'error')</span><br><span style="color: hsl(120, 100%, 40%);">+ error.text = 'could not run'</span><br><span style="color: hsl(120, 100%, 40%);">+ errors += 1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if report_fragment.output:</span><br><span style="color: hsl(120, 100%, 40%);">+ sout = et.SubElement(el, 'system-out')</span><br><span style="color: hsl(120, 100%, 40%);">+ sout.text = escape_xml_invalid_characters(report_fragment.output)</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.append(el)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.set('errors', str(errors))</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.set('failures', str(failed))</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.set('skipped', str(skipped))</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.set('disabled', str(skipped))</span><br><span> return testsuite</span><br><span> </span><br><span> def test_to_junit(t):</span><br><span>diff --git a/src/osmo_gsm_tester/core/test.py b/src/osmo_gsm_tester/core/test.py</span><br><span>index adfc444..3847e48 100644</span><br><span>--- a/src/osmo_gsm_tester/core/test.py</span><br><span>+++ b/src/osmo_gsm_tester/core/test.py</span><br><span>@@ -43,6 +43,7 @@</span><br><span> self._config_test_specific = config_test_specific</span><br><span> self.path = os.path.join(self.suite_run.definition.suite_dir, self.basename)</span><br><span> self.status = Test.UNKNOWN</span><br><span style="color: hsl(120, 100%, 40%);">+ self.report_fragments = []</span><br><span> self.start_timestamp = 0</span><br><span> self.duration = 0</span><br><span> self.fail_type = None</span><br><span>@@ -167,7 +168,7 @@</span><br><span> </span><br><span> def log_file(self):</span><br><span> for lt in self.log_targets:</span><br><span style="color: hsl(0, 100%, 40%);">- if isinstance(lt, FileLogTarget):</span><br><span style="color: hsl(120, 100%, 40%);">+ if isinstance(lt, log.FileLogTarget):</span><br><span> return lt</span><br><span> return None</span><br><span> </span><br><span>@@ -183,4 +184,54 @@</span><br><span> return ''</span><br><span> return lt.get_output(since_mark)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def report_fragment(self, name, result=None, **kwargs):</span><br><span style="color: hsl(120, 100%, 40%);">+ return Test.ReportFragment(parent_test=self, name=name, result=result, **kwargs)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ class ReportFragment:</span><br><span style="color: hsl(120, 100%, 40%);">+ '''Add additional test results in junit XML.</span><br><span style="color: hsl(120, 100%, 40%);">+ Convenient method that includes a test log:</span><br><span style="color: hsl(120, 100%, 40%);">+ with test.report_fragment('foo'):</span><br><span style="color: hsl(120, 100%, 40%);">+ do_test_steps()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ Or manually add a report fragment directly:</span><br><span style="color: hsl(120, 100%, 40%);">+ test.report_fragment('foo', result = test.PASS if worked else test.FAIL)</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __init__(self, parent_test, name, result=None, output=None, since_mark=None, start_time=0.0):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.parent_test = parent_test</span><br><span style="color: hsl(120, 100%, 40%);">+ self.name = name</span><br><span style="color: hsl(120, 100%, 40%);">+ self.result = Test.UNKNOWN</span><br><span style="color: hsl(120, 100%, 40%);">+ self.duration = 0.0</span><br><span style="color: hsl(120, 100%, 40%);">+ self.output = output</span><br><span style="color: hsl(120, 100%, 40%);">+ self.start_time = start_time</span><br><span style="color: hsl(120, 100%, 40%);">+ self.log_mark = since_mark</span><br><span style="color: hsl(120, 100%, 40%);">+ assert name not in (x.name for x in self.parent_test.report_fragments)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.parent_test.report_fragments.append(self)</span><br><span style="color: hsl(120, 100%, 40%);">+ if result is not None:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.got_result(result)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __str__(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ return '%s %s: %s (%.1fs)' % (self.parent_test.name(), self.name, self.result, self.duration)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __enter__(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.start_time = self.parent_test.elapsed_time()</span><br><span style="color: hsl(120, 100%, 40%);">+ self.log_mark = self.parent_test.get_log_mark()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __exit__(self, *exc_info):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.got_result(self.parent_test.PASS if exc_info[0] is None else self.parent_test.FAIL,</span><br><span style="color: hsl(120, 100%, 40%);">+ exc_info=exc_info)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def got_result(self, result, exc_info=None):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.result = result</span><br><span style="color: hsl(120, 100%, 40%);">+ self.duration = self.parent_test.elapsed_time() - self.start_time</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.log_mark is not None and self.output is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.output = self.parent_test.get_log_output(since_mark=self.log_mark)</span><br><span style="color: hsl(120, 100%, 40%);">+ if exc_info is not None and exc_info[0] is not None:</span><br><span style="color: hsl(120, 100%, 40%);">+ o = []</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.output:</span><br><span style="color: hsl(120, 100%, 40%);">+ o.append(self.output)</span><br><span style="color: hsl(120, 100%, 40%);">+ o.extend(traceback.format_exception(*exc_info))</span><br><span style="color: hsl(120, 100%, 40%);">+ self.output = '\n'.join(o)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.parent_test.log('----- Report fragment:', self)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> # vim: expandtab tabstop=4 shiftwidth=4</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/21516">change 21516</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/21516"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59 </div>
<div style="display:none"> Gerrit-Change-Number: 21516 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: neels <nhofmeyr@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>