<p>neels <strong>submitted</strong> this change.</p><p><a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/21516">View Change</a></p><div style="white-space:pre-wrap"></div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">add test.report_fragment()<br><br>Allow enriching the junit output with arbitrary subtasks within a test.<br><br>The current aim is, for handover tests, to not just show that a test<br>failed, but to show exactly which steps worked and which didn't, e.g.:<br><br> handover.py/01_bts0_started PASSED<br> handover.py/02.1_ms0_attach PASSED<br> handover.py/02.2_ms1_attach PASSED<br> handover.py/02.3_subscribed_in_msc PASSED<br> handover.py/03_call_established PASSED<br> handover.py/04.1_bts1_started FAILED<br><br>In this case it is immediately obvious from looking at the jenkins<br>results analyzer that bts1 is the cause of the test failure, and it is<br>visible which parts of the test are flaky, over time.<br><br>First user Will be the upcoming handover_2G suite, in<br>I0b2671304165a1aaae2b386af46fbd8b098e3bd8.<br><br>Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59<br>---<br>M selftest/suite_test/suite_test.ok<br>M selftest/suite_test/suite_test.ok.ign<br>M selftest/suite_test/suite_test.py<br>A selftest/suite_test/suitedirA/test_suite/test_report_fragment.py<br>M src/osmo_gsm_tester/core/report.py<br>M src/osmo_gsm_tester/core/test.py<br>6 files changed, 174 insertions(+), 12 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/selftest/suite_test/suite_test.ok b/selftest/suite_test/suite_test.ok</span><br><span>index 3790e1a..9f60b70 100644</span><br><span>--- a/selftest/suite_test/suite_test.ok</span><br><span>+++ b/selftest/suite_test/suite_test.ok</span><br><span>@@ -168,15 +168,67 @@</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite PASS</span><br><span> ---------------------------------------------------------------------</span><br><span style="color: hsl(0, 100%, 40%);">-PASS: test_suite (pass: 1, skip: 7)</span><br><span style="color: hsl(120, 100%, 40%);">+PASS: test_suite (pass: 1, skip: 8)</span><br><span> pass: hello_world.py (N.N sec)</span><br><span> skip: mo_mt_sms.py</span><br><span> skip: mo_sms.py</span><br><span> skip: test_error.py</span><br><span> skip: test_fail.py</span><br><span> skip: test_fail_raise.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_report_fragment.py</span><br><span> skip: test_suite_params.py</span><br><span> skip: test_timeout.py</span><br><span style="color: hsl(120, 100%, 40%);">+- run report fragment test</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+---------------------------------------------------------------------</span><br><span style="color: hsl(120, 100%, 40%);">+trial test_suite</span><br><span style="color: hsl(120, 100%, 40%);">+---------------------------------------------------------------------</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+----------------------------------------------</span><br><span style="color: hsl(120, 100%, 40%);">+trial test_suite test_report_fragment.py</span><br><span style="color: hsl(120, 100%, 40%);">+----------------------------------------------</span><br><span style="color: hsl(120, 100%, 40%);">+tst test_report_fragment.py:[LINENR]: a step in the first fragment [test_suite↪test_report_fragment.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+tst test_report_fragment.py:[LINENR]: ----- Report fragment: test_suite/test_report_fragment.py/fragment1: pass (N.Ns) [test_suite↪test_report_fragment.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+tst test_report_fragment.py:[LINENR]: a step in the second fragment [test_suite↪test_report_fragment.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+tst test_report_fragment.py:[LINENR]: ----- Report fragment: test_suite/test_report_fragment.py/fragment2: pass (N.Ns) [test_suite↪test_report_fragment.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+tst test_report_fragment.py:[LINENR]: a step in the third fragment [test_suite↪test_report_fragment.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+tst test_report_fragment.py:[LINENR]: ----- Report fragment: test_suite/test_report_fragment.py/fragment3: FAIL (N.Ns) [test_suite↪test_report_fragment.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+tst test_report_fragment.py:[LINENR]: ERR: Exception: failure in the third fragment [test_suite↪test_report_fragment.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+tst test_report_fragment.py:[LINENR]: Test FAILED (N.N sec) [test_suite↪test_report_fragment.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+---------------------------------------------------------------------</span><br><span style="color: hsl(120, 100%, 40%);">+trial test_suite FAIL</span><br><span style="color: hsl(120, 100%, 40%);">+---------------------------------------------------------------------</span><br><span style="color: hsl(120, 100%, 40%);">+FAIL: test_suite (fail: 1, skip: 8)</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: hello_world.py (N.N sec)</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: mo_mt_sms.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: mo_sms.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_error.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_fail.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_fail_raise.py</span><br><span style="color: hsl(120, 100%, 40%);">+ FAIL: test_report_fragment.py (N.N sec) Exception: failure in the third fragment</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_suite_params.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_timeout.py</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+################################### junit XML:</span><br><span style="color: hsl(120, 100%, 40%);">+<testsuite disabled="8" errors="0" failures="2" hostname="localhost" name="test_suite" skipped="8" tests="9" time="[VAL]" timestamp="[TIMESTAMP]"><testcase classname="test_suite" name="hello_world.py" time="[VAL]"><skipped /><system-out></span><br><span style="color: hsl(120, 100%, 40%);">+----------------------------------------------</span><br><span style="color: hsl(120, 100%, 40%);">+trial test_suite hello_world.py</span><br><span style="color: hsl(120, 100%, 40%);">+----------------------------------------------</span><br><span style="color: hsl(120, 100%, 40%);">+[TIMESTAMP] tst hello_world.py:[LINENR]: hello world</span><br><span style="color: hsl(120, 100%, 40%);">+[TIMESTAMP] tst hello_world.py:[LINENR]: I am 'test_suite' / 'hello_world.py'</span><br><span style="color: hsl(120, 100%, 40%);">+[TIMESTAMP] tst hello_world.py:[LINENR]: one</span><br><span style="color: hsl(120, 100%, 40%);">+two</span><br><span style="color: hsl(120, 100%, 40%);">+three</span><br><span style="color: hsl(120, 100%, 40%);">+[TIMESTAMP] tst hello_world.py:[LINENR] Test passed (N.N sec)</span><br><span style="color: hsl(120, 100%, 40%);">+</system-out></testcase><testcase classname="test_suite" name="mo_mt_sms.py" time="[VAL]"><skipped /><system-out>test log file not available</system-out></testcase><testcase classname="test_suite" name="mo_sms.py" time="[VAL]"><skipped /><system-out>test log file not available</system-out></testcase><testcase classname="test_suite" name="test_error.py" time="[VAL]"><skipped /><system-out>test log file not available</system-out></testcase><testcase classname="test_suite" name="test_fail.py" time="[VAL]"><skipped /><system-out>test log file not available</system-out></testcase><testcase classname="test_suite" name="test_fail_raise.py" time="[VAL]"><skipped /><system-out>test log file not available</system-out></testcase><testcase classname="test_suite" name="test_report_fragment.py" time="[VAL]"><failure type="Exception">failure in the third fragment</failure><system-err>[BACKTRACE]</span><br><span style="color: hsl(120, 100%, 40%);">+raise Exception('failure in the third fragment')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+Exception: failure in the third fragment</span><br><span style="color: hsl(120, 100%, 40%);">+</system-out></testcase><testcase classname="test_suite" name="test_suite_params.py" time="[VAL]"><skipped /><system-out>test log file not available</system-out></testcase><testcase classname="test_suite" name="test_timeout.py" time="[VAL]"><skipped /><system-out>test log file not available</system-out></testcase></testsuite></span><br><span style="color: hsl(120, 100%, 40%);">+###################################</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> </span><br><span> - a test with an error</span><br><span> </span><br><span>@@ -193,13 +245,14 @@</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite FAIL</span><br><span> ---------------------------------------------------------------------</span><br><span style="color: hsl(0, 100%, 40%);">-FAIL: test_suite (fail: 1, skip: 7)</span><br><span style="color: hsl(120, 100%, 40%);">+FAIL: test_suite (fail: 1, skip: 8)</span><br><span> skip: hello_world.py (N.N sec)</span><br><span> skip: mo_mt_sms.py</span><br><span> skip: mo_sms.py</span><br><span> FAIL: test_error.py (N.N sec) AssertionError: test_error.py:[LINENR]: assert False</span><br><span> skip: test_fail.py</span><br><span> skip: test_fail_raise.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_report_fragment.py (N.N sec)</span><br><span> skip: test_suite_params.py</span><br><span> skip: test_timeout.py</span><br><span> </span><br><span>@@ -218,13 +271,14 @@</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite FAIL</span><br><span> ---------------------------------------------------------------------</span><br><span style="color: hsl(0, 100%, 40%);">-FAIL: test_suite (fail: 1, skip: 7)</span><br><span style="color: hsl(120, 100%, 40%);">+FAIL: test_suite (fail: 1, skip: 8)</span><br><span> skip: hello_world.py (N.N sec)</span><br><span> skip: mo_mt_sms.py</span><br><span> skip: mo_sms.py</span><br><span> skip: test_error.py (N.N sec)</span><br><span> FAIL: test_fail.py (N.N sec) EpicFail: This failure is expected</span><br><span> skip: test_fail_raise.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_report_fragment.py (N.N sec)</span><br><span> skip: test_suite_params.py</span><br><span> skip: test_timeout.py</span><br><span> </span><br><span>@@ -242,13 +296,14 @@</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite FAIL</span><br><span> ---------------------------------------------------------------------</span><br><span style="color: hsl(0, 100%, 40%);">-FAIL: test_suite (fail: 1, skip: 7)</span><br><span style="color: hsl(120, 100%, 40%);">+FAIL: test_suite (fail: 1, skip: 8)</span><br><span> skip: hello_world.py (N.N sec)</span><br><span> skip: mo_mt_sms.py</span><br><span> skip: mo_sms.py</span><br><span> skip: test_error.py (N.N sec)</span><br><span> skip: test_fail.py (N.N sec)</span><br><span> FAIL: test_fail_raise.py (N.N sec) ExpectedFail: This failure is expected</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_report_fragment.py (N.N sec)</span><br><span> skip: test_suite_params.py</span><br><span> skip: test_timeout.py</span><br><span> - test with half empty scenario</span><br><span>@@ -397,13 +452,14 @@</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite PASS</span><br><span> ---------------------------------------------------------------------</span><br><span style="color: hsl(0, 100%, 40%);">-PASS: test_suite (pass: 1, skip: 7)</span><br><span style="color: hsl(120, 100%, 40%);">+PASS: test_suite (pass: 1, skip: 8)</span><br><span> pass: hello_world.py (N.N sec)</span><br><span> skip: mo_mt_sms.py</span><br><span> skip: mo_sms.py</span><br><span> skip: test_error.py</span><br><span> skip: test_fail.py</span><br><span> skip: test_fail_raise.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_report_fragment.py</span><br><span> skip: test_suite_params.py</span><br><span> skip: test_timeout.py</span><br><span> - test with scenario</span><br><span>@@ -552,13 +608,14 @@</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite PASS</span><br><span> ---------------------------------------------------------------------</span><br><span style="color: hsl(0, 100%, 40%);">-PASS: test_suite (pass: 1, skip: 7)</span><br><span style="color: hsl(120, 100%, 40%);">+PASS: test_suite (pass: 1, skip: 8)</span><br><span> pass: hello_world.py (N.N sec)</span><br><span> skip: mo_mt_sms.py</span><br><span> skip: mo_sms.py</span><br><span> skip: test_error.py</span><br><span> skip: test_fail.py</span><br><span> skip: test_fail_raise.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_report_fragment.py</span><br><span> skip: test_suite_params.py</span><br><span> skip: test_timeout.py</span><br><span> - test with scenario and modifiers</span><br><span>@@ -753,13 +810,14 @@</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite PASS</span><br><span> ---------------------------------------------------------------------</span><br><span style="color: hsl(0, 100%, 40%);">-PASS: test_suite (pass: 1, skip: 7)</span><br><span style="color: hsl(120, 100%, 40%);">+PASS: test_suite (pass: 1, skip: 8)</span><br><span> pass: hello_world.py (N.N sec)</span><br><span> skip: mo_mt_sms.py</span><br><span> skip: mo_sms.py</span><br><span> skip: test_error.py</span><br><span> skip: test_fail.py</span><br><span> skip: test_fail_raise.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_report_fragment.py</span><br><span> skip: test_suite_params.py</span><br><span> skip: test_timeout.py</span><br><span> - test with suite-specific config</span><br><span>@@ -962,13 +1020,14 @@</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite FAIL</span><br><span> ---------------------------------------------------------------------</span><br><span style="color: hsl(0, 100%, 40%);">-FAIL: test_suite (fail: 1, pass: 1, skip: 6)</span><br><span style="color: hsl(120, 100%, 40%);">+FAIL: test_suite (fail: 1, pass: 1, skip: 7)</span><br><span> skip: hello_world.py</span><br><span> skip: mo_mt_sms.py</span><br><span> skip: mo_sms.py</span><br><span> skip: test_error.py</span><br><span> skip: test_fail.py</span><br><span> skip: test_fail_raise.py</span><br><span style="color: hsl(120, 100%, 40%);">+ skip: test_report_fragment.py</span><br><span> pass: test_suite_params.py (N.N sec)</span><br><span> FAIL: test_timeout.py (N.N sec) Error: test_timeout.py:[LINENR] Test Timeout triggered: 1 seconds elapsed [test_suite↪test_timeout.py:[LINENR]↪test_timeout.py]</span><br><span> - test with template overlay</span><br><span>diff --git a/selftest/suite_test/suite_test.ok.ign b/selftest/suite_test/suite_test.ok.ign</span><br><span>index 460da92..d8f0224 100644</span><br><span>--- a/selftest/suite_test/suite_test.ok.ign</span><br><span>+++ b/selftest/suite_test/suite_test.ok.ign</span><br><span>@@ -1,6 +1,11 @@</span><br><span> /[^ ]*/selftest/ [PATH]/selftest/</span><br><span> \.py:[0-9]* .py:[LINENR]</span><br><span> \([0-9.]+ sec\) (N.N sec)</span><br><span style="color: hsl(120, 100%, 40%);">+\([0-9.]+s\) (N.Ns)</span><br><span> {combining_scenarios='resources', scenario='foo'}:.* {combining_scenarios='resources', scenario='foo'}: [RESOURCE_DICT]</span><br><span> test_suite-[0-9]*-[0-9]* test_suite-[ID_NUM]-[ID_NUM]</span><br><span> suiteC-[0-9]*-[0-9]* suiteC-[ID_NUM]-[ID_NUM]</span><br><span style="color: hsl(120, 100%, 40%);">+line [0-9]+ line [LINENR]</span><br><span style="color: hsl(120, 100%, 40%);">+[0-9][0-9]:[0-9][0-9]:[0-9][0-9]\.[0-9]+ [TIMESTAMP]</span><br><span style="color: hsl(120, 100%, 40%);">+time="[0-9]+" time="[VAL]"</span><br><span style="color: hsl(120, 100%, 40%);">+timestamp="[^"]+" timestamp="[TIMESTAMP]"</span><br><span>diff --git a/selftest/suite_test/suite_test.py b/selftest/suite_test/suite_test.py</span><br><span>index 9708037..be4b3c4 100755</span><br><span>--- a/selftest/suite_test/suite_test.py</span><br><span>+++ b/selftest/suite_test/suite_test.py</span><br><span>@@ -3,6 +3,7 @@</span><br><span> import sys</span><br><span> import _prep</span><br><span> import shutil</span><br><span style="color: hsl(120, 100%, 40%);">+import re</span><br><span> from osmo_gsm_tester.core import log</span><br><span> from osmo_gsm_tester.core import config</span><br><span> from osmo_gsm_tester.core import util</span><br><span>@@ -11,6 +12,8 @@</span><br><span> from osmo_gsm_tester.core import suite</span><br><span> from osmo_gsm_tester.core.schema import generate_schemas, get_all_schema</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+import xml.etree.ElementTree as et</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> config.override_conf = os.path.join(os.path.dirname(sys.argv[0]), 'paths.conf')</span><br><span> </span><br><span> example_trial_dir = os.path.join('test_trial_tmp')</span><br><span>@@ -51,6 +54,15 @@</span><br><span> results = s.run_tests('hello_world.py')</span><br><span> print(report.suite_to_text(s))</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+print('- run report fragment test')</span><br><span style="color: hsl(120, 100%, 40%);">+results = s.run_tests('test_report_fragment.py')</span><br><span style="color: hsl(120, 100%, 40%);">+print(report.suite_to_text(s))</span><br><span style="color: hsl(120, 100%, 40%);">+xml = et.tostring(report.suite_to_junit(s)).decode('utf-8')</span><br><span style="color: hsl(120, 100%, 40%);">+xml = re.sub('Traceback.*raise', '[BACKTRACE]\nraise', xml, flags=re.M + re.DOTALL)</span><br><span style="color: hsl(120, 100%, 40%);">+print('\n\n################################### junit XML:\n'</span><br><span style="color: hsl(120, 100%, 40%);">+ + xml</span><br><span style="color: hsl(120, 100%, 40%);">+ + '\n###################################\n\n')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> log.style_change(src=True)</span><br><span> #log.style_change(trace=True)</span><br><span> print('\n- a test with an error')</span><br><span>diff --git a/selftest/suite_test/suitedirA/test_suite/test_report_fragment.py b/selftest/suite_test/suitedirA/test_suite/test_report_fragment.py</span><br><span>new file mode 100644</span><br><span>index 0000000..06ff37d</span><br><span>--- /dev/null</span><br><span>+++ b/selftest/suite_test/suitedirA/test_suite/test_report_fragment.py</span><br><span>@@ -0,0 +1,11 @@</span><br><span style="color: hsl(120, 100%, 40%);">+from osmo_gsm_tester.testenv import *</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+with test.report_fragment('fragment1'):</span><br><span style="color: hsl(120, 100%, 40%);">+ print('a step in the first fragment')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+with test.report_fragment('fragment2'):</span><br><span style="color: hsl(120, 100%, 40%);">+ print('a step in the second fragment')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+with test.report_fragment('fragment3'):</span><br><span style="color: hsl(120, 100%, 40%);">+ print('a step in the third fragment')</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('failure in the third fragment')</span><br><span>diff --git a/src/osmo_gsm_tester/core/report.py b/src/osmo_gsm_tester/core/report.py</span><br><span>index c3390fe..c5e185f 100644</span><br><span>--- a/src/osmo_gsm_tester/core/report.py</span><br><span>+++ b/src/osmo_gsm_tester/core/report.py</span><br><span>@@ -132,14 +132,37 @@</span><br><span> testsuite.set('time', str(math.ceil(suite.duration)))</span><br><span> testsuite.set('tests', str(len(suite.tests)))</span><br><span> passed, skipped, failed, errors = suite.count_test_results()</span><br><span style="color: hsl(0, 100%, 40%);">- testsuite.set('errors', str(errors))</span><br><span style="color: hsl(0, 100%, 40%);">- testsuite.set('failures', str(failed))</span><br><span style="color: hsl(0, 100%, 40%);">- testsuite.set('skipped', str(skipped))</span><br><span style="color: hsl(0, 100%, 40%);">- testsuite.set('disabled', str(skipped))</span><br><span> for suite_test in suite.tests:</span><br><span> testcase = test_to_junit(suite_test)</span><br><span> testcase.set('classname', suite.name())</span><br><span> testsuite.append(testcase)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for report_fragment in suite_test.report_fragments:</span><br><span style="color: hsl(120, 100%, 40%);">+ full_name = '%s/%s' % (suite_test.name(), report_fragment.name)</span><br><span style="color: hsl(120, 100%, 40%);">+ el = et.Element('testcase')</span><br><span style="color: hsl(120, 100%, 40%);">+ el.set('name', full_name)</span><br><span style="color: hsl(120, 100%, 40%);">+ el.set('time', str(math.ceil(report_fragment.duration)))</span><br><span style="color: hsl(120, 100%, 40%);">+ if report_fragment.result == test.Test.SKIP:</span><br><span style="color: hsl(120, 100%, 40%);">+ et.SubElement(el, 'skipped')</span><br><span style="color: hsl(120, 100%, 40%);">+ skipped += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ elif report_fragment.result == test.Test.FAIL:</span><br><span style="color: hsl(120, 100%, 40%);">+ failure = et.SubElement(el, 'failure')</span><br><span style="color: hsl(120, 100%, 40%);">+ failure.set('type', suite_test.fail_type or 'failure')</span><br><span style="color: hsl(120, 100%, 40%);">+ failed += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ elif report_fragment.result != test.Test.PASS:</span><br><span style="color: hsl(120, 100%, 40%);">+ error = et.SubElement(el, 'error')</span><br><span style="color: hsl(120, 100%, 40%);">+ error.text = 'could not run'</span><br><span style="color: hsl(120, 100%, 40%);">+ errors += 1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if report_fragment.output:</span><br><span style="color: hsl(120, 100%, 40%);">+ sout = et.SubElement(el, 'system-out')</span><br><span style="color: hsl(120, 100%, 40%);">+ sout.text = escape_xml_invalid_characters(report_fragment.output)</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.append(el)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.set('errors', str(errors))</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.set('failures', str(failed))</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.set('skipped', str(skipped))</span><br><span style="color: hsl(120, 100%, 40%);">+ testsuite.set('disabled', str(skipped))</span><br><span> return testsuite</span><br><span> </span><br><span> def test_to_junit(t):</span><br><span>diff --git a/src/osmo_gsm_tester/core/test.py b/src/osmo_gsm_tester/core/test.py</span><br><span>index ec85c45..3f86c6d 100644</span><br><span>--- a/src/osmo_gsm_tester/core/test.py</span><br><span>+++ b/src/osmo_gsm_tester/core/test.py</span><br><span>@@ -43,6 +43,7 @@</span><br><span> self._config_test_specific = config_test_specific</span><br><span> self.path = os.path.join(self.suite_run.definition.suite_dir, self.basename)</span><br><span> self.status = Test.UNKNOWN</span><br><span style="color: hsl(120, 100%, 40%);">+ self.report_fragments = []</span><br><span> self.start_timestamp = 0</span><br><span> self.duration = 0</span><br><span> self.fail_type = None</span><br><span>@@ -183,4 +184,55 @@</span><br><span> return ''</span><br><span> return lt.get_output(since_mark)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def report_fragment(self, name, result=None, **kwargs):</span><br><span style="color: hsl(120, 100%, 40%);">+ return Test.ReportFragment(parent_test=self, name=name, result=result, **kwargs)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ class ReportFragment:</span><br><span style="color: hsl(120, 100%, 40%);">+ '''Add additional test results in junit XML.</span><br><span style="color: hsl(120, 100%, 40%);">+ Convenient method that includes a test log:</span><br><span style="color: hsl(120, 100%, 40%);">+ with test.report_fragment('foo'):</span><br><span style="color: hsl(120, 100%, 40%);">+ do_test_steps()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ Or manually add a report fragment directly:</span><br><span style="color: hsl(120, 100%, 40%);">+ test.report_fragment('foo', result = test.PASS if worked else test.FAIL)</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __init__(self, parent_test, name, result=None, output=None, since_mark=None, start_time=0.0):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.parent_test = parent_test</span><br><span style="color: hsl(120, 100%, 40%);">+ self.name = name</span><br><span style="color: hsl(120, 100%, 40%);">+ self.result = Test.UNKNOWN</span><br><span style="color: hsl(120, 100%, 40%);">+ self.duration = 0.0</span><br><span style="color: hsl(120, 100%, 40%);">+ self.output = output</span><br><span style="color: hsl(120, 100%, 40%);">+ self.start_time = start_time</span><br><span style="color: hsl(120, 100%, 40%);">+ self.log_mark = since_mark</span><br><span style="color: hsl(120, 100%, 40%);">+ assert name not in (x.name for x in self.parent_test.report_fragments)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.parent_test.report_fragments.append(self)</span><br><span style="color: hsl(120, 100%, 40%);">+ if result is not None:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.got_result(result)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __str__(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ return '%s/%s/%s: %s (%.1fs)' % (self.parent_test.suite_run.name(),</span><br><span style="color: hsl(120, 100%, 40%);">+ self.parent_test.name(), self.name, self.result, self.duration)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __enter__(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.start_time = self.parent_test.elapsed_time()</span><br><span style="color: hsl(120, 100%, 40%);">+ self.log_mark = self.parent_test.get_log_mark()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __exit__(self, *exc_info):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.got_result(self.parent_test.PASS if exc_info[0] is None else self.parent_test.FAIL,</span><br><span style="color: hsl(120, 100%, 40%);">+ exc_info=exc_info)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def got_result(self, result, exc_info=None):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.result = result</span><br><span style="color: hsl(120, 100%, 40%);">+ self.duration = self.parent_test.elapsed_time() - self.start_time</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.log_mark is not None and self.output is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.output = self.parent_test.get_log_output(since_mark=self.log_mark)</span><br><span style="color: hsl(120, 100%, 40%);">+ if exc_info is not None and exc_info[0] is not None:</span><br><span style="color: hsl(120, 100%, 40%);">+ o = []</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.output:</span><br><span style="color: hsl(120, 100%, 40%);">+ o.append(self.output)</span><br><span style="color: hsl(120, 100%, 40%);">+ o.extend(traceback.format_exception(*exc_info))</span><br><span style="color: hsl(120, 100%, 40%);">+ self.output = '\n'.join(o)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.parent_test.log('----- Report fragment:', self)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> # vim: expandtab tabstop=4 shiftwidth=4</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/21516">change 21516</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/21516"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59 </div>
<div style="display:none"> Gerrit-Change-Number: 21516 </div>
<div style="display:none"> Gerrit-PatchSet: 4 </div>
<div style="display:none"> Gerrit-Owner: neels <nhofmeyr@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder </div>
<div style="display:none"> Gerrit-Reviewer: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>