<p>laforge <strong>submitted</strong> this change.</p><p><a href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15943">View Change</a></p><div style="white-space:pre-wrap">Approvals:
laforge: Looks good to me, but someone else must approve
pespin: Looks good to me, approved
Jenkins Builder: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">re-implement compare-results.sh as compare-results.py<br><br>The compare-results.sh is annoyingly slow. Since our ttcn3 tests containers<br>support Python 2, re-implement in Python for much quicker evaluation.<br><br>Change-Id: I0747c9d66ffc7e4121497a2416fca78d7b56c8e6<br>---<br>A compare-results.py<br>D compare-results.sh<br>M start-testsuite.sh<br>3 files changed, 138 insertions(+), 214 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/compare-results.py b/compare-results.py</span><br><span>new file mode 100755</span><br><span>index 0000000..d1adb20</span><br><span>--- /dev/null</span><br><span>+++ b/compare-results.py</span><br><span>@@ -0,0 +1,137 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/usr/bin/env python</span><br><span style="color: hsl(120, 100%, 40%);">+# Copyright 2018 sysmocom - s.f.m.c. GmbH</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+# Licensed under the Apache License, Version 2.0 (the "License");</span><br><span style="color: hsl(120, 100%, 40%);">+# you may not use this file except in compliance with the License.</span><br><span style="color: hsl(120, 100%, 40%);">+# You may obtain a copy of the License at</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+# http://www.apache.org/licenses/LICENSE-2.0</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+# Unless required by applicable law or agreed to in writing, software</span><br><span style="color: hsl(120, 100%, 40%);">+# distributed under the License is distributed on an "AS IS" BASIS,</span><br><span style="color: hsl(120, 100%, 40%);">+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span><br><span style="color: hsl(120, 100%, 40%);">+# See the License for the specific language governing permissions and</span><br><span style="color: hsl(120, 100%, 40%);">+# limitations under the License.</span><br><span style="color: hsl(120, 100%, 40%);">+import argparse</span><br><span style="color: hsl(120, 100%, 40%);">+import re</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+doc = "Compare TTCN3 test run results with expected results by junit logs."</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# The nicest would be to use an XML library, but I don't want to introduce dependencies on the build slaves.</span><br><span style="color: hsl(120, 100%, 40%);">+re_testcase = re.compile(r'''<testcase classname=['"]([^'"]+)['"].* name=['"]([^'"]+)['"].*>''')</span><br><span style="color: hsl(120, 100%, 40%);">+re_testcase_end = re.compile(r'''(</testcase>|<testcase [^>]*/>)''')</span><br><span style="color: hsl(120, 100%, 40%);">+re_failure = re.compile(r'''(<failure\b|<error\b)''')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+RESULT_PASS = 'pass'</span><br><span style="color: hsl(120, 100%, 40%);">+RESULT_FAIL = 'pass->FAIL'</span><br><span style="color: hsl(120, 100%, 40%);">+RESULT_SKIP = 'skip'</span><br><span style="color: hsl(120, 100%, 40%);">+RESULT_XFAIL = 'xfail'</span><br><span style="color: hsl(120, 100%, 40%);">+RESULT_FIXED = 'xfail->PASS'</span><br><span style="color: hsl(120, 100%, 40%);">+RESULT_NEW_PASS = 'NEW: PASS'</span><br><span style="color: hsl(120, 100%, 40%);">+RESULT_NEW_FAIL = 'NEW: FAIL'</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+RESULTS = (</span><br><span style="color: hsl(120, 100%, 40%);">+ RESULT_FAIL,</span><br><span style="color: hsl(120, 100%, 40%);">+ RESULT_NEW_FAIL,</span><br><span style="color: hsl(120, 100%, 40%);">+ RESULT_XFAIL,</span><br><span style="color: hsl(120, 100%, 40%);">+ RESULT_FIXED,</span><br><span style="color: hsl(120, 100%, 40%);">+ RESULT_PASS,</span><br><span style="color: hsl(120, 100%, 40%);">+ RESULT_NEW_PASS,</span><br><span style="color: hsl(120, 100%, 40%);">+ RESULT_SKIP,</span><br><span style="color: hsl(120, 100%, 40%);">+ )</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def count(counter, name, result):</span><br><span style="color: hsl(120, 100%, 40%);">+ v = counter.get(result) or 0</span><br><span style="color: hsl(120, 100%, 40%);">+ v += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ counter[result] = v</span><br><span style="color: hsl(120, 100%, 40%);">+ if result != RESULT_SKIP:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('%s %s' % (result, name))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def compare_one(name, expect, result, counter):</span><br><span style="color: hsl(120, 100%, 40%);">+ if result is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ count(counter, name, RESULT_SKIP)</span><br><span style="color: hsl(120, 100%, 40%);">+ elif result == RESULT_PASS:</span><br><span style="color: hsl(120, 100%, 40%);">+ if expect == RESULT_PASS:</span><br><span style="color: hsl(120, 100%, 40%);">+ count(counter, name, RESULT_PASS)</span><br><span style="color: hsl(120, 100%, 40%);">+ elif expect == RESULT_FAIL:</span><br><span style="color: hsl(120, 100%, 40%);">+ count(counter, name, RESULT_FIXED)</span><br><span style="color: hsl(120, 100%, 40%);">+ elif expect is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ count(counter, name, RESULT_NEW_PASS)</span><br><span style="color: hsl(120, 100%, 40%);">+ elif result == RESULT_FAIL:</span><br><span style="color: hsl(120, 100%, 40%);">+ if expect == RESULT_PASS:</span><br><span style="color: hsl(120, 100%, 40%);">+ count(counter, name, RESULT_FAIL)</span><br><span style="color: hsl(120, 100%, 40%);">+ elif expect == RESULT_FAIL:</span><br><span style="color: hsl(120, 100%, 40%);">+ count(counter, name, RESULT_XFAIL)</span><br><span style="color: hsl(120, 100%, 40%);">+ elif expect is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ count(counter, name, RESULT_NEW_FAIL)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def compare(cmdline, f_expected, f_current):</span><br><span style="color: hsl(120, 100%, 40%);">+ expected_list = parse_results(f_expected)</span><br><span style="color: hsl(120, 100%, 40%);">+ current_list = parse_results(f_current)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ expected_dict = dict(expected_list)</span><br><span style="color: hsl(120, 100%, 40%);">+ current_dict = dict(current_list)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ counter = {}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for expected_name, expected_result in expected_list:</span><br><span style="color: hsl(120, 100%, 40%);">+ compare_one(expected_name, expected_result, current_dict.get(expected_name), counter)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Also count new tests</span><br><span style="color: hsl(120, 100%, 40%);">+ for current_name, current_result in current_list:</span><br><span style="color: hsl(120, 100%, 40%);">+ if current_name in expected_dict:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ compare_one(current_name, None, current_result, counter)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ print('\nSummary:')</span><br><span style="color: hsl(120, 100%, 40%);">+ for r in RESULTS:</span><br><span style="color: hsl(120, 100%, 40%);">+ v = counter.get(r)</span><br><span style="color: hsl(120, 100%, 40%);">+ if not v:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ print(' %s: %d' % (r, v))</span><br><span style="color: hsl(120, 100%, 40%);">+ print('\n')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def parse_results(f):</span><br><span style="color: hsl(120, 100%, 40%);">+ tests = []</span><br><span style="color: hsl(120, 100%, 40%);">+ name = None</span><br><span style="color: hsl(120, 100%, 40%);">+ result = None</span><br><span style="color: hsl(120, 100%, 40%);">+ for line in f:</span><br><span style="color: hsl(120, 100%, 40%);">+ m = re_testcase.search(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ if m:</span><br><span style="color: hsl(120, 100%, 40%);">+ class_name, test_name = m.groups()</span><br><span style="color: hsl(120, 100%, 40%);">+ name = '%s.%s' % (class_name, test_name)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ m = re_failure.search(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ if m:</span><br><span style="color: hsl(120, 100%, 40%);">+ result = RESULT_FAIL</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ m = re_testcase_end.search(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ if m:</span><br><span style="color: hsl(120, 100%, 40%);">+ if not name:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ if result is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ result = RESULT_PASS</span><br><span style="color: hsl(120, 100%, 40%);">+ tests.append((name, result))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ name = None</span><br><span style="color: hsl(120, 100%, 40%);">+ result = None</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return tests</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def main(cmdline):</span><br><span style="color: hsl(120, 100%, 40%);">+ with open(cmdline.expected_results, 'r') as f_expected:</span><br><span style="color: hsl(120, 100%, 40%);">+ with open(cmdline.current_results, 'r') as f_current:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('\nComparing expected results %r against results in %r\n--------------------'</span><br><span style="color: hsl(120, 100%, 40%);">+ % (cmdline.expected_results, cmdline.current_results))</span><br><span style="color: hsl(120, 100%, 40%);">+ compare(cmdline, f_expected, f_current)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if __name__ == '__main__':</span><br><span style="color: hsl(120, 100%, 40%);">+ parser = argparse.ArgumentParser(description=doc)</span><br><span style="color: hsl(120, 100%, 40%);">+ parser.add_argument('expected_results', metavar='expected.junit-xml',</span><br><span style="color: hsl(120, 100%, 40%);">+ help='junit XML file listing the expected test results.')</span><br><span style="color: hsl(120, 100%, 40%);">+ parser.add_argument('current_results', metavar='current.junit-xml',</span><br><span style="color: hsl(120, 100%, 40%);">+ help='junit XML file listing the current test results.')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ cmdline = parser.parse_args()</span><br><span style="color: hsl(120, 100%, 40%);">+ main(cmdline)</span><br><span>diff --git a/compare-results.sh b/compare-results.sh</span><br><span>deleted file mode 100755</span><br><span>index 6037174..0000000</span><br><span>--- a/compare-results.sh</span><br><span>+++ /dev/null</span><br><span>@@ -1,213 +0,0 @@</span><br><span style="color: hsl(0, 100%, 40%);">-#!/usr/bin/env bash</span><br><span style="color: hsl(0, 100%, 40%);">-expected_file="$1"</span><br><span style="color: hsl(0, 100%, 40%);">-results_file="$2"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# Copyright 2018 sysmocom - s.f.m.c. GmbH</span><br><span style="color: hsl(0, 100%, 40%);">-#</span><br><span style="color: hsl(0, 100%, 40%);">-# Licensed under the Apache License, Version 2.0 (the "License");</span><br><span style="color: hsl(0, 100%, 40%);">-# you may not use this file except in compliance with the License.</span><br><span style="color: hsl(0, 100%, 40%);">-# You may obtain a copy of the License at</span><br><span style="color: hsl(0, 100%, 40%);">-#</span><br><span style="color: hsl(0, 100%, 40%);">-# http://www.apache.org/licenses/LICENSE-2.0</span><br><span style="color: hsl(0, 100%, 40%);">-#</span><br><span style="color: hsl(0, 100%, 40%);">-# Unless required by applicable law or agreed to in writing, software</span><br><span style="color: hsl(0, 100%, 40%);">-# distributed under the License is distributed on an "AS IS" BASIS,</span><br><span style="color: hsl(0, 100%, 40%);">-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span><br><span style="color: hsl(0, 100%, 40%);">-# See the License for the specific language governing permissions and</span><br><span style="color: hsl(0, 100%, 40%);">-# limitations under the License.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-usage() {</span><br><span style="color: hsl(0, 100%, 40%);">- echo "</span><br><span style="color: hsl(0, 100%, 40%);">-Usage:</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- $(basename "$0") expected_results.junit-log current_results.junit-log [--allow-* [...]]</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-Return 0 if the expected results match the current results exactly.</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- --allow-skip Allow runnning less tests than are listed in the expected file.</span><br><span style="color: hsl(0, 100%, 40%);">- Default is to return failure on any skipped tests.</span><br><span style="color: hsl(0, 100%, 40%);">- --allow-new Allow more test results than found in the expected file.</span><br><span style="color: hsl(0, 100%, 40%);">- Default is to return failure on any unknown tests.</span><br><span style="color: hsl(0, 100%, 40%);">- --allow-xpass If a test was expected to fail but passed, return success.</span><br><span style="color: hsl(0, 100%, 40%);">- Default is to return failure on any mismatch.</span><br><span style="color: hsl(0, 100%, 40%);">-"</span><br><span style="color: hsl(0, 100%, 40%);">-}</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ ! -f "$expected_file" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- usage</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Expected file not found: '$expected_file'"</span><br><span style="color: hsl(0, 100%, 40%);">- exit 1</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ ! -f "$results_file" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- usage</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Current results file not found: '$results_file'"</span><br><span style="color: hsl(0, 100%, 40%);">- exit 1</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-shift</span><br><span style="color: hsl(0, 100%, 40%);">-shift</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-allow_xpass=0</span><br><span style="color: hsl(0, 100%, 40%);">-allow_skip=0</span><br><span style="color: hsl(0, 100%, 40%);">-allow_new=0</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-while test -n "$1"; do</span><br><span style="color: hsl(0, 100%, 40%);">- arg="$1"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$arg" = "x--allow-xpass" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- allow_xpass=1</span><br><span style="color: hsl(0, 100%, 40%);">- elif [ "x$arg" = "x--allow-skip" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- allow_skip=1</span><br><span style="color: hsl(0, 100%, 40%);">- elif [ "x$arg" = "x--allow-new" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- allow_new=1</span><br><span style="color: hsl(0, 100%, 40%);">- else</span><br><span style="color: hsl(0, 100%, 40%);">- usage</span><br><span style="color: hsl(0, 100%, 40%);">- echo "Unknown argument: '$arg'"</span><br><span style="color: hsl(0, 100%, 40%);">- exit 1</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- shift</span><br><span style="color: hsl(0, 100%, 40%);">-done</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-echo "Comparing expected results $expected_file against results in $results_file</span><br><span>---------------------"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-parse_testcase() {</span><br><span style="color: hsl(0, 100%, 40%);">- line="$1"</span><br><span style="color: hsl(0, 100%, 40%);">- suite_name="$(echo "$line" | sed 's,.*classname='"'"'\([^'"'"']*\)'"'"'.*,\1,')"</span><br><span style="color: hsl(0, 100%, 40%);">- test_name="$(echo "$line" | sed 's,.*\<name='"'"'\([^'"'"']*\)'"'"'.*,\1,')"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ -n "$(echo "$line" | grep '/>$')" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- test_result="pass"</span><br><span style="color: hsl(0, 100%, 40%);">- else</span><br><span style="color: hsl(0, 100%, 40%);">- test_result="FAIL"</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-}</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-pass=0</span><br><span style="color: hsl(0, 100%, 40%);">-xfail=0</span><br><span style="color: hsl(0, 100%, 40%);">-more_failures=0</span><br><span style="color: hsl(0, 100%, 40%);">-more_successes=0</span><br><span style="color: hsl(0, 100%, 40%);">-skipped=0</span><br><span style="color: hsl(0, 100%, 40%);">-new=0</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-while read line; do</span><br><span style="color: hsl(0, 100%, 40%);">- parse_testcase "$line"</span><br><span style="color: hsl(0, 100%, 40%);">- exp_suite_name="$suite_name"</span><br><span style="color: hsl(0, 100%, 40%);">- exp_test_name="$test_name"</span><br><span style="color: hsl(0, 100%, 40%);">- exp_test_result="$test_result"</span><br><span style="color: hsl(0, 100%, 40%);">- matched="0"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- while read line; do</span><br><span style="color: hsl(0, 100%, 40%);">- parse_testcase "$line"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$exp_suite_name" != "x$suite_name" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- continue</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$exp_test_name" != "x$test_name" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- continue</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$exp_test_result" = "x$test_result" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$exp_test_result" = "xFAIL" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- exp_test_result="xfail"</span><br><span style="color: hsl(0, 100%, 40%);">- (( xfail += 1 ))</span><br><span style="color: hsl(0, 100%, 40%);">- else</span><br><span style="color: hsl(0, 100%, 40%);">- (( pass += 1 ))</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- echo "$exp_test_result $suite_name.$test_name"</span><br><span style="color: hsl(0, 100%, 40%);">- else</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$exp_test_result" = "xFAIL" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- exp_test_result="xfail"</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- echo "$exp_test_result->$test_result $suite_name.$test_name"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$test_result" = "xFAIL" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- (( more_failures += 1 ))</span><br><span style="color: hsl(0, 100%, 40%);">- else</span><br><span style="color: hsl(0, 100%, 40%);">- (( more_successes += 1 ))</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- matched="1"</span><br><span style="color: hsl(0, 100%, 40%);">- break</span><br><span style="color: hsl(0, 100%, 40%);">- done <<< "$(grep "<testcase.*$exp_test_name" "$results_file")"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$matched" = "x0" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "skipped $exp_suite_name.$exp_test_name"</span><br><span style="color: hsl(0, 100%, 40%);">- (( skipped += 1 ))</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-done <<< "$(grep "<testcase" "$expected_file")"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-# Also catch all new tests that aren't covered in the expected results</span><br><span style="color: hsl(0, 100%, 40%);">-while read line; do</span><br><span style="color: hsl(0, 100%, 40%);">- parse_testcase "$line"</span><br><span style="color: hsl(0, 100%, 40%);">- got_suite_name="$suite_name"</span><br><span style="color: hsl(0, 100%, 40%);">- got_test_name="$test_name"</span><br><span style="color: hsl(0, 100%, 40%);">- got_test_result="$test_result"</span><br><span style="color: hsl(0, 100%, 40%);">- matched="0"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- while read line; do</span><br><span style="color: hsl(0, 100%, 40%);">- parse_testcase "$line"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$got_suite_name" != "x$suite_name" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- continue</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$got_test_name" != "x$test_name" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- continue</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- matched="1"</span><br><span style="color: hsl(0, 100%, 40%);">- break</span><br><span style="color: hsl(0, 100%, 40%);">- done <<< "$(grep "<testcase.*$test_name" "$expected_file")"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$matched" = "x0" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "NEW-$got_test_result $got_suite_name.$got_test_name"</span><br><span style="color: hsl(0, 100%, 40%);">- (( new += 1 ))</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-done <<< "$(grep "<testcase" "$results_file")"</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-echo "--------------------"</span><br><span style="color: hsl(0, 100%, 40%);">-overall_verdict=0</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-ask_update=""</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ "x$pass" != x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "$pass pass"</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ "x$xfail" != x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "$xfail xfail"</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ "x$skipped" != x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "$skipped skipped"</span><br><span style="color: hsl(0, 100%, 40%);">- ask_update="$ask_update removed=$skipped"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$allow_skip" = x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- overall_verdict=4</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ "x$new" != x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "$new new"</span><br><span style="color: hsl(0, 100%, 40%);">- ask_update="$ask_update new=$new"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$allow_new" = x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- overall_verdict=3</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ "x$more_successes" != x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "$more_successes pass unexpectedly"</span><br><span style="color: hsl(0, 100%, 40%);">- ask_update="$ask_update xpass=$more_successes"</span><br><span style="color: hsl(0, 100%, 40%);">- if [ "x$allow_xpass" = x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- overall_verdict=2</span><br><span style="color: hsl(0, 100%, 40%);">- fi</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ "x$more_failures" != x0 ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo "$more_failures FAIL"</span><br><span style="color: hsl(0, 100%, 40%);">- overall_verdict=1</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-if [ -n "$ask_update" ]; then</span><br><span style="color: hsl(0, 100%, 40%);">- echo</span><br><span style="color: hsl(0, 100%, 40%);">- echo "(Please update the expected results:$ask_update)"</span><br><span style="color: hsl(0, 100%, 40%);">-fi</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span style="color: hsl(0, 100%, 40%);">-exit $overall_verdict</span><br><span>diff --git a/start-testsuite.sh b/start-testsuite.sh</span><br><span>index e5bc756..a6380b7 100755</span><br><span>--- a/start-testsuite.sh</span><br><span>+++ b/start-testsuite.sh</span><br><span>@@ -60,7 +60,7 @@</span><br><span> exit 1</span><br><span> fi</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-compare="$SUITE_DIR/../compare-results.sh"</span><br><span style="color: hsl(120, 100%, 40%);">+compare="$SUITE_DIR/../compare-results.py"</span><br><span> if [ ! -x "$compare" ]; then</span><br><span> echo "ERROR: cannot find $compare"</span><br><span> exit 1</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15943">change 15943</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15943"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-ttcn3-hacks </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I0747c9d66ffc7e4121497a2416fca78d7b56c8e6 </div>
<div style="display:none"> Gerrit-Change-Number: 15943 </div>
<div style="display:none"> Gerrit-PatchSet: 4 </div>
<div style="display:none"> Gerrit-Owner: neels <nhofmeyr@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder </div>
<div style="display:none"> Gerrit-Reviewer: laforge <laforge@osmocom.org> </div>
<div style="display:none"> Gerrit-Reviewer: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>