[PATCH] osmo-ttcn3-hacks[master]: add compare-results.sh, call from start-testsuite.sh

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/gerrit-log@lists.osmocom.org/.

Neels Hofmeyr gerrit-no-reply at lists.osmocom.org
Thu Apr 5 15:57:42 UTC 2018


Review at  https://gerrit.osmocom.org/7646

add compare-results.sh, call from start-testsuite.sh

Compare current test results to the expected results, and exit in error on
discrepancies.

Add compare-result.sh: (trivially) grep junit xml output to determine which
tests passed and which didn't, and compare against an expected-result.log,
another junit file from a previous run. Summarize and determine success.

Include an "xfail" feature: tests that are expected to fail are marked as
"xfail", unexpected failures as "FAIL".

In various subdirs, copy the current jenkins jobs' junit xml outputs as
expected-results.log, so that we will start getting useful output in both
jenkins runs and manual local runs.

In start-testsuite.sh, after running the tests, invoke the results comparison.

Due to the single-line parsing nature, the script so far does not distinguish
between error and failure. I doubt that we actually need to do that though.

Related: OS#3136
Change-Id: I87d62a8be73d73a5eeff61a842e7c27a0066079d
---
M .gitignore
A bsc/expected-results.log
A compare-results.sh
A ggsn_tests/expected-results.log
A hlr/expected-results.log
A mgw/expected-results.log
A msc/expected-results.log
A sgsn/expected-results.log
A sip/expected-results.log
M start-testsuite.sh
A sysinfo/expected-results.log
11 files changed, 633 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/46/7646/1

diff --git a/.gitignore b/.gitignore
index bf5baa0..b92be2f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,7 @@
 deps/*/
 *.o
 *.log
+!expected-results.log
 *.so
 compile
 */.gitignore
diff --git a/bsc/expected-results.log b/bsc/expected-results.log
new file mode 100644
index 0000000..18a9a20
--- /dev/null
+++ b/bsc/expected-results.log
@@ -0,0 +1,95 @@
+<?xml version="1.0"?>
+<testsuite name='BSC_Tests' tests='68' failures='4' errors='2' skipped='0' inconc='0' time='932.00'>
+  <testcase classname='BSC_Tests' name='TC_ctrl_msc_connection_status' time='2.267915'/>
+  <testcase classname='BSC_Tests' name='TC_ctrl_msc0_connection_status' time='2.225696'/>
+  <testcase classname='BSC_Tests' name='TC_ctrl' time='4.242800'/>
+  <testcase classname='BSC_Tests' name='TC_chan_act_noreply' time='2.229171'/>
+  <testcase classname='BSC_Tests' name='TC_chan_act_counter' time='2.226685'/>
+  <testcase classname='BSC_Tests' name='TC_chan_act_ack_noest' time='5.225675'/>
+  <testcase classname='BSC_Tests' name='TC_chan_act_ack_est_ind_noreply' time='24.248343'/>
+  <testcase classname='BSC_Tests' name='TC_chan_act_ack_est_ind_refused' time='4.231838'/>
+  <testcase classname='BSC_Tests' name='TC_chan_act_nack' time='2.728319'/>
+  <testcase classname='BSC_Tests' name='TC_chan_exhaustion' time='2.239509'/>
+  <testcase classname='BSC_Tests' name='TC_chan_rel_rll_rel_ind' time='8.276291'/>
+  <testcase classname='BSC_Tests' name='TC_chan_rel_conn_fail' time='8.290978'/>
+  <testcase classname='BSC_Tests' name='TC_chan_rel_hard_clear' time='7.431206'/>
+  <testcase classname='BSC_Tests' name='TC_chan_rel_hard_rlsd' time='4.451855'/>
+  <testcase classname='BSC_Tests' name='TC_chan_rel_a_reset' time='4.437509'/>
+  <testcase classname='BSC_Tests' name='TC_outbound_connect' time='2.230366'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_cic_only' time='2.432230'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_csd' time='2.435962'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_ctm' time='2.431946'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_sign' time='3.139859'>
+    <error type='DTE'></error>
+  </testcase>
+  <testcase classname='BSC_Tests' name='TC_assignment_fr_a5_0' time='5.362074'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_fr_a5_1' time='5.360999'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_fr_a5_1_codec_missing' time='3.371507'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_fr_a5_3' time='5.373823'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_fr_a5_4' time='5.385205'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_codec_fr' time='5.370247'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_codec_hr' time='5.370154'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_codec_efr' time='5.373357'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_codec_amr_f' time='5.369318'/>
+  <testcase classname='BSC_Tests' name='TC_assignment_codec_amr_h' time='5.362888'/>
+  <testcase classname='BSC_Tests' name='TC_rll_est_ind_inact_lchan' time='2.224803'/>
+  <testcase classname='BSC_Tests' name='TC_rll_est_ind_inval_sapi1' time='4.226237'/>
+  <testcase classname='BSC_Tests' name='TC_rll_est_ind_inval_sapi3' time='4.231547'/>
+  <testcase classname='BSC_Tests' name='TC_rll_est_ind_inval_sacch' time='4.246674'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan' time='8.753319'/>
+  <testcase classname='BSC_Tests' name='TC_paging_tmsi_nochan' time='8.759194'/>
+  <testcase classname='BSC_Tests' name='TC_paging_tmsi_any' time='8.750134'/>
+  <testcase classname='BSC_Tests' name='TC_paging_tmsi_sdcch' time='8.754159'/>
+  <testcase classname='BSC_Tests' name='TC_paging_tmsi_tch_f' time='8.758687'/>
+  <testcase classname='BSC_Tests' name='TC_paging_tmsi_tch_hf' time='9.289936'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_cgi' time='8.956272'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lac_ci' time='8.956649'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_ci' time='8.969455'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lai' time='8.763181'>
+    <failure type='fail-verdict'>Timeout expecting { msg_disc := { msg_group := RSL_MDISC_CCHAN (6), transparent := false }, msg_type := RSL_MT_PAGING_CMD (21), ies := { { iei := ?, body := { chan_nr := { u := { ch0 := RSL_CHAN_NR_PCH_AGCH (18) }, tn := ? } } }, { iei := ?, body := { paging_group := ? } }, { iei := ?, body := { ms_identity := { len := ?, payload := ? } } }, * } }
+      BSC_Tests.ttcn:2203 BSC_Tests control part
+      BSC_Tests.ttcn:1131 TC_paging_imsi_nochan_lai testcase
+    </failure>
+  </testcase>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lac' time='8.752368'>
+    <error type='DTE'></error>
+  </testcase>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_all' time='8.751176'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_plmn_lac_rnc' time='8.563604'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_rnc' time='8.551772'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lac_rnc' time='8.568040'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lacs' time='8.762731'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lacs_empty' time='8.548303'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_cgi_unknown_cid' time='8.557896'/>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_a_reset' time='8.759365'>
+    <failure type='fail-verdict'>Received PAGING after A-RESET
+      BSC_Tests.ttcn:2212 BSC_Tests control part
+      BSC_Tests.ttcn:1306 TC_paging_imsi_a_reset testcase
+    </failure>
+  </testcase>
+  <testcase classname='BSC_Tests' name='TC_paging_imsi_load' time='12.961436'/>
+  <testcase classname='BSC_Tests' name='TC_paging_counter' time='20.788631'/>
+  <testcase classname='BSC_Tests' name='TC_rsl_drop_counter' time='2.227479'/>
+  <testcase classname='BSC_Tests' name='TC_rsl_unknown_unit_id' time='0.010705'/>
+  <testcase classname='BSC_Tests' name='TC_oml_unknown_unit_id' time='0.010646'/>
+  <testcase classname='BSC_Tests' name='TC_classmark' time='3.185732'/>
+  <testcase classname='BSC_Tests' name='TC_unsol_ass_fail' time='8.149850'/>
+  <testcase classname='BSC_Tests' name='TC_unsol_ass_compl' time='8.155615'/>
+  <testcase classname='BSC_Tests' name='TC_unsol_ho_fail' time='8.147592'/>
+  <testcase classname='BSC_Tests' name='TC_err_82_short_msg' time='8.161174'/>
+  <testcase classname='BSC_Tests' name='TC_err_84_unknown_msg' time='3.174456'/>
+  <testcase classname='BSC_Tests' name='TC_ho_int' time='7.423814'/>
+  <testcase classname='BSC_Tests' name='TC_bssap_rlsd_does_not_cause_bssmap_reset' time='14.310727'/>
+  <testcase classname='BSC_Tests' name='TC_bssmap_clear_does_not_cause_bssmap_reset' time='45.459198'>
+    <failure type='fail-verdict'>Timeout of T_guard
+      BSC_Tests.ttcn:2230 BSC_Tests control part
+      BSC_Tests.ttcn:2056 TC_bssmap_clear_does_not_cause_bssmap_reset testcase
+    </failure>
+  </testcase>
+  <testcase classname='BSC_Tests' name='TC_ms_rel_ind_does_not_cause_bssmap_reset' time='45.320202'>
+    <failure type='fail-verdict'>Timeout of T_guard
+      BSC_Tests.ttcn:2231 BSC_Tests control part
+      BSC_Tests.ttcn:2115 TC_ms_rel_ind_does_not_cause_bssmap_reset testcase
+    </failure>
+  </testcase>
+</testsuite>
diff --git a/compare-results.sh b/compare-results.sh
new file mode 100755
index 0000000..cb80a0d
--- /dev/null
+++ b/compare-results.sh
@@ -0,0 +1,198 @@
+#!/usr/bin/env bash
+expected_file="$1"
+results_file="$2"
+
+usage() {
+  echo "
+Usage:
+
+  $(basename "$0") expected_results.junit-log current_results.junit-log [--allow-* [...]]
+
+Return 0 if the expected results match the current results exactly.
+
+  --allow-skip   Allow runnning less tests than are listed in the expected file.
+                 Default is to return failure on any skipped tests.
+  --allow-new    Allow more test results than found in the expected file.
+                 Default is to return failure on any unknown tests.
+  --allow-xpass  If a test was expected to fail but passed, return success.
+                 Default is to return failure on any mismatch.
+"
+}
+
+if [ ! -f "$expected_file" ]; then
+  usage
+  echo "Expected file not found: '$expected_file'"
+  exit 1
+fi
+
+if [ ! -f "$results_file" ]; then
+  usage
+  echo "Current results file not found: '$results_file'"
+  exit 1
+fi
+
+shift
+shift
+
+allow_xpass=0
+allow_skip=0
+allow_new=0
+
+while test -n "$1"; do
+  arg="$1"
+  if [ "x$arg" = "x--allow-xpass" ]; then
+    allow_xpass=1
+  elif [ "x$arg" = "x--allow-skip" ]; then
+    allow_skip=1
+  elif [ "x$arg" = "x--allow-new" ]; then
+    allow_new=1
+  else
+    usage
+    echo "Unknown argument: '$arg'"
+    exit 1
+  fi
+  shift
+done
+
+echo "Comparing expected results $expected_file against results in $results_file
+--------------------"
+
+parse_testcase() {
+  line="$1"
+  suite_name="$(echo "$line" | sed 's,.*classname='"'"'\([^'"'"']*\)'"'"'.*,\1,')"
+  test_name="$(echo "$line" | sed 's,.*\<name='"'"'\([^'"'"']*\)'"'"'.*,\1,')"
+  if [ -n "$(echo "$line" | grep '/>$')" ]; then
+    test_result="pass"
+  else
+    test_result="FAIL"
+  fi
+}
+
+pass=0
+xfail=0
+more_failures=0
+more_successes=0
+skipped=0
+new=0
+
+while read line; do
+  parse_testcase "$line"
+  exp_suite_name="$suite_name"
+  exp_test_name="$test_name"
+  exp_test_result="$test_result"
+  matched="0"
+
+  while read line; do
+    parse_testcase "$line"
+    if [ "x$exp_suite_name" != "x$suite_name" ]; then
+      continue
+    fi
+    if [ "x$exp_test_name" != "x$test_name" ]; then
+      continue
+    fi
+
+    if [ "x$exp_test_result" = "x$test_result" ]; then
+      if [ "x$exp_test_result" = "xFAIL" ]; then
+        exp_test_result="xfail"
+	(( xfail += 1 ))
+      else
+        (( pass += 1 ))
+      fi
+      echo "$exp_test_result $suite_name.$test_name"
+    else
+      if [ "x$exp_test_result" = "xFAIL" ]; then
+        exp_test_result="xfail"
+      fi
+      echo "$exp_test_result->$test_result $suite_name.$test_name"
+      if [ "x$test_result" = "xFAIL" ]; then
+        (( more_failures += 1 ))
+      else
+	(( more_successes += 1 ))
+      fi
+    fi
+    matched="1"
+    break
+  done <<< "$(grep "<testcase" "$results_file")"
+
+  if [ "x$matched" = "x0" ]; then
+    echo "skipped $exp_suite_name.$exp_test_name"
+    (( skipped += 1 ))
+  fi
+
+done <<< "$(grep "<testcase" "$expected_file")"
+
+# Also catch all new tests that aren't covered in the expected results
+while read line; do
+  parse_testcase "$line"
+  got_suite_name="$suite_name"
+  got_test_name="$test_name"
+  got_test_result="$test_result"
+  matched="0"
+
+  while read line; do
+    parse_testcase "$line"
+    if [ "x$got_suite_name" != "x$suite_name" ]; then
+      continue
+    fi
+    if [ "x$got_test_name" != "x$test_name" ]; then
+      continue
+    fi
+
+    matched="1"
+    break
+  done <<< "$(grep "<testcase" "$expected_file")"
+
+  if [ "x$matched" = "x0" ]; then
+    echo "NEW-$got_test_result $got_suite_name.$got_test_name"
+    (( new += 1 ))
+  fi
+
+done <<< "$(grep "<testcase" "$results_file")"
+
+echo "--------------------"
+overall_verdict=0
+
+if [ "x$pass" != x0 ]; then
+  echo "$pass pass"
+fi
+
+if [ "x$xfail" != x0 ]; then
+  echo "$xfail xfail"
+fi
+
+if [ "x$skipped" != x0 ]; then
+  echo "$skipped skipped"
+  if [ "x$allow_skip" = x0 ]; then
+    overall_verdict=4
+    echo "   Failing due to skipped tests. If tests were removed, update the expected results!"
+  fi
+fi
+
+if [ "x$new" != x0 ]; then
+  echo "$new new"
+  if [ "x$allow_new" = x0 ]; then
+    overall_verdict=3
+    echo "   Failing due to new tests. Update the expected results!"
+  fi
+fi
+
+if [ "x$more_successes" != x0 ]; then
+  echo "$more_successes pass unexpectedly"
+  if [ "x$allow_xpass" = x0 ]; then
+    overall_verdict=2
+    echo "   Update the expected results!"
+  fi
+fi
+
+if [ "x$more_failures" != x0 ]; then
+  echo "$more_failures FAIL"
+  overall_verdict=1
+fi
+
+echo
+if [ "x$overall_verdict" != x0 ]; then
+  echo "FAILURE"
+else
+  echo "ok"
+fi
+exit $overall_verdict
diff --git a/ggsn_tests/expected-results.log b/ggsn_tests/expected-results.log
new file mode 100644
index 0000000..bff953a
--- /dev/null
+++ b/ggsn_tests/expected-results.log
@@ -0,0 +1,14 @@
+<?xml version="1.0"?>
+<testsuite name='GGSN_Tests' tests='11' failures='0' errors='0' skipped='0' inconc='0' time='205.00'>
+  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact' time='0.006364'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_ipcp' time='0.003752'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_pcodns' time='0.003510'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_gtpu_access' time='6.011141'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp4_clients_interact' time='0.006517'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact' time='0.003312'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_pcodns' time='0.003403'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_icmp6' time='0.005114'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_gtpu_access' time='9.018065'/>
+  <testcase classname='GGSN_Tests' name='TC_pdp6_clients_interact' time='6.014695'/>
+  <testcase classname='GGSN_Tests' name='TC_echo_req_resp' time='0.001674'/>
+</testsuite>
diff --git a/hlr/expected-results.log b/hlr/expected-results.log
new file mode 100644
index 0000000..be8000c
--- /dev/null
+++ b/hlr/expected-results.log
@@ -0,0 +1,23 @@
+<?xml version="1.0"?>
+<testsuite name='Titan' tests='10' failures='2' errors='0' skipped='0' inconc='0' time='118.00'>
+  <testcase classname='HLR_Tests' name='TC_gsup_sai_err_invalid_imsi' time='0.008962'>
+    <failure type='fail-verdict'>Unexpected SAI ERROR Cause
+      HLR_Tests.ttcn:558 HLR_Tests control part
+      HLR_Tests.ttcn:364 TC_gsup_sai_err_invalid_imsi testcase
+    </failure>
+  </testcase>
+  <testcase classname='HLR_Tests' name='TC_gsup_sai' time='0.431623'/>
+  <testcase classname='HLR_Tests' name='TC_gsup_ul_unknown_imsi' time='0.009124'/>
+  <testcase classname='HLR_Tests' name='TC_gsup_sai_err_unknown_imsi' time='0.008056'/>
+  <testcase classname='HLR_Tests' name='TC_gsup_ul' time='0.402349'/>
+  <testcase classname='HLR_Tests' name='TC_vty' time='0.151754'/>
+  <testcase classname='HLR_Tests' name='TC_vty_msisdn_isd' time='5.076033'>
+    <failure type='fail-verdict'>Timeout waiting for ISD.req
+      HLR_Tests.ttcn:564 HLR_Tests control part
+      HLR_Tests.ttcn:485 TC_vty_msisdn_isd testcase
+    </failure>
+  </testcase>
+  <testcase classname='HLR_Tests' name='TC_gsup_purge_cs' time='0.459958'/>
+  <testcase classname='HLR_Tests' name='TC_gsup_purge_ps' time='0.467724'/>
+  <testcase classname='HLR_Tests' name='TC_gsup_purge_unknown' time='0.008034'/>
+</testsuite>
diff --git a/mgw/expected-results.log b/mgw/expected-results.log
new file mode 100644
index 0000000..c67d5d9
--- /dev/null
+++ b/mgw/expected-results.log
@@ -0,0 +1,36 @@
+<?xml version="1.0"?>
+<testsuite name='Titan' tests='26' failures='1' errors='0' skipped='1' inconc='0' time='177.00'>
+  <testcase classname='MGCP_Test' name='TC_selftest' time='0.002990'>
+    <skipped>no verdict</skipped>
+  </testcase>
+  <testcase classname='MGCP_Test' name='TC_crcx' time='0.008097'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_noprefix' time='0.007103'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_unsupp_mode' time='0.006485'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_early_bidir_mode' time='0.023678'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_unsupp_param' time='0.005827'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_missing_callid' time='0.006864'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_missing_mode' time='0.006328'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_unsupp_packet_intv' time='0.005716'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_illegal_double_lco' time='0.014705'>
+    <failure type='fail-verdict'>
+      MGCP_Test.ttcn:887 MGCP_Test control part
+      MGCP_Test.ttcn:436 TC_crcx_illegal_double_lco testcase
+    </failure>
+  </testcase>
+  <testcase classname='MGCP_Test' name='TC_crcx_sdp' time='0.008481'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_wildcarded' time='0.006528'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_wildcarded_exhaust' time='0.033504'/>
+  <testcase classname='MGCP_Test' name='TC_mdcx_without_crcx' time='0.023214'/>
+  <testcase classname='MGCP_Test' name='TC_dlcx_without_crcx' time='0.006506'/>
+  <testcase classname='MGCP_Test' name='TC_mdcx_wildcarded' time='0.006675'/>
+  <testcase classname='MGCP_Test' name='TC_dlcx_wildcarded' time='0.022936'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_and_dlcx_ep_callid_connid' time='0.006310'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_and_dlcx_ep_callid' time='0.006509'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_and_dlcx_ep' time='0.006822'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_and_dlcx_ep_callid_inval' time='0.007519'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_and_dlcx_ep_callid_connid_inval' time='0.006912'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_and_dlcx_retrans' time='0.006681'/>
+  <testcase classname='MGCP_Test' name='TC_crcx_dlcx_30ep' time='0.032986'/>
+  <testcase classname='MGCP_Test' name='TC_rtpem_selftest' time='5.514538'/>
+  <testcase classname='MGCP_Test' name='TC_two_crcx_and_rtp' time='1.111828'/>
+</testsuite>
diff --git a/msc/expected-results.log b/msc/expected-results.log
new file mode 100644
index 0000000..d436978
--- /dev/null
+++ b/msc/expected-results.log
@@ -0,0 +1,110 @@
+<?xml version="1.0"?>
+<testsuite name='Titan' tests='45' failures='12' errors='1' skipped='0' inconc='0' time='497.00'>
+  <testcase classname='MSC_Tests' name='TC_cr_before_reset' time='4.009127'>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_noauth_tmsi' time='1.035697'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_noauth_notmsi' time='1.045502'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_reject' time='6.041282'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1829 MSC_Tests control part
+      MSC_Tests.ttcn:502 TC_lu_imsi_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_timeout_gsup' time='11.044559'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1830 MSC_Tests control part
+      MSC_Tests.ttcn:531 TC_lu_imsi_timeout_gsup testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi' time='1.036704'/>
+  <testcase classname='MSC_Tests' name='TC_cmserv_imsi_unknown' time='1.021692'/>
+  <testcase classname='MSC_Tests' name='TC_lu_and_mo_call' time='4.291881'/>
+  <testcase classname='MSC_Tests' name='TC_lu_auth_sai_timeout' time='6.032964'/>
+  <testcase classname='MSC_Tests' name='TC_lu_auth_sai_err' time='1.028503'/>
+  <testcase classname='MSC_Tests' name='TC_lu_clear_request' time='2.029726'/>
+  <testcase classname='MSC_Tests' name='TC_lu_disconnect' time='2.028908'/>
+  <testcase classname='MSC_Tests' name='TC_lu_by_imei' time='6.030599'/>
+  <testcase classname='MSC_Tests' name='TC_lu_by_tmsi_noauth_unknown' time='1.246818'/>
+  <testcase classname='MSC_Tests' name='TC_imsi_detach_by_imsi' time='1.238427'/>
+  <testcase classname='MSC_Tests' name='TC_imsi_detach_by_tmsi' time='1.228194'/>
+  <testcase classname='MSC_Tests' name='TC_imsi_detach_by_imei' time='1.233577'/>
+  <testcase classname='MSC_Tests' name='TC_emerg_call_imei_reject' time='6.025962'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1843 MSC_Tests control part
+      MSC_Tests.ttcn:903 TC_emerg_call_imei_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_emerg_call_imsi' time='4.295068'/>
+  <testcase classname='MSC_Tests' name='TC_cm_serv_req_vgcs_reject' time='6.251082'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1845 MSC_Tests control part
+      MSC_Tests.ttcn:940 TC_cm_serv_req_vgcs_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_cm_serv_req_vbs_reject' time='6.249901'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1846 MSC_Tests control part
+      MSC_Tests.ttcn:961 TC_cm_serv_req_vbs_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_cm_serv_req_lcs_reject' time='6.244740'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1847 MSC_Tests control part
+      MSC_Tests.ttcn:982 TC_cm_serv_req_lcs_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_cm_reest_req_reject' time='6.250134'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1848 MSC_Tests control part
+      MSC_Tests.ttcn:1003 TC_cm_reest_req_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_lu_auth_2G_fail' time='1.042594'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi_encr_13_13' time='1.052763'/>
+  <testcase classname='MSC_Tests' name='TC_cl3_no_payload' time='1.021188'/>
+  <testcase classname='MSC_Tests' name='TC_cl3_rnd_payload' time='5.996049'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand or SCCP Release
+      MSC_Tests.ttcn:1852 MSC_Tests control part
+      MSC_Tests.ttcn:1125 TC_cl3_rnd_payload testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_establish_and_nothing' time='6.263255'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1853 MSC_Tests control part
+      MSC_Tests.ttcn:1142 TC_establish_and_nothing testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_mo_setup_and_nothing' time='31.481429'>
+    <failure type='fail-verdict'>Timeout waiting for ClearCommand/Release
+      MSC_Tests.ttcn:1854 MSC_Tests control part
+      MSC_Tests.ttcn:1166 TC_mo_setup_and_nothing testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_mo_crcx_ran_timeout' time='5.267724'/>
+  <testcase classname='MSC_Tests' name='TC_mo_crcx_ran_reject' time='31.279836'>
+    <failure type='fail-verdict'>Timeout waiting for channel release
+      MSC_Tests.ttcn:1856 MSC_Tests control part
+      MSC_Tests.ttcn:1283 TC_mo_crcx_ran_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_mt_crcx_ran_reject' time='31.062826'>
+    <failure type='fail-verdict'>Timeout waiting for channel release
+      MSC_Tests.ttcn:1857 MSC_Tests control part
+      MSC_Tests.ttcn:1380 TC_mt_crcx_ran_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_mo_setup_and_dtmf_dup' time='2.468132'/>
+  <testcase classname='MSC_Tests' name='TC_gsup_cancel' time='1.259044'>
+    <error type='DTE'></error>
+  </testcase>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi_encr_1_13' time='1.049565'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi_encr_3_13' time='1.054199'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi_encr_3_1' time='1.034815'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi_encr_3_1_no_cm' time='1.038335'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi_encr_13_2' time='1.042416'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi_encr_013_2' time='1.050732'/>
+  <testcase classname='MSC_Tests' name='TC_mo_release_timeout' time='25.264303'/>
+  <testcase classname='MSC_Tests' name='TC_lu_and_mt_call_no_dlcx_resp' time='10.273413'/>
+  <testcase classname='MSC_Tests' name='TC_reset_two' time='2.999986'/>
+  <testcase classname='MSC_Tests' name='TC_lu_and_mt_call' time='4.284560'/>
+  <testcase classname='MSC_Tests' name='TC_lu_imsi_auth_tmsi_encr_3_1_log_msc_debug' time='1.040196'/>
+</testsuite>
diff --git a/sgsn/expected-results.log b/sgsn/expected-results.log
new file mode 100644
index 0000000..502dde8
--- /dev/null
+++ b/sgsn/expected-results.log
@@ -0,0 +1,54 @@
+<?xml version="1.0"?>
+<testsuite name='Titan' tests='22' failures='5' errors='2' skipped='0' inconc='0' time='491.00'>
+  <testcase classname='SGSN_Tests' name='TC_attach' time='7.164349'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_mnc3' time='7.007076'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_auth_id_timeout' time='35.991976'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_auth_sai_timeout' time='36.007949'>
+    <failure type='fail-verdict'>Tguard timeout
+      SGSN_Tests.ttcn:1157 SGSN_Tests control part
+      SGSN_Tests.ttcn:480 TC_attach_auth_sai_timeout testcase
+    </failure>
+  </testcase>
+  <testcase classname='SGSN_Tests' name='TC_attach_auth_sai_reject' time='36.020705'>
+    <failure type='fail-verdict'>Tguard timeout
+      SGSN_Tests.ttcn:1158 SGSN_Tests control part
+      SGSN_Tests.ttcn:501 TC_attach_auth_sai_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='SGSN_Tests' name='TC_attach_gsup_lu_timeout' time='37.025962'>
+    <failure type='fail-verdict'>Tguard timeout
+      SGSN_Tests.ttcn:1159 SGSN_Tests control part
+      SGSN_Tests.ttcn:529 TC_attach_gsup_lu_timeout testcase
+    </failure>
+  </testcase>
+  <testcase classname='SGSN_Tests' name='TC_attach_gsup_lu_reject' time='37.024621'>
+    <failure type='fail-verdict'>Tguard timeout
+      SGSN_Tests.ttcn:1160 SGSN_Tests control part
+      SGSN_Tests.ttcn:558 TC_attach_gsup_lu_reject testcase
+    </failure>
+  </testcase>
+  <testcase classname='SGSN_Tests' name='TC_attach_combined' time='7.007181'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_accept_all' time='7.995753'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_closed' time='8.017589'/>
+  <testcase classname='SGSN_Tests' name='TC_rau_unknown' time='6.998580'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_rau' time='7.008848'/>
+  <testcase classname='SGSN_Tests' name='TC_detach_unknown_nopoweroff' time='6.993647'>
+    <error type='DTE'>Dynamic test case error: Error message was received from MC: The connect operation refers to test component with component reference 77, which has already terminated.</error>
+  </testcase>
+  <testcase classname='SGSN_Tests' name='TC_detach_unknown_poweroff' time='6.982069'>
+    <error type='DTE'>Dynamic test case error: Error message was received from MC: The connect operation refers to test component with component reference 83, which has already terminated.</error>
+  </testcase>
+  <testcase classname='SGSN_Tests' name='TC_detach_nopoweroff' time='7.010464'/>
+  <testcase classname='SGSN_Tests' name='TC_detach_poweroff' time='12.011997'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_pdp_act' time='6.050101'/>
+  <testcase classname='SGSN_Tests' name='TC_pdp_act_unattached' time='5.999444'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_pdp_act_user' time='6.053814'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_pdp_act_ggsn_reject' time='6.038840'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_pdp_act_user_deact_mo' time='6.041516'/>
+  <testcase classname='SGSN_Tests' name='TC_attach_pdp_act_user_deact_mt' time='36.021021'>
+    <failure type='fail-verdict'>Tguard timeout
+      SGSN_Tests.ttcn:1175 SGSN_Tests control part
+      SGSN_Tests.ttcn:1148 TC_attach_pdp_act_user_deact_mt testcase
+    </failure>
+  </testcase>
+</testsuite>
diff --git a/sip/expected-results.log b/sip/expected-results.log
new file mode 100644
index 0000000..54123a5
--- /dev/null
+++ b/sip/expected-results.log
@@ -0,0 +1,7 @@
+<?xml version="1.0"?>
+<testsuite name='Titan' tests='4' failures='0' errors='0' skipped='0' inconc='0' time='106.00'>
+  <testcase classname='SIP_Tests' name='TC_mt_success_rel_gsm' time='5.032948'/>
+  <testcase classname='SIP_Tests' name='TC_mt_success_rel_sip' time='5.032518'/>
+  <testcase classname='SIP_Tests' name='TC_mo_success_rel_gsm' time='5.032548'/>
+  <testcase classname='SIP_Tests' name='TC_mo_success_rel_sip' time='5.059944'/>
+</testsuite>
diff --git a/start-testsuite.sh b/start-testsuite.sh
index 8affdba..5b187fa 100755
--- a/start-testsuite.sh
+++ b/start-testsuite.sh
@@ -10,7 +10,9 @@
 fi
 
 SUITE=$1
-CFG=$(basename "$SUITE").cfg
+SUITE_DIR="$(dirname "$SUITE")"
+SUITE_NAME="$(basename "$SUITE")"
+CFG="$SUITE_NAME.cfg"
 if [ $# -gt 1 ]; then
 	CFG=$2
 fi
@@ -19,4 +21,26 @@
 	TEST=$3
 fi
 
-LD_LIBRARY_PATH=$(dirname "$SUITE"):/usr/lib/titan:/usr/ttcn3/lib ttcn3_start $SUITE $CFG $TEST
+LD_LIBRARY_PATH="$SUITE_DIR:/usr/lib/titan:/usr/ttcn3/lib" ttcn3_start $SUITE $CFG $TEST
+
+expected="$SUITE_DIR/expected-results.log"
+if [ ! -f "$expected" ]; then
+  echo "No expected results found, not comparing outcome. ($expected)"
+  exit 0
+fi
+
+# find the most recent junit output log here
+last_log="$(ls -1tr junit*.log | tail -n 1)"
+if [ ! -f "$last_log" ]; then
+  echo "No junit log found."
+  exit 1
+fi
+
+compare="$SUITE_DIR/../compare-results.sh"
+if [ ! -x "$compare" ]; then
+  echo "ERROR: cannot find $compare"
+  exit 1
+fi
+
+set -e
+"$compare" "$expected" "$last_log" $OSMO_TTCN3_COMPARE_ARGS
diff --git a/sysinfo/expected-results.log b/sysinfo/expected-results.log
new file mode 100644
index 0000000..cea6f54
--- /dev/null
+++ b/sysinfo/expected-results.log
@@ -0,0 +1,69 @@
+<?xml version="1.0"?>
+<testsuite name='osmo-ttcn3-hacks/sysinfo' tests='22' failures='3' errors='16' skipped='0' inconc='0' time='200.00'>
+  <testcase classname='Test' name='TC_si1' time='0.001430'/>
+  <testcase classname='Test' name='TC_telnet' time='0.004703'/>
+  <testcase classname='Test' name='TC_si_default' time='8.009958'>
+    <failure type='fail-verdict'>No SI messages for TC=0!
+
+    </failure>
+  </testcase>
+  <testcase classname='Test' name='TC_si_sched_2bis' time='10.034880'>
+    <failure type='fail-verdict'>No SI messages for TC=0!
+
+    </failure>
+  </testcase>
+  <testcase classname='Test' name='TC_si_sched_2ter' time='10.035191'>
+    <failure type='fail-verdict'>No SI messages for TC=0!
+
+    </failure>
+  </testcase>
+  <testcase classname='Test' name='TC_si_sched_2ter_2bis' time='9.020499'>
+    <error type='DTE'>Test.ttcn:162: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_si_sched_2quater' time='2.027051'/>
+  <testcase classname='Test' name='TC_si_sched_13' time='9.024905'>
+    <error type='DTE'>Test.ttcn:162: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_si_sched_13_2bis_2ter_2quater' time='9.024221'>
+    <error type='DTE'>Test.ttcn:162: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_neci' time='11.005209'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_cell_resel_hyst' time='11.020545'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_rxlev_acc_min' time='11.016459'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_cellid' time='11.014834'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_lac' time='11.014391'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_rach_tx_int' time='11.016352'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_rach_max_tx' time='11.014782'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_attach' time='11.005593'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_dtx_ul' time='10.013151'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_emerg_allowed' time='11.014455'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_bs_pa_mfrms' time='11.005825'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_bs_ag_blks_res' time='11.014585'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+  <testcase classname='Test' name='TC_radio_link_timeout' time='11.025433'>
+    <error type='DTE'>Test.ttcn:478: Dynamic test case error: testcase.stop</error>
+  </testcase>
+</testsuite>

-- 
To view, visit https://gerrit.osmocom.org/7646
To unsubscribe, visit https://gerrit.osmocom.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I87d62a8be73d73a5eeff61a842e7c27a0066079d
Gerrit-PatchSet: 1
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Owner: Neels Hofmeyr <nhofmeyr at sysmocom.de>



More information about the gerrit-log mailing list