osmo-ttcn3-hacks[master]: add compare-results.sh, call from start-testsuite.sh

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/gerrit-log@lists.osmocom.org/.

Neels Hofmeyr gerrit-no-reply at lists.osmocom.org
Thu Apr 5 21:40:04 UTC 2018


Patch Set 1:

for illustration, I tested the new msc fsm with this and got:

    Comparing expected results /osmo-ttcn3-hacks/msc/expected-results.log against results in junit-xml-105.log
    --------------------
    xfail->pass MSC_Tests.TC_cr_before_reset
    pass MSC_Tests.TC_lu_imsi_noauth_tmsi
    pass MSC_Tests.TC_lu_imsi_noauth_notmsi
    xfail MSC_Tests.TC_lu_imsi_reject
    xfail MSC_Tests.TC_lu_imsi_timeout_gsup
    pass MSC_Tests.TC_lu_imsi_auth_tmsi
    pass MSC_Tests.TC_cmserv_imsi_unknown
    pass MSC_Tests.TC_lu_and_mo_call
    pass MSC_Tests.TC_lu_auth_sai_timeout
    pass MSC_Tests.TC_lu_auth_sai_err
    pass->FAIL MSC_Tests.TC_lu_clear_request
    pass MSC_Tests.TC_lu_disconnect
    pass MSC_Tests.TC_lu_by_imei
    pass MSC_Tests.TC_lu_by_tmsi_noauth_unknown
    pass MSC_Tests.TC_imsi_detach_by_imsi
    pass MSC_Tests.TC_imsi_detach_by_tmsi
    pass MSC_Tests.TC_imsi_detach_by_imei
    xfail->pass MSC_Tests.TC_emerg_call_imei_reject
    pass MSC_Tests.TC_emerg_call_imsi
    xfail->pass MSC_Tests.TC_cm_serv_req_vgcs_reject
    xfail->pass MSC_Tests.TC_cm_serv_req_vbs_reject
    xfail->pass MSC_Tests.TC_cm_serv_req_lcs_reject
    xfail->pass MSC_Tests.TC_cm_reest_req_reject
    pass MSC_Tests.TC_lu_auth_2G_fail
    pass MSC_Tests.TC_lu_imsi_auth_tmsi_encr_13_13
    pass MSC_Tests.TC_cl3_no_payload
    xfail->pass MSC_Tests.TC_cl3_rnd_payload
    xfail MSC_Tests.TC_establish_and_nothing
    xfail MSC_Tests.TC_mo_setup_and_nothing
    pass MSC_Tests.TC_mo_crcx_ran_timeout
    xfail MSC_Tests.TC_mo_crcx_ran_reject
    xfail MSC_Tests.TC_mt_crcx_ran_reject
    pass MSC_Tests.TC_mo_setup_and_dtmf_dup
    xfail MSC_Tests.TC_gsup_cancel
    pass MSC_Tests.TC_lu_imsi_auth_tmsi_encr_1_13
    pass MSC_Tests.TC_lu_imsi_auth_tmsi_encr_3_13
    pass MSC_Tests.TC_lu_imsi_auth_tmsi_encr_3_1
    pass MSC_Tests.TC_lu_imsi_auth_tmsi_encr_3_1_no_cm
    pass MSC_Tests.TC_lu_imsi_auth_tmsi_encr_13_2
    pass MSC_Tests.TC_lu_imsi_auth_tmsi_encr_013_2
    pass MSC_Tests.TC_mo_release_timeout
    pass MSC_Tests.TC_lu_and_mt_call_no_dlcx_resp
    pass MSC_Tests.TC_reset_two
    pass MSC_Tests.TC_lu_and_mt_call
    pass MSC_Tests.TC_lu_imsi_auth_tmsi_encr_3_1_log_msc_debug
    --------------------
    30 pass
    7 xfail
    7 pass unexpectedly
       Update the expected results!
    1 FAIL
    
    FAILURE

So this tells me that I fixed 7 tests, but made one break. So I still need to look at that one:

    pass->FAIL MSC_Tests.TC_lu_clear_request

And if that one passes I haven't made anything worse (that is picked up by the test suite).

-- 
To view, visit https://gerrit.osmocom.org/7646
To unsubscribe, visit https://gerrit.osmocom.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I87d62a8be73d73a5eeff61a842e7c27a0066079d
Gerrit-PatchSet: 1
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Owner: Neels Hofmeyr <nhofmeyr at sysmocom.de>
Gerrit-Reviewer: Harald Welte <laforge at gnumonks.org>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: Neels Hofmeyr <nhofmeyr at sysmocom.de>
Gerrit-HasComments: No



More information about the gerrit-log mailing list