Hi all,
I was planning to do this long time ago, but somehow kept leaving this for later. Today I revisited the state of ttcn3-bts-test, which shows quite a few regressions. I believe they need to be investigated and eventually fixed, so I created several tickets for tracking these regressions in Osmocom's Redmine:
* OS#5951: TC_early_immediate_assignment, * OS#5952: TC_ho_physical_info, * OS#5953: TC_ms_pwr_ctrl_{constant,pf_ewma}, * OS#5954: TC_pcu_data_ind_lqual_cb, * OS#5955: TC_pcu_[ext_]rach_content, TC_pcu_ptcch, * OS#5956: TC_rsl_rf_resource_ind.
The following tickets already existed prior to my checkup:
* OS#4023: TC_pcu_oml_alert, * OS#5242: TC_ipa_crcx_ack_addr, * OS#5517: TC_tx_power_ramp_adm_state_change.
So far this is all about non-hopping configuration:
https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple...
The freq. hopping configuration exhibits slightly more red TCs:
https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple...
92% vs 84% passing TCs to be precise, but this is kinda expected because of the limitations we have in fake_trx: FAKE_* CTRL commands are applied to just one transceiver, not affecting others, which are also part of the Mobile Allocation. Mostly the meas related TCs are affected.
The following TC groups are quite stable and mostly all green:
https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple... https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple...
https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple... https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple...
The following two TC groups have never been stable/all green, AFAICT:
https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple... https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple...
We may want to create more tickets for those, maybe less granular (i.e. one ticket per group). I am not familiar with these two TC groups, so would be nice to get some input from those who write them. What would it take to get them fixed? Are there any tickets already? If could not find any, but I only did a quick search.
Best regards, Vadim.
Hi Vadim!
On Wed, Mar 22, 2023 at 12:01:52AM +0700, Vadim Yanitskiy wrote:
I was planning to do this long time ago, but somehow kept leaving this for later. Today I revisited the state of ttcn3-bts-test, which shows quite a few regressions. I believe they need to be investigated and eventually fixed, so I created several tickets for tracking these regressions in Osmocom's Redmine: [...]
Thanks for noticing and keeping track of it!
The following two TC groups have never been stable/all green, AFAICT:
https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple...
BTS_Tests_OML.TC_bts_opstart_noattr: /* RADIO CARRIER: Test OPSTART without SET BTS ATTRIBUTES; expect NACK */ error: Unexpected OPSTART ACK for NM_OC_BTS (1) { bts_nr := 0, trx_nr := 255, ts_nr := 255 } => So this looks like an actual osmo-bts bug that needs fixing => https://osmocom.org/issues/5961
BTS_Tests_OML.TC_initial_state_reports: /* Ensure the initial Event State Change Reports after OML connect match our expectation */ error: Timeout of T_guard => Not all expected state change event reports were received. Needs investigation if test or BTS to blame => https://osmocom.org/issues/5962
BTS_Tests_OML.TC_ipa_osmo_pcu_anr_fwd /* Make sure that the IUT forwards Container PCUIF messages between BSC and PCU */ error: Unexpected message received: { msg_type := PCU_IF_MSG_CONTAINER (128), bts_nr := 0, spare := '0000'O, u := { container := { msg_type := PCU_IF_MSG_NEIGH_ADDR_REQ (129), spare := '00'O, len := 300, u := { neigh_addr_req := { local_lac := 13712, local_ci := 49851, tgt_arfcn := 7662, tgt_bsic := 23 } } } } } vs exp: { data := { msg_type := PCU_IF_MSG_CONTAINER (128), bts_nr := 0, spare := '0000'O, u := { container := { msg_type := PCU_IF_MSG_NEIGH_ADDR_REQ (129), spare := '00'O, len := ?, u := { other := '3590C2BB1DEE1734F887C794E8ABF5EB0419C6E0ADA9F4A46C2875F4C064BEDBD4E089315117E14EFB5C09628C1451087EEB556D5B5694F6970306093A5ABECB4B6B10757027BADEBF947139024A510C6DCEAAC9DC7D6F328E4721690A9801BE54B327354E72A2E07DC50E31FA4CF36D658D5253DC7B212BF649281FDE009836BB103FF433FE6782FF67AD7909E8FDEFE8AB319359AE6F64A5E70845C7244449F7B28A5B6B482456E9649B3A21287A2B0FF6D9702D992D3855AF1261FB2224A887EC2B796AE62F857907208F44C0453341AB359F6F20CB3C02BEB90D17BAB507A24FFB7E157EE48623DA2F02F0F010DF7AA009C03B5E036076BBCD7B5E1E8B6AB302645C9D96D7CDA9EBC92CD25D77329BFBC8AE538D305B8186AD4076FA52C71EF3D4E385C0EE0919FD97D5'O } } } }, id := 0 } => Looks like the message cannot be decoded by the TITAN codec; maybe the protocol has changed but somebody forgot to update the TTCN3 test? => https://osmocom.org/issues/5963
BTS_Tests_OML.TC_ipa_rsl_connect_nack /* Make sure that the IUT sends RSL Connect NACK when the remote is not reachable. */ error: Timeout waiting for RSL Connect NACK => BTS cannot connect to RSL IP, but still ACKs to BSC. Minor issue, but actual bug in BTS => https://osmocom.org/issues/5964
BTS_Tests_OML.TC_radio_carrier_opstart_noattr /* RADIO CARRIER: Test OPSTART without SET BTS ATTRIBUTES; expect NACK */ error: Unexpected OPSTART ACK for NM_OC_RADIO_CARRIER (2) { bts_nr := 0, trx_nr := 0, ts_nr := 255 } => actual BTS bug => https://osmocom.org/issues/5961
BTS_Tests_OML.TC_ts_opstart /* BTS: Test OPSTART after SET BTS ATTRIBUTES; expect ACK */ error: none => Unknown => https://osmocom.org/issues/5965
BTS_Tests_OML.TC_ts_opstart_noattr /* BTS: Test OPSTART without SET BTS ATTRIBUTES; expect NACK */ error: Unexpected OPSTART ACK for NM_OC_CHANNEL (3) { bts_nr := 0, trx_nr := 0, ts_nr := 1 } => actual BTS bug => https://osmocom.org/issues/5961
BTS_Tests_OML.TC_wrong_obj_class /* test behavior for unsupported obj_class */ error: Timeout waiting for NACK => actual BTS bug => https://osmocom.org/issues/5966
BTS_Tests_OML.TC_wrong_trx_nr /* test behavior for wrong TRX number in object instance */ error: Timeout waiting for NACK => actual BTS bug => https://osmocom.org/issues/5967
https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/lastComple...
BTS_Tests_LAPDm.TC_nr_seq_error /* 25.2.6.2 nr sequence error */ error: Missing DISC from BTS => actual libosmocore LAPDm bug => https://osmocom.org/issues/5968
BTS_Tests_LAPDm.TC_ns_seq_error /* 25.2.6.1 ns sequence error error: Missing second REJ => actual libosmocore LAPDm bug => https://osmocom.org/issues/5969
BTS_Tests_LAPDm.TC_sabm_retransmit_bts /* we test that a re-transmitted SABM with identical payload will result in the retransmission of a * UA. This is required during the contention resolution procedure as specified in 8.4.1.4 */ error: Incorrect number of SABM re-transmissions of observed: 7 => actual libosmocore LAPDm bug => https://osmocom.org/issues/5970
BTS_Tests_LAPDm.TC_sabm_ua_dcch_sapi0_nopayload error: Initial SABM/UA must contain L3 payload but BTS accepts without => actual libosmocore LAPDm bug => https://osmocom.org/issues/5971
BTS_Tests_LAPDm.TC_t200_n200 error: "BTS_Tests.ttcn:721 : Tguard timeout" => likely actual libosmocore LAPDm bug => https://osmocom.org/issues/5972
-
Harald Welte -
Vadim Yanitskiy