Dear all,

I've started to implement some test cases that validate the accuracy of OsmoBTS and OsmoBSC SYSTEM INFORMATION generation.

Those test cases are written in TTCN-3, which is a special programming language specifically designed for testing and validation of protocol stacks. For those present at OsmoDevCon, I've briefly presented about it there.

In terms of general core infrastructure, I've written:

* Definition of GSM RR and some other commonly used GSM types using the TTCN-3 type description language and the TITAN RAW (binary) encoder/decoder. The beauty of this is that one can simply describe the message structure without having to generate any code for encoding/decoding the message. See http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/GSM_Types.ttcn and http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/GSM_SystemInformation.t...

* A GSMTAP Test Port, whcih is basically how a TTCN-3 testcase/testsuite interacts with a given IUT (implementation under test). It is implemented as dual-faced port on top of the TITAN IPL4asp (ip layer 4) port and implements GSMTAP header encoding/decoding from/to TTCN-3 structured data types: http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/GSMTAP_PortType.ttcn which uses http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/GSMTAP_Types.ttcn describing the GSMTAP header format.

* Helper functions to emulate interactive use of the VTY based on the existing TITAN TELNETasp Test Port: http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/Osmocom_VTY_Functions.t...

* L1CTL type definition (again using auto-generated encoder/decoders!) http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/L1CTL_Types.ttcn as well as an L1CTL Test Port at http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/L1CTL_PortType.ttcn Using the above one can easily interface from TTCN3 with L1CTL of virtphy in order to e.g. sync to a BTS on a given ARFCN and establish a dedicated radio channel. Here is where the beautoy of the TTCN3 type language, templates and matching can be seen. It's very easy to perform something like "does this IMMEDIATE ASSIGNMENT match the RA + FN of the RACH request that I sent?

Based on all of the above, there's two test suites:

== Test Suites

1) SYSTEM INFORMATION contents and scheduling validation

I've written some SI validation code which can be found at http://git.osmocom.org/osmo-ttcn3-hacks/tree/sysinfo

SI scheduling is quite complex, particularly when you have many of the 2ter/2bis/2quater/13/2n/... enabled. At some point actually you will overflow what's possible to comply with the rules in a BCCH Norm. and you have to go to a BCCH Extd. (OsmoBTS doesn't implement the latter yet, btw).

See e.g. function f_validate_si_scheduling() for validation of scheduling constraints: http://git.osmocom.org/osmo-ttcn3-hacks/tree/sysinfo/Test.ttcn#n229

Later in the file you can find test cases TC_cellid() and below for the individual test cases that validate that individual VTY configured values such as BSIC, CellId, LAC, ... are actually broadcast in the SI message. IT will update its internal matching template and then perform a sequence of VTY commands to e.g. change the LAC followed by verifying that the LAC of the decoded SI has actually changed.

The above tools have helped to discover (and fix) the following bugs: * http://osmocom.org/issues/2368 * http://osmocom.org/issues/2367 * http://osmocom.org/issues/2365 (very critical, I believe) and also the not-yet-fixed but low-importance * http://osmocom.org/issues/2364

2) LAPDm testing

I've written some (early) LAPDm testing code at http://git.osmocom.org/osmo-ttcn3-hacks/tree/lapdm

The idea here is to test for valid and invalid transactions on the Um interface and see how OsmoBTS and the libosmcore lapdm code perform.

With very few lines, this has already uncovered the following issues: * http://osmocom.org/issues/2370 * http://osmocom.org/issues/2380

Unfortunately my progress was severely affected over the weekend as I spent about one working day wrapping my head around something that turned out to be a bug in the Eclipse TITAN RAW coder, see https://www.eclipse.org/forums/index.php/t/1087557/ - fortuantely the Titan team has provided both a work-around and a fix today, which I count as excellent service.

If you're curious about TTCN-3, I strongly recommend http://www.ttcn-3.org/files/TTCN3_P.pdf as an introductory slide deck.

== Outlook

I want to add more test cases to the sysinfo and lapdm test suites and get hands-on experience with the Junit XML output plugin. At that point, a Debian 9 build slave of our jenkins should be able to automatically build + run the latest test cases from osmo-ttcn3-hacks.git and present the individual test case results inside jenkins.

I also want to add test suites for validation of: * proper generation + scheduling of of PAGING REQUEST in the BTS, considering paging groups particularly in the context of different BS_AG_BLKS_RES values * CBCH validation * proper generation of CCCH load indications by the BTS, i.e. generate a certian load e.g. on uplink RACH and check the reports * TA loop verification (add timing information to GSMTAP and then dynamically change timing from MS side to see that BTS is compensating as expected * power control loop verification: GSMTAP already has a field for transmitting RSSI in dBm. We can adjust this from the MS side to validate the BTS will adjus the instructed transmit power level of the virtual MS accordingly * actually exhaust all channels of a multi-TRX virtual BTS in different channel combinations, verify we can establish all existing logical channels in each configuration * combine the last topic with configurations that have dynamic PDCH/TCH channels to verify we properly switch forward and backward * various link failure conditions, i.e. does the BTS close the logical channel in the expected timing after uplink loss?

Despite my current excitement about all of this, I'm not sure how much time I will be able to spend on this, given that the sysmocom customer priority is typically not the development of automatic test suites. But I'll try my best.

If anyone wants to join: TITAN is just one "apt-get install eclipse-titan" away!

== What's not good to test with TTCN-3 / TITAN:

While the existing infrastructure I've built is very suitable to test low-level protocol topics, I think it's not particularly useful/applicable in other areas:

* SI REST OCTETS and GPRS RLC/MAC, as they're defined in CSN.1 and TITAN has no CSN.1 encoder/decoder. For easy mathing/generation of those messages, we'd have to find something like an external CSN.1-to-ASN.1-BER or CSN.1-to-XML converter which we could use. I'd be happy to consider paying a reasonable amount for a proprietary product, just as long as we are sure that it will provde the kind of productivity that's useful here.

* full end-to-end tests with actual mobile phones, RF and BTSs in the picture. We already have osmo-gsm-tester for this, and Titan with its powerful type language, templates and matching capabilities doesn't provide all that much benefit here over other languages/approaches

* higher-layer protocol testing. We do have OsmocomBB as a L2 + L3 MS-side implementation, and I don't think it makes sense to replicate this. Even while the message encoding/decoding is super easy with TITAN, we still have all the related state machines. So I'd rather extend OsmocomBB where needed for automatic test cases than reinvent the wheel here.

Regards, Harald