BSC+BTS SI + LAPD validation, TTCN-3 Testsuite

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/OpenBSC@lists.osmocom.org/.

Harald Welte laforge at gnumonks.org
Wed Jul 19 18:43:54 UTC 2017


Dear all,

I've started to implement some test cases that validate the accuracy of
OsmoBTS and OsmoBSC SYSTEM INFORMATION generation.

Those test cases are written in TTCN-3, which is a special programming
language specifically designed for testing and validation of protocol
stacks.  For those present at OsmoDevCon, I've briefly presented about
it there.

In terms of general core infrastructure, I've written:

* Definition of GSM RR and some other commonly used GSM types using the
  TTCN-3 type description language and the TITAN RAW (binary) encoder/decoder.
  The beauty of this is that one can simply describe the message
  structure without having to generate any code for encoding/decoding
  the message.  See 
  http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/GSM_Types.ttcn
  and
  http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/GSM_SystemInformation.ttcn

* A GSMTAP Test Port, whcih is basically how a TTCN-3 testcase/testsuite
  interacts with a given IUT (implementation under test).  It is
  implemented as dual-faced port on top of the TITAN IPL4asp (ip layer 4)
  port and implements GSMTAP header encoding/decoding from/to TTCN-3
  structured data types:
  http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/GSMTAP_PortType.ttcn
  which uses
  http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/GSMTAP_Types.ttcn
  describing the GSMTAP header format.

* Helper functions to emulate interactive use of the VTY based on the
  existing TITAN TELNETasp Test Port:
  http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/Osmocom_VTY_Functions.ttcn

* L1CTL type definition (again using auto-generated encoder/decoders!)
  http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/L1CTL_Types.ttcn
  as well as an L1CTL Test Port at 
  http://git.osmocom.org/osmo-ttcn3-hacks/tree/library/L1CTL_PortType.ttcn
  Using the above one can easily interface from TTCN3 with L1CTL of
  virtphy in order to e.g. sync to a BTS on a given ARFCN and establish
  a dedicated radio channel.  Here is where the beautoy of the TTCN3
  type language, templates and matching can be seen. It's very easy to
  perform something like "does this IMMEDIATE ASSIGNMENT match the RA +
  FN of the RACH request that I sent?

Based on all of the above, there's two test suites:


== Test Suites

1) SYSTEM INFORMATION contents and scheduling validation

I've written some SI validation code which can be found at
http://git.osmocom.org/osmo-ttcn3-hacks/tree/sysinfo

SI scheduling is quite complex, particularly when you have many of the
2ter/2bis/2quater/13/2n/... enabled.  At some point actually you will
overflow what's possible to comply with the rules in a BCCH Norm. and
you have to go to a BCCH Extd.  (OsmoBTS doesn't implement the latter
yet, btw).

See e.g. function f_validate_si_scheduling() for validation of scheduling
constraints:
  http://git.osmocom.org/osmo-ttcn3-hacks/tree/sysinfo/Test.ttcn#n229

Later in the file you can find test cases TC_cellid() and below for the
individual test cases that validate that individual VTY configured
values such as BSIC, CellId, LAC, ... are actually broadcast in the SI
message.  IT will update its internal matching template and then perform
a sequence of VTY commands to e.g. change the LAC followed by verifying
that the LAC of the decoded SI has actually changed.

The above tools have helped to discover (and fix) the following bugs:
* http://osmocom.org/issues/2368
* http://osmocom.org/issues/2367
* http://osmocom.org/issues/2365 (very critical, I believe)
and also the not-yet-fixed but low-importance
* http://osmocom.org/issues/2364


2) LAPDm testing

I've written some (early) LAPDm testing code at
http://git.osmocom.org/osmo-ttcn3-hacks/tree/lapdm

The idea here is to test for valid and invalid transactions on the Um
interface and see how OsmoBTS and the libosmcore lapdm code perform.

With very few lines, this has already uncovered the following issues:
* http://osmocom.org/issues/2370
* http://osmocom.org/issues/2380

Unfortunately my progress was severely affected over the weekend as I
spent about one working day wrapping my head around something that
turned out to be a bug in the Eclipse TITAN RAW coder, see
https://www.eclipse.org/forums/index.php/t/1087557/ - fortuantely the
Titan team has provided both a work-around and a fix today, which I
count as excellent service.

If you're curious about TTCN-3, I strongly recommend
http://www.ttcn-3.org/files/TTCN3_P.pdf as an introductory slide deck.


== Outlook

I want to add more test cases to the sysinfo and lapdm test suites and
get hands-on experience with the Junit XML output plugin.  At that
point, a Debian 9 build slave of our jenkins should be able to
automatically build + run the latest test cases from
osmo-ttcn3-hacks.git and present the individual test case results inside
jenkins.

I also want to add test suites for validation of:
* proper generation + scheduling of of PAGING REQUEST in the BTS,
  considering paging groups particularly in the context of different
  BS_AG_BLKS_RES values
* CBCH validation
* proper generation of CCCH load indications by the BTS, i.e. generate a
  certian load e.g. on uplink RACH and check the reports
* TA loop verification (add timing information to GSMTAP and then
  dynamically change timing from MS side to see that BTS is compensating
  as expected
* power control loop verification: GSMTAP already has a field for
  transmitting RSSI in dBm.  We can adjust this from the MS side to
  validate the BTS will adjus the instructed transmit power level of the
  virtual MS accordingly
* actually exhaust all channels of a multi-TRX virtual BTS in different
  channel combinations, verify we can establish all existing logical
  channels in each configuration
* combine the last topic with configurations that have dynamic PDCH/TCH
  channels to verify we properly switch forward and backward
* various link failure conditions, i.e. does the BTS close the logical
  channel in the expected timing after uplink loss?

Despite my current excitement about all of this, I'm not sure how much
time I will be able to spend on this, given that the sysmocom customer
priority is typically not the development of automatic test suites.  But
I'll try my best.  

If anyone wants to join: TITAN is just one "apt-get install eclipse-titan"
away!


== What's not good to test with TTCN-3 / TITAN:

While the existing infrastructure I've built is very suitable to test
low-level protocol topics, I think it's not particularly
useful/applicable in other areas:

* SI REST OCTETS and GPRS RLC/MAC, as they're defined in CSN.1 and TITAN
  has no CSN.1 encoder/decoder.  For easy mathing/generation of those
  messages, we'd have to find something like an external
  CSN.1-to-ASN.1-BER or CSN.1-to-XML converter which we could use.  I'd
  be happy to consider paying a reasonable amount for a proprietary
  product, just as long as we are sure that it will provde the kind of
  productivity that's useful here.

* full end-to-end tests with actual mobile phones, RF and BTSs in the
  picture.  We already have osmo-gsm-tester for this, and Titan with its
  powerful type language, templates and matching capabilities doesn't
  provide all that much benefit here over other languages/approaches

* higher-layer protocol testing.  We do have OsmocomBB as a L2 + L3
  MS-side implementation, and I don't think it makes sense to replicate
  this.  Even while the message encoding/decoding is super easy with
  TITAN, we still have all the related state machines.  So I'd rather
  extend OsmocomBB where needed for automatic test cases than reinvent
  the wheel here.


Regards,
	Harald
-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)



More information about the OpenBSC mailing list