Hello Osmocom CNI folks,
I got my first GSM network up and running, using nanoBTS hardware and Osmocom CNI sw stack. I compiled everything from source (to my knowledge, no one makes binary packages for Slackware), but rather than grabbing the state of git master on some random day, I checked out tagged versions corresponding to the latest official 2021-11 CNI release. My configuration is 2G-only, CS-only, minimal split-NITB, thus the 5 processes I am running are OsmoSTP, OsmoHLR, OsmoMGW, OsmoMSC and OsmoBSC. I'm using the same OsmoMGW instance for both MSC and BSC.
The functional state is: phones with FCSIM1 cards in them whose programming matches the data I entered into HLR are able to connect (location update succeeds), SMS from one phone to another works, and USSD requests *#100# and *#101# also work. However, voice calls don't work. The user-visible symptoms are:
1) When I dial a call from one connected phone to another, the destination phone starts ringing, and it shows the calling number on the display - thus CC signaling works correctly at least in the initial phases.
2) However, a short time (I measured a little under 20 s) after the start of the process, the destination phone stops ringing and shows a missed call indication. However, the origin phone continues playing its NOIBT ringing tone as if it is still waiting for the call to be answered, and it keeps going for another 15-20 s or so, before it gives up.
3) Trying to press the answer button on the destination phone before it stops ringing and goes into the "missed call" state does NOT produce a connected call state either.
To see what's going on, I turned on maximum verbosity logging in all OsmoCNI components and plowed through the resulting massive log. (I am logging via syslog - the whole point of running Slackware is principled opposition to systemd.) And I see what appears to be the culprit - see lines 7558 and 7559 in the attached log, UTC timestamp 02:48:54, 11 seconds after I initiated the test call from the origin phone. It appears that the BTS is telling the BSC that this error occurred: "Timer T200 expired (N200+1) times", and based on the lchan(0-0-2-TCH_F-0) annotation, this error is happening on the origin leg of the call, rather than the destination leg.
I looked up what T200 is:
https://www.rfwireless-world.com/Terminology/GSM-timers.html
T200: "It is used as retransmission on data link layer. Value varies depending on different messages (for FACCH it is set to 155ms)"
Why in the world are FACCH (or is it SACCH?) data link transmissions suddenly failing on the origin leg of the call, some 11 to 13 seconds in? I have tried 4 different phones in the call-origin role (Motorola C139 running its original fw, Pirelli DP-L10 likewise, Nokia C3-00 and my own FreeCalypso fw), and the behaviour is always the same.
Has anyone seen this problem before? Is it perhaps a known issue with nanoBTS that requires some special workaround? This tarball:
https://www.freecalypso.org/members/falcon/osmo-cni/t200-fail-config+log.tar...
contains my configuration files for all 5 OsmoCNI components and an excerpt from syslog. The syslog excerpt begins with all 5 OsmoCNI components booting up, followed by two phones connecting, and finally the test call.
I would really appreciate it if someone could give me some pointers for debugging this voice call show-stopper.
TIA, Mychaela
Hello again everyone,
I have done a little more investigation into the problem of why voice calls aren't working in my OsmoCNI+nanoBTS setup. Here are my configs and a syslog excerpt showing the problem:
https://www.freecalypso.org/members/falcon/osmo-cni/t200-fail-config+log.tar...
The behaviour I'm seeing seems to point toward a bug in nanoBTS fw, probably tickled by the officially-unsupported combo of running it with OsmoCNI instead of whatever proprietary sw ip.access requires officially.
Looking at the OsmoBSC log (captured with maximum verbosity in the posted syslog excerpt), the behaviour I'm seeing is that the BTS appears to stop successfully receiving transmissions from the MS when the lchan moves from SDCCH to TCH. For as long as the conversation with the MS stays on SDCCH, everything is happy: measurement reports from the BTS in the OsmoBSC log look good (including DL measurements received from the MS), and successful LU/SMS/USSD prove that all LAPDm control traffic passes as it should. But the moment the lchan moves from SDCCH to TCH, all subsequent measurement reports in the OsmoBSC log show "NOT VALID" for the part that comes from MS, and the UL RXQ numbers reported by the BTS are also highly suspect - usually 7. And then the BTS signals an error to the BSC, saying that T200 expired N200+1 times - whatever LAPDm exchange it is trying to do with the MS times out when the BTS is failing to receive anything sent by the MS. And then a little further down the road, the BTS declares Radio Link Failure, probably on the basis of not receiving any SACCH from the MS. All of this misbehaviour is visible in the syslog excerpt contained in the tarball linked above.
If the problem is indeed a bug in nanoBTS fw that can't be troubleshot because it's a black box, then I may be SOL until I can shell out big bucks for a sysmoBTS unit to replace this nanoBTS. However, I am doing my due diligence first by demonstrating to the community the nanoBTS misbehaviour I'm seeing, and asking in the hope that someone might know of a solution. These nanoBTS units are extremely attractive for economic reasons: it's a box that is purpose-made to be a GSM BTS, rather than just an SDR from which you have to build a BTS yourself with major extra effort, yet they are often available for dirt-cheap prices - hence it would be really awesome to have these units actually work with OsmoCNI, including voice calls.
The part that I find difficult to reconcile is that these nanoBTS units did once work with OpenBSC/OsmoNITB/OsmoCNI, and presumably voice calls worked too, otherwise there would have been warnings all over the docs that only SDCCH-based services work. So what has changed between the setups of other people in the past who had working Osmocom+nanoBTS combos and my present setup in which the BTS goes bonkers when the lchan moves from SDCCH to TCH? Could it be that my nanoBTS fw version is newer or older or otherwise different from what Osmocom people got to work in the past through reverse eng? Or is it perhaps the case that today's OsmoCNI sw is not tested as much with nanoBTS any more because the community moved on to OsmoBTS-based solutions, meaning either sysmoBTS or SDR platforms?
Here are the versions of all Osmocom CNI components and supporting libraries I am currently running:
libasn1c-0.9.33 libdbi-0.9.0 libdbi-drivers-0.9.0 libosmo-abis-1.2.0 libosmo-netif-1.1.0 libosmo-sccp-1.5.0 libosmocore-1.6.0 libsmpp34-1.14.1 lksctp-tools-1.0.17 osmo-bsc-1.8.1 osmo-hlr-1.4.0 osmo-mgw-1.9.0 osmo-msc-1.8.0
(Any non-Osmocom components in the list above are external libraries which were required for getting OsmoCNI to compile and run, but which were not already present in Slackware 14.2 and which I thus count as being strictly OsmoCNI dependencies.)
My nanoBTS (PCS1900) hw and fw versions are:
Equipment_Version='165b029_49' Software_Version='168d462_v200b207d0'
Is there anyone in this community who still runs with nanoBTS hw (hasn't moved on to OsmoBTS-based sysmoBTS or SDR) and who has working voice calls? If any such successful Osmocom+nanoBTS users exist, may you perhaps be able and willing to share the following:
* Your Osmocom sw version - the closer it is to latest OsmoCNI releases, the better;
* Your nanoBTS hw and fw versions;
* Your working configs for OsmoBSC and other Osmocom sw components, with any private stuff redacted out, obviously.
TIA, Mychaela
Hi Mychaela,
On Wed, Apr 20, 2022 at 12:16:08PM -0800, Mychaela Falconia wrote:
calls aren't working in my OsmoCNI+nanoBTS setup. Here are my configs and a syslog excerpt showing the problem:
https://www.freecalypso.org/members/falcon/osmo-cni/t200-fail-config+log.tar...
Thanks. The most important part for any kind of debugging however is always the pcap data of the A-bis and A interfaces, which are unfortunately not provided. If possible, please re-run.
If you in addition to that configure gsmtap log output, we will get a single time line with the Abis/A signaling messages and the log output of the osmocom programs, which is the easiest way to see what's happening all in one place (wirehark).
Regards, Harald
Hi Harald,
Thanks. The most important part for any kind of debugging however is always the pcap data of the A-bis and A interfaces, which are unfortunately not provided. If possible, please re-run.
I just made another run and got the pcap:
https://www.freecalypso.org/members/falcon/osmo-cni/session2.pcap
I ran tcpdump with the following filter script:
tcp port 3002 or tcp port 3003 or udp port 2427 or udp port 4729 or proto 132
which should have captured Abis between OsmoBSC and nanoBTS, SCTP carrying A, MGCP and GSMTAP. I also configured gsmtap logging in OsmoMSC and OsmoBSC for this run - are there any more components that need to have gsmtap logging enabled? I started logging after all of OsmoCNI processes were up and running (I used vty to enable gsmtap logging without saving it), but before I connected the PoE cable to the nanoBTS - thus everything from nanoBTS power-up onward should be in the log. Once nanoBTS booted up and started transmitting, I registered two phones and dialed a call from 3302 to 3301. The call failed exactly like before.
M~
Hi Mychaela,
On Thu, Apr 21, 2022 at 10:51:13PM -0800, Mychaela Falconia wrote:
I just made another run and got the pcap:
https://www.freecalypso.org/members/falcon/osmo-cni/session2.pcap
excellent.
What can be seen is that * Frame 2801: CHAN RQD (RACH) with Access Delay = 0 * Frame 2840: CHAN ACT (SDCCH) with TA = 0 * Frame 3035: MEAS RES with TA=0, Timing Offset = 63 * Frame 3142: MEAS RES with TA=0, Timing Offset = 191 * Frame 3150: MEAS RES with TA=0, Timing Offset = 195 * Frame 3248: MEAS RES with TA=8, Timing Offset = 198 ... * Frame 5218: MEAS RES Actual TA=0, Timing Offset 12 * Frame 5312: MEAS RES Actual TA=8, Timing Offset 11 * Frame 5407: MEAS RES Actual TA=8, Timing Offset 1 * Frame 5689: CHAN ACT (TCH) with TA=8
So somehow your MS is moving extremely fast away from the cell, reporting enormous timing offset and causing a high TA to be used on the TCH. And despite that high TA, the MS still reports a timing offset of 63.
I suspect some serious problem in the TA control loop either on MS or BTS side.
Hi Harald,
Thanks for looking at my log!
So somehow your MS is moving extremely fast away from the cell,
In actuality the MS was stationary during the entire test, and the fixed distance between the BTS and the MS was the distance between two rooms in one apartment turned lab - a few meters.
reporting enormous timing offset and causing a high TA to be used on the TCH. And despite that high TA, the MS still reports a timing offset of 63.
I suspect some serious problem in the TA control loop either on MS or BTS side.
The two MS in this test were Motorola C139 for the origin leg of the call and Pirelli DP-L10 for the destination leg. Both phones were running their respective manufacturers' original fw, i.e., no FreeCalypso fw in this test - hence FC is not to blame here. With both MS being of the standard historical type-approved variety, incorrect logic in the MS can probably be ruled out - so the finger of suspicion points at nanoBTS then?
Aside from the generally unfortunate situation that the combo of OsmoCNI+nanoBTS is not officially supported by anyone, there is also the fact that this particular nanoBTS unit came from ebay in completely unknown condition - a defect can't be ruled out... I have another nanoBTS unit that is also from ebay but from a much more reputable seller, and that one is probably defect-free - but that nanoBTS is for GSM850 band, and I don't feel safe transmitting in that band without a license - looking at my SA, that GSM850 band is much more crowded than PCS1900 around here.
M~
Hello Mychaela,
On 22/04/2022 19:21, Mychaela Falconia wrote:
Aside from the generally unfortunate situation that the combo of OsmoCNI+nanoBTS is not officially supported by anyone, there is also the fact that this particular nanoBTS unit came from ebay in completely unknown condition - a defect can't be ruled out... I have another nanoBTS unit that is also from ebay but from a much more reputable seller, and that one is probably defect-free - but that nanoBTS is for GSM850 band, and I don't feel safe transmitting in that band without a license - looking at my SA, that GSM850 band is much more crowded than PCS1900 around here.
I recommend you to buy from eBay a shielded box and do all your tests inside it even if you have a license to transmit.
Look for Rohde & Schwarz CMW-Z10 or CMU-Z11 which one is cheaper. If you need to test 5G C-band you will need to buy the CMW-Z10 (up to 6GHz).
Regards, R.
-
Harald Welte -
Labs -
Mychaela Falconia