Hi list, Hi all.
This is quite Rhizomatica, or at least litecel 2050 specific, please excuse me for that. I also don't know, maybe it applies to other use cases.
Rhizomatica has one community where an attempt was made to increase coverage at the same time as capacity, by installing two 2050s with a somewhat overlapping coverage area. We have currently plans to do the same in other places.
The goal of this email is to try to establish the work needed to be done in order to fix what I outline below, I may just be lacking knowledge in some cases, in other It seems we need to implement some things.
I have grabbed fairwaves' meas_json and meas_web and modified a little to give me a visualisation of the neighbour meas reports.
So my first doubt is about idle cell reselection on the part of the MS. It doesn't seem to be operating quite as I would imagine it should and I am wondering is there any parameter that might make a difference, such as cell_identity? Is cell reselection hysteresis/offset relevant when handover is disabled? These parameters are not described in the manual. If I understood them I could update that. In the meantime, we seem to be getting reports from users that "the signal is weak" near what I might call the secondary BTS. At the same time, user reports are not terribly reliable. However what seems to be happening is that the MS is camped on the primary BTS, subsequently they have moved away so their handset is showing less 'bars'. I'm sometimes seeing TCH in use on the primary BTS that are showing neighbour measurements from the secondary that are more favourable.
Of course, I am often seeing messages such as:
handover_decision.c:203 (bts=3,trx=0,ts=2): Cell on ARFCN 242 is better: Skipping, Handover disabled
Obviously, this makes sense, and I am not sure why historically we have handover disabled, I presume it is issues with rtp, although we are using the rtp_proxy. I don't have two BTS so I can't test locally here. Could anybody comment on the state of that?
I am assuming that there is no way we can do load balancing unless we first have handover enabled, at the same time it is unclear to me if it will even work with handover. I believe this is called "Traffic Handover" Is this already implemented in the BSC? As in, can the BSC assign a channel on another ARFCN to the MS if the current ARFCN (the one the MS makes the access request on) has no TCH available?
I also have a doubt about operation with a Litecel/2050. Obviously you don't really want rescue handover operating between the two BTS that are present in the Litecel as it is the same antenna, and this shouldn't happen anyway as the signal levels should never be sufficiently different, but we do want load balancing handover. I do see "no resources available" log messages, (especially related to the BROKEN channels issue) when one BTS in a litecel has no more capacity but the other does, so it would seem that the BSC is not managing this. Is that correct?
Would it be correct to assume that maybe some handover rules are required that would specify: Handover from BTS 0 to BTS 3 but not from BTS 0 to BTS 1?
Finally, I have a doubt about the rtp proxy mode. Was it ever discussed to implement SIP Mobility aka re-Invite in order to be able to do BTS-MGW rtp and also handover? This would seem to me to be a good thing to have?
k/
Hi Keith,
we have had hand-over between BTSs working back in 2009, it is for sure not a new feature at all. This was working long before Abis/IP was implemented, and yes, it is also working with Abis/IP. I don't know why Rhizomatica has it disabled. Maybe you were seeing a particular bug or incompatibility at the time?
Regarding cell reselection offset / cell reselection hysteresis: We generally try to not document parameters which are taken 1:1 from the specifications, as there are plenty of textbooks on the subject of GSM network planning and optimization. From our point of view we just provide the operator a way to specify a parameter which is then broadcast to the mobile phones. It is nothing that OpenBSC/OsmoBTS act upon or treat in any way. You simply specify what you want and we pass it through.
And yes, the "cell reselection" parameters effect excatly cell-reselection in idle mode. This is completely unrelated to hand-over which happens in dedicated mode. Whether you have hand-over enabled or how it is configured thus has no influence on cell reselection and its parameters.
I don't understand your question regarding SIP or re-invite. If you're using the rtp_proxy, any hand-overs are completely transparent to MNCC. You don't even have a way to know a hand-over has happened behind the MNCC interface.
The proper solution for the two BTSs inside a LiteCell or sysmoBTS 2050 is to run them as one BTS with two TRX. This way you gain one timeslot, and you avoid any discussion about unwanted hand-over betwen those two transceivers.
At sysmocom we never implemented support for this, as there was and still is insufficient customer interest in the sysmoBTS 2050 to rectify spending any resources on developing this.
What would be needed is roughly: * OsmoBTS which can run on a single TRX without assuming that that TRX is C0 and transmits a BCCH/CCCH. * An OML proxy that terminates the OML connection to the BSC and which routes messages based on their TRX number to either of the two (or "N") instances of OsmoBTS.
At that point, one board in the 2050 runs OsmoBTS for C0, and the OML proxy. The other board runs OsmoBTS for C1. Both OsmoBTS connect to the OML proxy on C0, and the OML proxy establishes OML to the BSC. Each OsmoBTS establishes a separate RSL connection (each from its own IP address) towards the BSC.
On the BSC side we handle this since 2010 or 2011, as this is also how a multi-trx setup with nanoBTSs looks like on the Abis side.
All other OsmoBTS-supported Multi-TRX hardware runs all transceivers with a single OsmoBTS instance, so they don't need the kind of support required for the 2050. Hence, we are back to step one: Somebody would have to spend some time on the required code. I'd be more than happy to review and/or merge any related changes.
Regards, Harald
Hi Harald.
I appreciate the clarification on things that are pure GSM spec, I know where to look now. :)
I tried activating handover this morning very briefly on that site and it started a ping-pong game between the two bts/trx in the LiteCell. Increasing handover power budget hysteresis a lot, >=15dB of course tames that, but surprisingly does not eliminate it. I thought to increase then the neighbour averaging, but 10 is the maximum. It seems it would be difficult to tune that for permanent operation. Notwithstanding what you said about the solution for the LiteCell, which is noted, it still suggests to me some kind of simpler solution such as a ruleset about which BTSs to use for rescue HO.
On the question about SIP re-invite, I am talking about operating without rtp_proxy, so that one can have the advantage of BTS<->BTS RTP streams at the same time as handover. From what I've read, this is quite feasable, as part of the SIP spec. I think this is already considered as part of the development of the osmo-sip-connector, which is a project I really want to see moving forward and hope to find time to contribute to over the next few months.
However, and again notwithstanding your suggesting for how to resolve the LiteCell issue of the two BTS/TRX, I understand that load balancing (aka traffic handover?) is not implemented in OpenBSC. I am certainly seeing this, for example in the case where all available TCH are in use, a call cannot be established, even though another BTS has resources available. Is this not something that is desirable to implement? Regardless of whether it were a LiteCell, or two or more other types of BTS such as SysmoBTS 1020? Or what is the intended strategy for increasing channel capacity at a site?
Hi Keith,
On Tue, Feb 21, 2017 at 03:40:54PM +0100, Keith wrote:
I tried activating handover this morning very briefly on that site and it started a ping-pong game between the two bts/trx in the LiteCell.
this is expected, as per GSM architecture such as system should operate as two TRXs of one BTS and not as two BTSs. No "normal" operator would put two BTSs with same transmit power and same coverage in place.
Increasing handover power budget hysteresis a lot, >=15dB of course tames that, but surprisingly does not eliminate it.
This might relate to the fact that the power control loop algorithm is very simplistic and has a tendency to oscillate. It has been a long-time TODO item to avoid the algorithm in the PHY and use a more dampened algorithm with a PID or kalman filter in OsmoBTS. But then, there are so many open TODO items for so many years :/
However, and again notwithstanding your suggesting for how to resolve the LiteCell issue of the two BTS/TRX, I understand that load balancing (aka traffic handover?) is not implemented in OpenBSC. I am certainly seeing this, for example in the case where all available TCH are in use, a call cannot be established, even though another BTS has resources available. Is this not something that is desirable to implement? Regardless of whether it were a LiteCell, or two or more other types of BTS such as SysmoBTS 1020?
Just a quick response: jolly implemented "traffic handover" aka "load balancing" 3 years ago as part of a branch (I believe for a customer), but unfortunately this never was merged back to master.
Meanwhile, we already have all the code for execution of hand-overs in place. We also have dynamic timeslots in place. So the "mechanics" should all be there to add the two missing features from those old branches: a) de-fragmenting of channels (i.e. merging two separate TCH/H into one timeslot, so the other timeslot can be switched to TCH/F or PDCH on demand b) load/traffic based hand-over
see jolly/new_handover and jolly/dyn_pdch branches.
Or what is the intended strategy for increasing channel capacity at a site?
The general strategy I would normally assume is to * switch to half-rate channels with AMR whenever possible * use dynamic channel switching to switch between TCH/H (default) and TCH/F (if neded) * use BTSs with more TRX. Even high-end Ericsson equipment supporting up to 3 sectors of 4 TRX each (12 TRX) is actaully available surprisingly inexpensive these days. We've recently done some improvements in Ericsson RBS2000 support. Even PCU integration for GRPS support is coming up.
Dear Harald,
I noticed this part:
Even PCU integration for GRPS support is coming up.
Can I ask how specific will that part be to the RBS2000?
I have Nokia Ultra and MetroSite sets (both with EDGE capability), the EDAP part is accessbile, but I was not able to find a lot of details about the Nokia Abis implementation, especially of the PCU part.
Thanks!
Csaba
----- Eredeti üzenet ----- Feladó: "Harald Welte" laforge@gnumonks.org Címzett: "Keith" keith@rhizomatica.org Másolatot kap: "OpenBSC Mailing List" openbsc@lists.osmocom.org Elküldött üzenetek: Kedd, 2017. Február 21. 16:14:05 Tárgy: Re: Handover and load balancing on SysmoBTS 2050
Hi Keith,
On Tue, Feb 21, 2017 at 03:40:54PM +0100, Keith wrote:
I tried activating handover this morning very briefly on that site and it started a ping-pong game between the two bts/trx in the LiteCell.
this is expected, as per GSM architecture such as system should operate as two TRXs of one BTS and not as two BTSs. No "normal" operator would put two BTSs with same transmit power and same coverage in place.
Increasing handover power budget hysteresis a lot, >=15dB of course tames that, but surprisingly does not eliminate it.
This might relate to the fact that the power control loop algorithm is very simplistic and has a tendency to oscillate. It has been a long-time TODO item to avoid the algorithm in the PHY and use a more dampened algorithm with a PID or kalman filter in OsmoBTS. But then, there are so many open TODO items for so many years :/
However, and again notwithstanding your suggesting for how to resolve the LiteCell issue of the two BTS/TRX, I understand that load balancing (aka traffic handover?) is not implemented in OpenBSC. I am certainly seeing this, for example in the case where all available TCH are in use, a call cannot be established, even though another BTS has resources available. Is this not something that is desirable to implement? Regardless of whether it were a LiteCell, or two or more other types of BTS such as SysmoBTS 1020?
Just a quick response: jolly implemented "traffic handover" aka "load balancing" 3 years ago as part of a branch (I believe for a customer), but unfortunately this never was merged back to master.
Meanwhile, we already have all the code for execution of hand-overs in place. We also have dynamic timeslots in place. So the "mechanics" should all be there to add the two missing features from those old branches: a) de-fragmenting of channels (i.e. merging two separate TCH/H into one timeslot, so the other timeslot can be switched to TCH/F or PDCH on demand b) load/traffic based hand-over
see jolly/new_handover and jolly/dyn_pdch branches.
Or what is the intended strategy for increasing channel capacity at a site?
The general strategy I would normally assume is to * switch to half-rate channels with AMR whenever possible * use dynamic channel switching to switch between TCH/H (default) and TCH/F (if neded) * use BTSs with more TRX. Even high-end Ericsson equipment supporting up to 3 sectors of 4 TRX each (12 TRX) is actaully available surprisingly inexpensive these days. We've recently done some improvements in Ericsson RBS2000 support. Even PCU integration for GRPS support is coming up.
Hi Sipos,
On Tue, Feb 21, 2017 at 10:08:17PM +0100, Sipos Csaba wrote:
Even PCU integration for GRPS support is coming up.
Can I ask how specific will that part be to the RBS2000?
*very* specific, as all related protocols (Ericsson L2TP, P-GSL, TFP, ericsson modified RSL) are Ericsson specific.
On 21/02/2017 16:14, Harald Welte wrote:
Just a quick response: jolly implemented "traffic handover" aka "load balancing" 3 years ago as part of a branch (I believe for a customer), but unfortunately this never was merged back to master.
Reading the commit log, this looks terrific...
Meanwhile, we already have all the code for execution of hand-overs in place. We also have dynamic timeslots in place. So the "mechanics" should all be there to add the two missing features from those old branches: a) de-fragmenting of channels (i.e. merging two separate TCH/H into one timeslot, so the other timeslot can be switched to TCH/F or PDCH on demand b) load/traffic based hand-over
So essentially everything I was talking about is already implemented!
My question then would be, and I understand maybe the answer is not so easily available without a full code review, but
Was it unfortunately never merged back to master because Simply no one ever made the effort to do so. OR There were problems with the code, disagreement on the implementation or incompatibilities with other developments etc..
If it's the former, I'll have a go at this. If it is the latter, it's probably beyond my skills.
The general strategy I would normally assume is to
- switch to half-rate channels with AMR whenever possible
- use dynamic channel switching to switch between TCH/H (default) and TCH/F (if neded)
Absolutely, I must revisit this one. It was shelved for us due to my understanding a few months ago that if we implemented AMR, then we couldn't also use FR. As that's all now resolved, I can implement, test and report. :-)
Hi Keith,
On Fri, Feb 24, 2017 at 05:12:02PM +0100, Keith wrote:
Was it unfortunately never merged back to master because Simply no one ever made the effort to do so.
correct. I really love all the work jolly did, and I'm thankful for that. However, in order to really work productively in a collaborative FOSS project, it is part of the contributors task to submit the code in chunks digestible, go through the review process, address the review that might occur.
Yes, that's annoying and much less exciting than writing the code in the first place. But it's necessary, as it ensures the code has been reviewed by other experts in the codebase, is understood by more than one person (in case of later issues), and also ensures a common architecture of the overall code. It's what set's a one-man-show apart from a collaborative effort.
So the code that is present in some of the many branches that developers created might work perfectly fine and might even be used in production by some users. But the code in there has never been submitted for review or merge. And of course it bit-rots over time, which would have been avoidable had it gone through review + merge at the time it was completed.
There were problems with the code, disagreement on the implementation or incompatibilities with other developments etc..
That may or may not be the case. As it was never presented for review/merge, we simply don't know. I think what needs to be done is it has to be re-based, understood, tested (both in terms of manual testing and in terms of adding unit tests where missing), split up in digestible patch sets and submitted for review in gerrit.
Regards, Harald
On Fri, Feb 24, 2017 at 05:26:28PM +0100, Harald Welte wrote:
Hi Keith,
On Fri, Feb 24, 2017 at 05:12:02PM +0100, Keith wrote:
Was it unfortunately never merged back to master because Simply no one ever made the effort to do so.
correct.
And let me add that it would also need an effort to duly test it. I've touched the lchan defrag patch (while peeling the dynamic timeslot code from it) and it looked like it works, but I assume we'll still find some problems in tests.
~N
On 25/02/2017 17:55, Neels Hofmeyr wrote:
On Fri, Feb 24, 2017 at 05:26:28PM +0100, Harald Welte wrote:
Hi Keith,
On Fri, Feb 24, 2017 at 05:12:02PM +0100, Keith wrote:
Was it unfortunately never merged back to master because Simply no one ever made the effort to do so.
correct.
And let me add that it would also need an effort to duly test it.
I'm very happy to put that effort in. The effort required for me to test (using test BTSs and phones in the lab as well as coordinated real world deployment tests) is minimal compared to moving the code into the master branch and writing tests as in "make test" for it. :-)
On 21 Feb 2017, at 21:40, Keith keith@rhizomatica.org wrote:
Hi Keith,
On the question about SIP re-invite, I am talking about operating without rtp_proxy, so that one can have the advantage of BTS<->BTS RTP streams at the same time as handover. From what I've read, this is quite feasable, as part of the SIP spec. I think this is already considered as part of the development of the osmo-sip-connector, which is a project I really want to see moving forward and hope to find time to contribute to over the next few months.
yes for the osmo-sip-connector I have ignored the topic of handover. In general I try to see how long I get away with not having to touch the UDP/RTP streams.
In case of hand-over the new BTS might send from a different src IP, src Port and will most likely have a new SSRC, timestamp seqno. It is a bit of a question of how "your" PBX will handle it. I can see a couple of outcomes.
a.) It has some support for "NAT" handling and will just use the new stream and return packages to the new src IP/src port. It might go back and forth but at some point the old bts will stop sending things.
b.) It will accept the new stream but will try to send to the old BTS. The we would have one-way audio.
c.) It will reject it as it is a unknown src ip/port.
I think for b.) and c.) we will require SIP re-invite but also need to look at the AoIP spec to see if they say how to handle this scenario. But that depends on how the PBX handles this as well. So it means for handover we always need to look at BSC/SIP+PBX and how it is handling SDP/RTP.
cheers holger
PS: Do you have a docker container for your FreeSWITCH setup and maybe two numbers one to play audio and another to record the audio?
Hi Holger,
On Wed, Feb 22, 2017 at 04:02:17PM +0700, Holger Freyther wrote:
I think for b.) and c.) we will require SIP re-invite but also need to look at the AoIP spec to see if they say how to handle this scenario.
I "think" the assumption in AoIP is that the RTP "originates" from the BSC and not from the BTS. As such, there is no change to the RTP stream parameters in any intra-BSC hand-over. The reasoning/rationale would be simple: There are still virtually no Abis/IP based BTS in the field so far, and most commercial systems (even RBS6000 from Ericsson e.g.) still uses E1/T1 for the Abis. That Abis might be adapted over some propriatary L2TP-over-IP stacking for the back-haul.
So basically except the Osmocom implementations, and ip.access, I still don't think there are many native IP based Abis systems out there, and hence this is considered an implementation detail of the vendor. The A interface orignates at the BSC, and everything "below" the BSC is opaque.
But feel free to check yourself, I might remember it wrong.
On 22/02/2017 10:02, Holger Freyther wrote:
PS: Do you have a docker container for your FreeSWITCH setup and maybe two numbers one to play audio and another to record the audio?
The short answer is no. Docker is far too much in /vogue/ for me to be seen using it :-)
Joking aside, we have a docker fan currently lending a hand at the office and willing to put together whatever you need. What else do you need in the container?
We are still (somewhat ashamedly) using a rather old FreeSWITCH version in production, so the best might be to use the latest stable for this setup, rather than replicate ours exactly. All call numbers and setup are handled by our python dialplan, which is probably unneeded complexity for testing RTP streams. I can look at something for FreeSWITCH where you call a number that will playback audio and also trigger recording, if that's what you need.
k/
On 21 Feb 2017, at 00:50, Harald Welte laforge@gnumonks.org wrote:
Hi Keith,
Hi!
we have had hand-over between BTSs working back in 2009, it is for sure not a new feature at all. This was working long before Abis/IP was implemented, and yes, it is also working with Abis/IP. I don't know why Rhizomatica has it disabled. Maybe you were seeing a particular bug or incompatibility at the time?
I don't remember the details but we had several issues with it:
* Crashing or no audio (e.g. only used at XC3) * Delay increases by number of handovers
Since then we probably fixed the crash, I had removed the timestamp adjustment code. I still need to study the 3gpp AoIP specification but these days I wonder what should be done with the audio stream of the new BTS and I think we should generate a new SSRC, seq and timestamp.
This should trigger the start of a new probation period as of Annex A. But it is a theoretical question (or for E1->RTP systems) right now.
holger
-
Harald Welte -
Holger Freyther -
Keith -
Neels Hofmeyr -
Sipos Csaba