Dear Harald,
In this mail I would like to express my thoughts and ideas regarding to the 'external interface for USSD'. To provide some background for other mailing list members, who are not aware of this topic, I'll explain it in a few sentences.
We already have an external interface for SMS - SMPP. But unfortunately, the same cannot be said about USSD. There is no built-in support of any interface, which could be used to handle USSD requests in a separate application. So, at the moment we only handle *#100# (own number request), which is implemented as a part of both OsmoMSC and legacy OsmoNiTB. And if one would like to implement additional USSD commands, it's required to modify and recompile the whole project :/
Fairwaves team already have some draft implementation for legacy OsmoNiTB, which relays on the libosmocore's GSM 04.80 API. And having the external interface in the mainline would be great not only for Fairwaves, but for the whole Osmocom project and its users.
The problem is that the current libosmocore's GSM 04.80 API is not complete and requires some critical modifications, which of course cannot be integrated immediately. Moreover, this implementation doesn't follow Osmocom coding style, for example, unlike other functions, where return code rc = 0 means success, there it means error...
So, first of all, we need to know, how many projects are using the current API, especially non-Osmocom projects. After that (taking it into account), it shall be decided:
- Should we keep the old API / ABI and maintain all new features among with it. This way would force us to duplicate the existing code and use different symbols. Then the old API would continuously became deprecated...
- Should we make all the critical changes in the current API (increasing the libosmocore version?). This way would break builds of dependent projects and would suppose fixing compatibility with the new API. But, at the same time, the implementation would be cleaner and closer to the specification, without any deprecated duplication...
Personally, I prefer the second way. At the same time, I want to make the modification as much transparent as possible.
Speaking about Osmocom projects, the only users of the GSM 04.80 API are OsmoMSC and legacy OsmoNiTB. Both projects utilize it for '*#100#' handling only :/
To be more convincing, let's look at the following code:
#define MAX_LEN_USSD_STRING 31
struct ss_request { uint8_t opcode; uint8_t ss_code; uint8_t ussd_text[MAX_LEN_USSD_STRING + 1]; uint8_t transaction_id; uint8_t invoke_id; };
This is what the current API allows you to obtain from a SS-request (e.g. '*#100#') coming from user. And what's wrong here?
- The 'ss_request' doesn't indicate a part of the library it comes from. Would be better to use 'gsm0480_ss_request'.
- The 'ussd_text' length doesn't follow the specification, where USSD OCTET STRING length is 160 bytes. So, the amount of characters depends on used coding scheme.
- The information about SS-message type, component type, text length and used DCS (Data Coding Scheme), etc. is missing. Despite it could be important for the external application.
So, let's discuss together all the 'pros and cons', and decide together, how should we facilitate the external USSD interface development.
With best regards, Vadim Yanitskiy.
Hi Vadim,
I'm sorry to respond to your elaborate mail in a very short way:
We do not want to break building old versions of OsmoNITB & Co with new library versions. Every time we do this, we are creating problems to our users. We have done this several times in the past, and it's always a pain to handle.
Let's assume anyone wants to build an old OsmoNITB version, for example in order to test for a regression (e.g. using 'git bisect'). How would you do this, if for every version you want to test, you need also *all* the matching libosmo* of that time. To make things worse, you don't even know which of the versions you need to use, making it impossible to hunt down when a regression was introduced in an effective way.
It would be really nice to have some automatic jenkins build jobs that actually test at least building old code (e.g. openbsc.git) against current 'master' libraries to avoid accidents, and to even clean up some of the incompatibilities we may have introduced since, if possible.
It's bad enough if we break this by accident. It's much worse if we do it knowingly.
So whatever we do, we have to try our best to keep old APIs stable or backwards-compatible. In most cases, the old API would simply be a compatibility wrapper around the new API. So there's one implementation of a given functionality, but the new users use it directly, while the old users go through a compatibility wrapper.
I don't have the details of the USSD code in my head, but I'd be surprised if it wouldn't be possible to simply wrap the new API behind an old API compat wrapper? If you encounter any specific problems, feel free to discuss ways how we can do this.
Regards, Harald
Hi Vadim, Harald,
I wanted to speak up and second the need to fix the USSD system, as Vadim notes it's actually just broken at the moment and doesn't meet the standard. A student in my lab (Rowan, cc'd) recently had to extend it to be able to interface with ussd_airflow, I believe building off of earlier fairwaves patches. We'd also like to be able to upstream those fixes to the community, but it seems that Harald's legacy support issue is the big hangup? Wouldn't it be easier to put related library versions into the changelog or something similar? Git submodules for instance allow you to specify a specific submodule hash and a mechanism like that could free up the interface development.
Just some thoughts. Thanks to everyone for bringing this issue up.
On Sat, Jan 13, 2018 at 3:00 PM, Harald Welte laforge@gnumonks.org wrote:
Hi Vadim,
I'm sorry to respond to your elaborate mail in a very short way:
We do not want to break building old versions of OsmoNITB & Co with new library versions. Every time we do this, we are creating problems to our users. We have done this several times in the past, and it's always a pain to handle.
Let's assume anyone wants to build an old OsmoNITB version, for example in order to test for a regression (e.g. using 'git bisect'). How would you do this, if for every version you want to test, you need also *all* the matching libosmo* of that time. To make things worse, you don't even know which of the versions you need to use, making it impossible to hunt down when a regression was introduced in an effective way.
It would be really nice to have some automatic jenkins build jobs that actually test at least building old code (e.g. openbsc.git) against current 'master' libraries to avoid accidents, and to even clean up some of the incompatibilities we may have introduced since, if possible.
It's bad enough if we break this by accident. It's much worse if we do it knowingly.
So whatever we do, we have to try our best to keep old APIs stable or backwards-compatible. In most cases, the old API would simply be a compatibility wrapper around the new API. So there's one implementation of a given functionality, but the new users use it directly, while the old users go through a compatibility wrapper.
I don't have the details of the USSD code in my head, but I'd be surprised if it wouldn't be possible to simply wrap the new API behind an old API compat wrapper? If you encounter any specific problems, feel free to discuss ways how we can do this.
Regards, Harald --
- Harald Welte laforge@gnumonks.org
http://laforge.gnumonks.org/
================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
On Sun, Jan 14, 2018 at 3:16 PM, Kurtis Heimerl kheimerl@cs.washington.edu wrote:
I wanted to speak up and second the need to fix the USSD system, as Vadim notes it's actually just broken at the moment and doesn't meet the standard. A student in my lab (Rowan, cc'd) recently had to extend it to be able to interface with ussd_airflow, I believe building off of earlier fairwaves patches. We'd also like to be able to upstream those fixes to the community,
Do you have a repo with this integration? This might be interesting for us as well. We're currently implementing USSD services in Erlang but a more non-programmer friendly YAML might be a good option in some cases.
Hi Alexander,
Most of the work I did was based off the libosmocore fairwaves/sup-ussd branch. As that branch stands, the first USSD message and its response are handled fine, where it breaks down, is if the service requests more information from the user. On subsequent messages, it looks like the phone leaves out a few critical bytes including the opcode. After reading the standards documents I still do not understand why this is the case but it was observed on multiple handsets and seems to be normal. Most of the modifications I had to make involved handling the other USSD message types and dealing with the fallout from having the opcode missing. I also added some additional fields to the ss_request struct to track the text length, message type and component type. I changed the max string length from 31 to 160 but I'm not sure if it should be longer. USSD strings are transmitted in a 7 bit packed format where the maximum length is 160 bytes. When expanded out into standard ascii this ends up around 182 characters so maybe the max size should be 182 characters.
To tie into an external USSD service, I set up osmo-nitb to forward USSD requests to a python script using sup. This script would then forward the USSD message to another program, ussd_airflow, via a http post request. Ussd_airflow is a Django web server that handles USSD requests and can be configured using YAML files.
The version of libosmocore used was forked from the master branch in October and has not been kept up to date since then, although I doubt any of the USSD code has been changed much since then. link: https://github.com/rowanphipps/libosmocore
The version of openbsc used was forked from the fairwaves/sup-ussd branch. It was not modified very much although that branch had not been touched since October 2015 so it is reasonably old. link: https://github.com/rowanphipps/openbsc
I made some modifications to the ussd_airflow web server to allow it it handle multiple different service codes but the original would work fine as well. documentation: https://django-ussd-airflow.readthedocs.io/en/latest/ original: https://github.com/mwaaas/ussd_airflow modified: https://github.com/rowanphipps/ussd_airflow
And here is the repo with my python script to link all the parts together: https://github.com/rowanphipps/sup_airflow
As for creating an external interface, the current sup interface could be left exposed with the idea that people could just write scripts to parse these messages and forward them as needed. If this is chosen then the sup protocol needs to be better documented and standardized. Sup seems to have been changed in more recent branches and now functions differently and in a way that the old fairwaves code does not work with. The main advantage of keeping sup is that it is very simple to extend to use with other USSD systems.
Let me know if you have any questions regarding the changes I made and how it works.
On Mon, Jan 15, 2018 at 4:54 PM Alexander Chemeris < Alexander.Chemeris@fairwaves.co> wrote:
On Sun, Jan 14, 2018 at 3:16 PM, Kurtis Heimerl kheimerl@cs.washington.edu wrote:
I wanted to speak up and second the need to fix the USSD system, as Vadim notes it's actually just broken at the moment and doesn't meet the
standard.
A student in my lab (Rowan, cc'd) recently had to extend it to be able to interface with ussd_airflow, I believe building off of earlier fairwaves patches. We'd also like to be able to upstream those fixes to the
community,
Do you have a repo with this integration? This might be interesting for us as well. We're currently implementing USSD services in Erlang but a more non-programmer friendly YAML might be a good option in some cases.
-- Regards, Alexander Chemeris. CTO/Founder, Fairwaves, Inc. Skype: Alexander.Chemeris Mobile: +1(424)400-7626 <(424)%20400-7626> https://fairwaves.co
Subscribe to Fairwaves news: http://eepurl.com/baL_pf
Hi, Harald,
I'm surprised to read this because I'm quite sure that you suggested breaking the API/ABI and increasing the library number yourself, and Vadim worked under the assumption that this is the suggested way to go.
On Sun, Jan 14, 2018 at 8:00 AM, Harald Welte laforge@gnumonks.org wrote:
Let's assume anyone wants to build an old OsmoNITB version, for example in order to test for a regression (e.g. using 'git bisect'). How would you do this, if for every version you want to test, you need also *all* the matching libosmo* of that time.
I personally believe that switching all libraries versions is the only right way to do in this situation because there are quite a lot of synchronized changes in both OsmoNITB and the underlying libraries. And specifically with hunting a regression example I would want to use a version of all the code in the exact state as it was before.
To make things worse, you don't even know which of the versions you need to use, making it impossible to hunt down when a regression was introduced in an effective way.
I thought that library versions serve exactly this purpose - to know which version to build against?
Another way is something similar to what we've done in osmo-combo repo which allows you to specify specific commits in each sub-module to build against.
So whatever we do, we have to try our best to keep old APIs stable or backwards-compatible. In most cases, the old API would simply be a compatibility wrapper around the new API. So there's one implementation of a given functionality, but the new users use it directly, while the old users go through a compatibility wrapper.
I'll let Vadim comment here but from what I remember, the old implementation is broken enough to make it difficult to maintain. I would rather focus on making sure the new API is good enough and we don't need to break things again in the foreseeable future.
-
Alexander Chemeris -
Harald Welte -
Kurtis Heimerl -
Rowan Phipps -
Vadim Yanitskiy