Hi Max,
Thanks for your reply!
I think extending utils/conv_gen.py is the way to go because it will allow us to have concise description of the convolutional codes in one place and as close to the description in standards as possible.
Ok, I will keep your opinion in my mind.
Could you specify which files exactly are you referring to? Also you can use 'git blame' to clarify authorship.
No problem, they are:
- gsm0503_conv.c - gsm0503_interleaving.c - gsm0503_mapping.c - gsm0503_parity.c - gsm0503_tables.c
Almost all the gsm0503_*, excluding the 'gsm0503_coding.*'. Thank you for this hint, this command is very usable! :)
I don't think library functions should do any sort of logging by itself (unless it's a logging functions of course :) Instead they should return clearly distinguishable values and let caller do the logging as they see fit.
I am agree with you. It's time to use return codes.
Have a nice day!
With best regards, Vadim Yanitskiy.
Hi all!
I would like to inform, that this task is almost finished! Just tested my changes, and build was successful without any warnings. But I still have a couple of unresolved issues:
I have some problems with understanding how to add a new item into the convolutional codes generator (utils/conv_gen.py), so temporary I just used a single file from OsmoBTS (gsm0503_conv.c) to check, if all the rest forks fine. If someone (Max?) can help me to move missed tables (mostly EGPRS specific, see 73cb583e5147a267a370c576e8ac77507de6d0d7), I will be grateful, and the task will be completely finished soon.
Also, there are several external dependences from libosmocodec, such as:
- gsm620_unvoiced_bitorder - gsm620_voiced_bitorder - gsm660_bitorder - gsm610_bitorder
They are some uint16_t arrays, related to GSM HR, FR and EFR. It's sadly, that they cannot be included using exactly corresponding header files. I think there are three possible ways to solve this problem:
1. Link gsm620.c, gsm660.c and gsm610.c to libosmogsm sources. (I am not sure, that it is a good way...) 2. Copy required arrays to libosmogsm (also unlikely). 3. Add libosmocodec.la as dependence (I suggest exactly this solution): - libgsmint_la_LIBADD = ../libosmocore.la + libgsmint_la_LIBADD = ../libosmocore.la ../codec/libosmocodec.la
All other problems was solved.
With best regards, Vadim Yanitskiy.
2016-08-04 19:51 GMT+06:00 Vadim Yanitskiy axilirator@gmail.com:
Hi Max,
Thanks for your reply!
I think extending utils/conv_gen.py is the way to go because it will allow us to have concise description of the convolutional codes in one place and as close to the description in standards as possible.
Ok, I will keep your opinion in my mind.
Could you specify which files exactly are you referring to? Also you can use 'git blame' to clarify authorship.
No problem, they are:
- gsm0503_conv.c
- gsm0503_interleaving.c
- gsm0503_mapping.c
- gsm0503_parity.c
- gsm0503_tables.c
Almost all the gsm0503_*, excluding the 'gsm0503_coding.*'. Thank you for this hint, this command is very usable! :)
I don't think library functions should do any sort of logging by itself (unless it's a logging functions of course :) Instead they should return clearly distinguishable values and let caller do the logging as they see fit.
I am agree with you. It's time to use return codes.
Have a nice day!
With best regards, Vadim Yanitskiy.
Hi Vadim,
On Fri, Aug 5, 2016 at 1:46 PM, Vadim Yanitskiy axilirator@gmail.com wrote:
I would like to inform, that this task is almost finished! Just tested my changes, and build was successful without any warnings. But I still have a couple of unresolved issues:
This is great.
I have some problems with understanding how to add a new item into the convolutional codes generator (utils/conv_gen.py), so temporary I just used a single file from OsmoBTS (gsm0503_conv.c) to check, if all the rest forks fine. If someone (Max?) can help me to move missed tables (mostly EGPRS specific, see 73cb583e5147a267a370c576e8ac77507de6d0d7), I will be grateful, and the task will be completely finished soon.
With regards to convolutional coding, the main difference between GPRS and EGPRS will be the use of tail-biting recursive codes.
Max, can the utility generate recursive state tables?
-TT
Dear all,
There is a problem, related to GSM 05.03 code migration. Please, have a look:
https://gerrit.osmocom.org/#/c/841/1/src/gsm/Makefile.am
The problem is that the gsm0503_coding.c has some dependences from libosmocodec, they are:
- src/codec/gsm610.c - src/codec/gsm620.c - src/codec/gsm660.c
And I have some doubts how to link them properly. As we already discussed with Neels, there are the following possible ways:
- We can copy those files from ../codec/ during compilation, and remove them after make (dist)clean. This approach you can see implemented right now.
- We can link the libosmocodec to the libgsmint or libosmogsm:
- libgsmint_la_LIBADD = ../libosmocore.la
- libgsmint_la_LIBADD = ../libosmocore.la ../codec/libosmocodec.la
- Merge libosmocodec into libosmogsm.
Every mentioned approach has it's own disadvantages. Maybe there is another way to solve the problem? Any opinions are welcome!
With best regards, Vadim Yanitskiy.
2016-09-06 10:46 GMT+07:00 Sylvain Munaut 246tnt@gmail.com:
Hi,
With regards to convolutional coding, the main difference between GPRS and EGPRS will be the use of tail-biting recursive codes.
Max, can the utility generate recursive state tables?
Yes
Cheers,
Sylvain
Hi Vadim,
On Sun, Sep 11, 2016 at 09:43:58AM +0600, Vadim Yanitskiy wrote:
There is a problem, related to GSM 05.03 code migration. Please, have a look:
https://gerrit.osmocom.org/#/c/841/1/src/gsm/Makefile.am
The problem is that the gsm0503_coding.c has some dependences from libosmocodec, they are:
- src/codec/gsm610.c - src/codec/gsm620.c - src/codec/gsm660.c
And I have some doubts how to link them properly. As we already discussed with Neels, there are the following possible ways [...]
I think there's one option that I would prefer: Having a new library, like libosmogsmphy (or libosmogsm-phy or libosmogsm_phy?) which contains the gsm0503 code and has dependencies to libosmocore and libosmocodec.
This library could then either be inside libosmocore.git (if we can agree to re-license it as GPLv2), or we'd have to maintain it in a separate repository (if we have to keep it AGPL). This separation is not legally mandator and has no legal significance, it is juts merely not to confuse the user to obtain libosmocore.git which contained a mix of different licenses.
Regards, Harald
On Sun, Sep 11, 2016 at 12:12:05PM +0800, Harald Welte wrote:
I think there's one option that I would prefer: Having a new library,
Since you are basically the most proficient on the topic and with the licensing issues discussed, this looks like the way to go. https://lists.osmocom.org/pipermail/openbsc/2016-September/009673.html
This separation is not legally mandator and has no legal significance, it is juts merely not to confuse the user to obtain libosmocore.git which contained a mix of different licenses.
If one would like to avoid one of the two licenses, it would be cumbersome to have to install and then purge the part that has the wrong license. So it appears to me having a separate git makes sense.
However, will there be tight dependencies between libosmocore and the new lib? Where exactly will be the separation between libosmocore and the new lib?
libosmogsmphy (or libosmogsm-phy or libosmogsm_phy?)
We so far have only dashes in our git names, so not libosmogsm_phy. With libosmogsm as part of libosmocore, I would expect "libosmogsm-phy" to also be part of libosmocore. All newer libs have libosmo-*, so I'd bikeshed as
libosmo-gsmphy
From my meagre understanding of the topic I'd have chosen libosmo-gsmcodec, but I obviously trust that your choice for "phy" is sound.
So then, we need a new git.osmocom.org as well as gerrit repository, right? I can add a gerrit project, but not a git.osmocom.org one, and am not familiar with the replication process between the two...
~Neels
Hi,
This library could then either be inside libosmocore.git (if we can agree to re-license it as GPLv2), or we'd have to maintain it in a separate repository (if we have to keep it AGPL).
I'd definitely try to re-license to GPLv2+
Just my 2ct.
Cheers,
Sylvain
Hi,
I'd definitely try to re-license to GPLv2+
+1 Let's wait for copyright holders.
With best regards, Vadim Yanitskiy.
2016-09-12 21:48 GMT+07:00 Sylvain Munaut 246tnt@gmail.com:
Hi,
This library could then either be inside libosmocore.git (if we can agree to re-license it as GPLv2), or we'd have to maintain it in a separate repository (if we have to keep it AGPL).
I'd definitely try to re-license to GPLv2+
Just my 2ct.
Cheers,
Sylvain
Hi Harald,
I think there's one option that I would prefer: Having a new library
Ok, one may be a separate library in separate repo, or may be included into libosmocore as a sub-library, if re-licensing would be successful. Let's go this way.
like libosmogsmphy (or libosmogsm-phy or libosmogsm_phy?) which contains the gsm0503 code and has dependencies to libosmocore and libosmocodec.
IMHO, 'phy' may sound a little bit confusing. I would prefer something like 'libosmo-coding' or 'libosmocoding', without 'gsm' prefix, because there are GPRS and EDGE too. Also, this may be a potential place for other transcoding things, unrelated to GSM at all. We are talking about channel coging, right? :)
With best regards, Vadim Yanitskiy.
2016-09-11 11:12 GMT+07:00 Harald Welte laforge@gnumonks.org:
Hi Vadim,
On Sun, Sep 11, 2016 at 09:43:58AM +0600, Vadim Yanitskiy wrote:
There is a problem, related to GSM 05.03 code migration. Please, have a look:
https://gerrit.osmocom.org/#/c/841/1/src/gsm/Makefile.am
The problem is that the gsm0503_coding.c has some dependences from libosmocodec, they are:
- src/codec/gsm610.c
- src/codec/gsm620.c
- src/codec/gsm660.c
And I have some doubts how to link them properly. As we already discussed with Neels, there are the following possible ways [...]
I think there's one option that I would prefer: Having a new library, like libosmogsmphy (or libosmogsm-phy or libosmogsm_phy?) which contains the gsm0503 code and has dependencies to libosmocore and libosmocodec.
This library could then either be inside libosmocore.git (if we can agree to re-license it as GPLv2), or we'd have to maintain it in a separate repository (if we have to keep it AGPL). This separation is not legally mandator and has no legal significance, it is juts merely not to confuse the user to obtain libosmocore.git which contained a mix of different licenses.
Regards, Harald --
- Harald Welte laforge@gnumonks.org
http://laforge.gnumonks.org/
================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
Hi Vadim,
sorry for the late foll-wup.
On Mon, Sep 12, 2016 at 10:56:11PM +0600, Vadim Yanitskiy wrote:
I think there's one option that I would prefer: Having a new library
Ok, one may be a separate library in separate repo, or may be included into libosmocore as a sub-library, if re-licensing would be successful. Let's go this way.
As re-licensing is approved by all authors meahwile, a sub-library inside libosmocore.git seems the solution.
like libosmogsmphy (or libosmogsm-phy or libosmogsm_phy?) which contains the gsm0503 code and has dependencies to libosmocore and libosmocodec.
IMHO, 'phy' may sound a little bit confusing. I would prefer something like 'libosmo-coding' or 'libosmocoding', without 'gsm' prefix, because there are GPRS and EDGE too.
I wanted the 'gsm' to indicate 2G. Most people speak of "GSM" when they actually mean GSM+GPRS+EGPRS, i.e. second-generation technologies.
Also, this may be a potential place for other transcoding things, unrelated to GSM at all. We are talking about channel coging, right? :)
I would assume most projects would be interested in the coding of one specific technology, and not all of them at the same time. Yes, there might be exceptions as a multi-RAT signal analyzer, but that could then very easily link several libraries.
So my preference would be to have a '2g' specific library, and reflect that in the name by using either '2g' or 'gsm' as part of the name.
Regards, Harald
Hi Harald,
As re-licensing is approved by all authors meahwile, a sub-library inside libosmocore.git seems the solution.
Thanks for you assistance!
I would assume most projects would be interested in the coding of one specific technology, and not all of them at the same time. Yes, there might be exceptions as a multi-RAT signal analyzer, but that could then very easily link several libraries.
So my preference would be to have a '2g' specific library, and reflect that in the name by using either '2g' or 'gsm' as part of the name.
Ok, this is reasonable opinion and I am agree with you. There is only one possible variant in my mind - 'libosmogsm-coding'. What about this one?
As soon as we reach an agreement in this question, I will update the lastest change and, I hope, my changes will be merged. Also, I would like to ask anyone to check the change, because this is first time I am adding a new library :)
With best regards, Vadim Yanitskiy.
2016-09-26 11:05 GMT+07:00 Harald Welte laforge@gnumonks.org:
Hi Vadim,
sorry for the late foll-wup.
On Mon, Sep 12, 2016 at 10:56:11PM +0600, Vadim Yanitskiy wrote:
I think there's one option that I would prefer: Having a new library
Ok, one may be a separate library in separate repo, or may be included into libosmocore as a sub-library, if re-licensing would be successful. Let's go this way.
As re-licensing is approved by all authors meahwile, a sub-library inside libosmocore.git seems the solution.
like libosmogsmphy (or libosmogsm-phy or libosmogsm_phy?) which
contains
the gsm0503 code and has dependencies to libosmocore and libosmocodec.
IMHO, 'phy' may sound a little bit confusing. I would prefer something like 'libosmo-coding' or 'libosmocoding', without 'gsm' prefix, because there are GPRS and EDGE too.
I wanted the 'gsm' to indicate 2G. Most people speak of "GSM" when they actually mean GSM+GPRS+EGPRS, i.e. second-generation technologies.
Also, this may be a potential place for other transcoding things, unrelated to GSM at all. We are talking about channel coging, right? :)
I would assume most projects would be interested in the coding of one specific technology, and not all of them at the same time. Yes, there might be exceptions as a multi-RAT signal analyzer, but that could then very easily link several libraries.
So my preference would be to have a '2g' specific library, and reflect that in the name by using either '2g' or 'gsm' as part of the name.
Regards, Harald --
- Harald Welte laforge@gnumonks.org
http://laforge.gnumonks.org/
================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
Dear all,
I just wrote a simple test to ensure, that the new library performs valid transcoding. I used a DCCH "LAPDM U, func=UI" frame for simplicity, and good news is that test passed. See attachment for details.
In the test I wrote the best signal quality is assumed. So, ubit 0 is sbit -127 and ubit 1 is 127. But there is no such things as miracles, and on practice error correction takes place. So, we need to make sure, that correction works.
I am going to write tests and I have some questions:
1) How can I know the maximum count of bits can be corrected for a specific channel type? 2) In case of: gsm0503_*_decode(... int *n_errors, int *n_bits_total); What do the two last params mean? 3) Can anyone provide me some samples for GPRS and EDGE? It will take some time to set up PDCH support in my test network.
Thanks in advance!
With best regards, Vadim Yanitskiy.
2016-09-26 11:05 GMT+07:00 Harald Welte laforge@gnumonks.org:
Hi Vadim,
sorry for the late foll-wup.
On Mon, Sep 12, 2016 at 10:56:11PM +0600, Vadim Yanitskiy wrote:
I think there's one option that I would prefer: Having a new library
Ok, one may be a separate library in separate repo, or may be included into libosmocore as a sub-library, if re-licensing would be successful. Let's go this way.
As re-licensing is approved by all authors meahwile, a sub-library inside libosmocore.git seems the solution.
like libosmogsmphy (or libosmogsm-phy or libosmogsm_phy?) which
contains
the gsm0503 code and has dependencies to libosmocore and libosmocodec.
IMHO, 'phy' may sound a little bit confusing. I would prefer something like 'libosmo-coding' or 'libosmocoding', without 'gsm' prefix, because there are GPRS and EDGE too.
I wanted the 'gsm' to indicate 2G. Most people speak of "GSM" when they actually mean GSM+GPRS+EGPRS, i.e. second-generation technologies.
Also, this may be a potential place for other transcoding things, unrelated to GSM at all. We are talking about channel coging, right? :)
I would assume most projects would be interested in the coding of one specific technology, and not all of them at the same time. Yes, there might be exceptions as a multi-RAT signal analyzer, but that could then very easily link several libraries.
So my preference would be to have a '2g' specific library, and reflect that in the name by using either '2g' or 'gsm' as part of the name.
Regards, Harald --
- Harald Welte laforge@gnumonks.org
http://laforge.gnumonks.org/
================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
Dear Vadim,
it migt be a coincidence that I was in the "To" line of the mail, but just for clarification for me personally and the team at sysmocom: The BTS models we use don't use the gsm0503_* code, as we don't use osmo-bts-trx. The amount of involvement I had with the osmo-bts-trx code was merely to adopt it to some core infrastructural changes in osmo-bts.
Don't get me wrong, I'm very happy that osmo-bts-trx (and osmo-trx) exists, but it is unfortunately still pretty much in lack of proper maintenance.
On Wed, Sep 28, 2016 at 02:18:13AM +0600, Vadim Yanitskiy wrote:
- How can I know the maximum count of bits can be corrected
for a specific channel type?
I have no input on that, see above.
- In case of:
gsm0503_*_decode(... int *n_errors, int *n_bits_total); What do the two last params mean?
output variables: the number of error bits in the decoded frame and the number of total bits in the frame. Used to calculate BER.
- Can anyone provide me some samples for GPRS and EDGE?
It will take some time to set up PDCH support in my test network.
I hope the primary proponents, beneficiaries and users of the related code will be able to help out with that.
Regards, Harald
Hi Sylvain and Harald!
I am contact you again (sorry for so many mails), because Sylvain is a copyright holder of the 'libosmocore/utils/conv_gen.py', and Harald listed in the output of 'git blame' for almost every line.
There is a mismatch between the CS2 & CS3 convolutional code definitions in 'osmo-bts/src/osmo-bts-trx/gsm0503_conv.c' and 'libosmocore/utils/conv_gen.py'. In second source there is no puncture for both definitions:
const struct osmo_conv_code gsm0503_conv_cs2 = { .N = 2, .K = 5, .len = 290, .next_output = conv_xcch_next_output, .next_state = conv_xcch_next_state, };
const struct osmo_conv_code gsm0503_conv_cs3 = { .N = 2, .K = 5, .len = 334, .next_output = conv_xcch_next_output, .next_state = conv_xcch_next_state, };
But in first source there is. And I am not sure, which definition is correct. I paid my attention here, because I just integrated some tests from 'osmo-bts/tests/bursts' to libosmocoding. PDCH test fails until there is puncture in both definitions. When I removed punctures from definitions, test passed.
So, my question is do we need punctures for both CS2 & CS3?
With best regards, Vadim Yanitskiy.
2016-09-06 10:46 GMT+07:00 Sylvain Munaut 246tnt@gmail.com:
Hi,
With regards to convolutional coding, the main difference between GPRS and EGPRS will be the use of tail-biting recursive codes.
Max, can the utility generate recursive state tables?
Yes
Cheers,
Sylvain
From my memory:
* CS1 is rate 1/2 conv code with no puncturing. * CS2+3 apply different puncturing amounts on that conv code * CS4 has no error correction (=100% puncturing)
Why the code differs, I don't know. Maybe it was never completed in conv_gen?
Hi Harald,
Why the code differs, I don't know. Maybe it was never completed in conv_gen?
It seems, I found the answer. If you look closer at gsm0503_coding.c/gsm0503_pdtch_encode(), you will see:
// Around line 1300, CS-2 encoding // ...
osmo_pbit2ubit_ext(conv, 3, l2_data, 0, 271, 1); usf = l2_data[0] & 0x7;
osmo_crc16gen_set_bits(&gsm0503_cs234_crc16, conv + 3, 271, conv + 3 + 271);
memcpy(conv, gsm0503_usf2six[usf], 6);
osmo_conv_encode(&gsm0503_cs2, conv, cB);
// for (i = 0, j = 0; i < 588; i++) if (!gsm0503_puncture_cs2[i]) cB[j++] = cB[i];
hl_hn = gsm0503_pdtch_hl_hn_ubit[1];
// ...
So, as I understand, this code applies puncture itself. But in our case, the osmo_conv_encode() applies puncture too, from the gsm0503_cs2 convolutional code definition. The same things happens in the gsm0503_pdtch_decode().
There are two possible ways to go in my mind:
1) The simplest way is to merely remove puncture from both gsm0503_cs2 and gsm0503_cs3 definitions, but it may break some code, which already uses current variant. 2) Change exactly the gsm0503_coding.c to use puncture from shared convolutional code definitions. For me, this way is prefered, but I don't know how to change the code yet.
Any opinions? Does any project use the convolutional code definitions from 'utils/conv_gen.py'?
With best regards, Vadim Yanitskiy.
2016-09-29 21:40 GMT+07:00 Harald Welte laforge@gnumonks.org:
From my memory:
- CS1 is rate 1/2 conv code with no puncturing.
- CS2+3 apply different puncturing amounts on that conv code
- CS4 has no error correction (=100% puncturing)
Why the code differs, I don't know. Maybe it was never completed in conv_gen?
-- Sent from a mobile device. Please excuse my brevity.
Hi.
I think the puncturing code should be part of the library (otherwise every user will have to reimplement the same thing again and again). Unfortunately I could not find any tests for the code in question so the only way to test it is by:
- add code with puncturing to the library
- replace manual puncturing with generic call
- run gprs with fixes CS* to make sure necessary code is triggered
- verify that it works as before
On 10/02/2016 09:11 AM, Vadim Yanitskiy wrote:
Hi Harald,
Why the code differs, I don't know. Maybe it was never completed in conv_gen?
It seems, I found the answer. If you look closer at gsm0503_coding.c/gsm0503_pdtch_encode(), you will see:
// Around line 1300, CS-2 encoding // ...
osmo_pbit2ubit_ext(conv, 3, l2_data, 0, 271, 1); usf = l2_data[0] & 0x7;
osmo_crc16gen_set_bits(&gsm0503_cs234_crc16, conv + 3, 271, conv + 3 + 271);
memcpy(conv, gsm0503_usf2six[usf], 6);
osmo_conv_encode(&gsm0503_cs2, conv, cB);
// for (i = 0, j = 0; i < 588; i++) if (!gsm0503_puncture_cs2[i]) cB[j++] = cB[i];
hl_hn = gsm0503_pdtch_hl_hn_ubit[1];
// ...
So, as I understand, this code applies puncture itself. But in our case, the osmo_conv_encode() applies puncture too, from the gsm0503_cs2 convolutional code definition. The same things happens in the gsm0503_pdtch_decode().
There are two possible ways to go in my mind:
- The simplest way is to merely remove puncture from both gsm0503_cs2 and gsm0503_cs3 definitions, but it may break some code, which already uses current variant.
- Change exactly the gsm0503_coding.c to use puncture from shared convolutional code definitions. For me, this way is prefered, but I don't know how to change the code yet.
Any opinions? Does any project use the convolutional code definitions from 'utils/conv_gen.py'?
With best regards, Vadim Yanitskiy.
2016-09-29 21:40 GMT+07:00 Harald Welte <laforge@gnumonks.org mailto:laforge@gnumonks.org>:
From my memory: * CS1 is rate 1/2 conv code with no puncturing. * CS2+3 apply different puncturing amounts on that conv code * CS4 has no error correction (=100% puncturing) Why the code differs, I don't know. Maybe it was never completed in conv_gen? -- Sent from a mobile device. Please excuse my brevity.
-
Harald Welte -
Max -
Neels Hofmeyr -
Sylvain Munaut -
Tom Tsou -
Vadim Yanitskiy