Dear osmocom community,
This email to point to a recent development I made on CSN.1, which might be of interest to some of you. I extracted and consolidated all CSN.1 definitions from 24.008, 44.018 and 44.060 (the "consolidation" part was quite hard...). I have also developped a CSN.1 to Python translater, and a runtime to encode / decode any CSN.1 structures.
This is available here: https://github.com/ANSSI-FR/pycrate/tree/master/pycrate_csn1dir and there: https://github.com/ANSSI-FR/pycrate/tree/master/pycrate_csn1
And there are some quick explanations on how to use this in the wiki: https://github.com/ANSSI-FR/pycrate/wiki/Using-the-pycrate-csn1-translator-a...
Regards Benoit
Hi Benoit,
On Wed, Aug 01, 2018 at 09:34:27PM +0200, benoit michau wrote:
This email to point to a recent development I made on CSN.1, which might be of interest to some of you.
Thanks a lot for implementing this in the first place, and also for sharing information about it here.
It's amazing that we finally seem to be getting good CSN.1 tools
I extracted and consolidated all CSN.1 definitions from 24.008, 44.018 and 44.060 (the "consolidation" part was quite hard...).
I think the biggest problems are the many syntax errors in the official CSN.1 syntax in the specs. I once tried to parse it from proprietaryt CSN.1 tools, and it was a nightmare. In fact, there are companies selling the cleaned up and fully confirming syntax for money.
Somebody should submit your fixed versin to 3GPP so they can release a correct syntax at least...
I have also developped a CSN.1 to Python translater, and a runtime to encode / decode any CSN.1 structures.
Thanks. What we need to figure out now is how to make best use of this from within Osmocom. OsmoPCU is - as you know - implemented in C. Rewriting it in python is unfortunately not an option, if not alone for performance reasons :/
However, it would be great if we could benefit from your work in other ways, such as
a) for unit testing our code (particularly the hand-written encoders/decoders)
b) for integration testing. Unfortunately TTCN3/TITAN doesn't have any CSN.1 tools, so we cannot use a "trusted, aut-generated" encoder/decoder from our existing integration tests. However, maybe there are ways we can integrate your python CSN.1 with TTCN-3? MAybe it's possible to auto-generate ASN.1 syntax and an ASN.1(BER)-to-CSN.1 transcoder in python. Then we could use TTCN3 templates in the TITAN world, and we'd simply call a function to encode as ASN.1-BER, and then into your transcoder to create CSN.1 from that (and vice versa)?
c) something like a MS-side (E)GPRS RLC/MAC implementation in python. We could then use this over "virtual/simulated" radio against OsmoPCU.
d) maybe generate C / C++ encoder/decoder from your python framework, since it fully parses and understands the syntax tree...
I'm not a python expert and I have not used your code yet, so I'm not sure what is realistic/possible. But for sure it would be really great to use generated parser/encoders for CSN.1 in one way or another...
If you (or anyone else) have any further ideas to share, please feel free to do so!
Regards, Harald
Hi Harald,
vacation time, hence slow response. Thank you for your feedback. Here are few answers about your several ideas : b) converting CSN.1 to ASN.1 and back is not straightforward ; there are some structures in CSN.1 (such as recursive construct with a "continuation" bit) which may be hard to translate to ASN.1. I am not confident this would work well. c) this would be great. I was wondering if there exists a real-world GPRS L1 and transceiver that we could use, in addition to the direct / simulated connection to the OsmoPCU ? I am not aware of any unfortunately. d) I am not comfortable to generate a full C or C++ code. On the other side, it would be quite easy to generate headers with proper structs for each object. This would have to link with an appropriate C / C++ encoding runtime.
Regards Benoit
2018-08-01 21:58 GMT+02:00 Harald Welte laforge@gnumonks.org:
Hi Benoit,
On Wed, Aug 01, 2018 at 09:34:27PM +0200, benoit michau wrote:
This email to point to a recent development I made on CSN.1, which might be of interest to some of you.
Thanks a lot for implementing this in the first place, and also for sharing information about it here.
It's amazing that we finally seem to be getting good CSN.1 tools
I extracted and consolidated all CSN.1 definitions from 24.008, 44.018 and 44.060 (the "consolidation" part was quite hard...).
I think the biggest problems are the many syntax errors in the official CSN.1 syntax in the specs. I once tried to parse it from proprietaryt CSN.1 tools, and it was a nightmare. In fact, there are companies selling the cleaned up and fully confirming syntax for money.
Somebody should submit your fixed versin to 3GPP so they can release a correct syntax at least...
I have also developped a CSN.1 to Python translater, and a runtime to encode / decode any CSN.1 structures.
Thanks. What we need to figure out now is how to make best use of this from within Osmocom. OsmoPCU is - as you know - implemented in C. Rewriting it in python is unfortunately not an option, if not alone for performance reasons :/
However, it would be great if we could benefit from your work in other ways, such as
a) for unit testing our code (particularly the hand-written encoders/decoders)
b) for integration testing. Unfortunately TTCN3/TITAN doesn't have any CSN.1 tools, so we cannot use a "trusted, aut-generated" encoder/decoder from our existing integration tests. However, maybe there are ways we can integrate your python CSN.1 with TTCN-3? MAybe it's possible to auto-generate ASN.1 syntax and an ASN.1(BER)-to-CSN.1 transcoder in python. Then we could use TTCN3 templates in the TITAN world, and we'd simply call a function to encode as ASN.1-BER, and then into your transcoder to create CSN.1 from that (and vice versa)?
c) something like a MS-side (E)GPRS RLC/MAC implementation in python. We could then use this over "virtual/simulated" radio against OsmoPCU.
d) maybe generate C / C++ encoder/decoder from your python framework, since it fully parses and understands the syntax tree...
I'm not a python expert and I have not used your code yet, so I'm not sure what is realistic/possible. But for sure it would be really great to use generated parser/encoders for CSN.1 in one way or another...
If you (or anyone else) have any further ideas to share, please feel free to do so!
Regards, Harald --
- Harald Welte laforge@gnumonks.org http://laforge.gnumonks.org/
============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
Hi,
This is available here: https://github.com/ANSSI-FR/pycrate/tree/master/pycrate_csn1dir and there: https://github.com/ANSSI-FR/pycrate/tree/master/pycrate_csn1
Very cool !
I've just tried parsing the GMR-1 CSN with it. Doesn't work out of the box because of some minor syntax issues, but minor tweaks and then it seems to load them fine.
I'm hoping to generate some template code to parse system informations in wireshark with this :)
Cheers,
Sylvain
Hello,
I did not know GMR-1 was so close from GSM / GPRS at the RRC / NAS layer. I found those GMR-1 specs which seem the most recent, with CSN.1 structures definition: - for RRC: https://www.etsi.org/deliver/etsi_ts/101300_101399/1013760413/03.05.01_60/ts... - for NAS: https://www.etsi.org/deliver/etsi_ts/101300_101399/1013760408/03.04.01_60/ts...
I may be able to process them and create corresponding directories in pycrate.
Regards, Benoit
2018-08-01 22:47 GMT+02:00 Sylvain Munaut 246tnt@gmail.com:
Hi,
This is available here: https://github.com/ANSSI-FR/pycrate/tree/master/pycrate_csn1dir and there: https://github.com/ANSSI-FR/pycrate/tree/master/pycrate_csn1
Very cool !
I've just tried parsing the GMR-1 CSN with it. Doesn't work out of the box because of some minor syntax issues, but minor tweaks and then it seems to load them fine.
I'm hoping to generate some template code to parse system informations in wireshark with this :)
Cheers,
Sylvain
Hi,
I did not know GMR-1 was so close from GSM / GPRS at the RRC / NAS layer.
GMR-1 is heavily derived from GSM. Mostly just all the physical medium specific stuff had to be adapted.
I found those GMR-1 specs which seem the most recent, with CSN.1 structures definition:
Yes, and there are also some in
https://www.etsi.org/deliver/etsi_ts/101300_101399/1013760408/03.01.01_60/ts...
Unfortunately that doc has plenty of reference to the "previous" versions : https://www.etsi.org/deliver/etsi_ts/101300_101399/1013760408/02.03.01_60/ts... https://www.etsi.org/deliver/etsi_ts/101300_101399/1013760408/01.03.01_60/ts...
I may be able to process them and create corresponding directories in pycrate.
That would be really nice :)
Do you have some tool to automatically extract the definitions from the pdf ?
Cheers,
Sylvain
Hi, I have been using the scripts here: https://github.com/ANSSI-FR/pycrate/tree/master/pycrate_csn1dir/generator/ - extract.py: to extract csn.1 definitions from those 3GPP 44.018, 44.060 and 24.008 after converting them to txt files with microsoft word. I am not sure it would work well with ETSI specs in PDF converted to text. - gen.py: to convert csn.1 specs to Python files by batch. Beware some pathes are hardcoded, this may have to be adjusted in the header of those scripts.
Regards Benoit
2018-08-22 23:01 GMT+02:00 Sylvain Munaut 246tnt@gmail.com:
Hi,
I did not know GMR-1 was so close from GSM / GPRS at the RRC / NAS layer.
GMR-1 is heavily derived from GSM. Mostly just all the physical medium specific stuff had to be adapted.
I found those GMR-1 specs which seem the most recent, with CSN.1 structures definition:
Yes, and there are also some in
https://www.etsi.org/deliver/etsi_ts/101300_101399/1013760408/03.01.01_60/ts...
Unfortunately that doc has plenty of reference to the "previous" versions : https://www.etsi.org/deliver/etsi_ts/101300_101399/1013760408/02.03.01_60/ts... https://www.etsi.org/deliver/etsi_ts/101300_101399/1013760408/01.03.01_60/ts...
I may be able to process them and create corresponding directories in pycrate.
That would be really nice :)
Do you have some tool to automatically extract the definitions from the pdf ?
Cheers,
Sylvain
-
benoit michau -
Harald Welte -
Neels Hofmeyr -
Sylvain Munaut