Dear Osmocom community,
pySim-prog was nice when there were only 5 parameters on a SIM that we could program, and where the use case was pretty limited. Today, we have SIM/USIM/ISIM cards with hundreds of files and even more parameters to program. We cannot add a command line argument for each file to pySim-prog.
Instead, this introduces an interactive command-line shell / REPL, in which one can navigate the file system of the card, read and update files both in raw format and in decoded/parsed format.
The idea is primarily inspired by Henryk Ploetz' venerable cyberflex-shell, but implemented on a more modern basis using the cmd2 python module.
You can see the very first prototype in the laforge/shell branch of pysim.git
You can do things with it like this:
===> Start-up and authenticate with adm pin ---------------------------------------------------------------------- $ ./pysim-shell.py -p 0 Using PC/SC reader interface Autodetected card type: sysmoISIM-SJA2 AIDs on card: ['a0000000871002ffffffff8907090000', 'a0000000871004ffffffff8907090000'] Welcome to pySim-shell! pySIM-shell (3f00)> verify_adm 92990895 ----------------------------------------------------------------------
===> interactive help ---------------------------------------------------------------------- pySIM-shell (3f00)> help
Documented commands (use 'help -v' for verbose/'help <topic>' for details):
ISO7816 Commands ================ read_binary select_adf select_file update_binary update_record verify_chv
pySim Commands ============== intro verify_adm
USIM Commands ============= read_ehplmn ust_service_activate ust_service_deactivate
pySim-shell built-in commands ============================= alias help macro quit run_script shell edit history py run_pyscript set shortcuts
----------------------------------------------------------------------
===> more interactive help ---------------------------------------------------------------------- pySIM-shell (3f00)> help read_binary usage: read_binary [-h] [--file-id FILE_ID] [--offset OFFSET] [--length LENGTH] [--record-nr RECORD_NR]
Read binary data from a transparent EF
optional arguments: -h, --help show this help message and exit --file-id FILE_ID File ID --offset OFFSET Byte offset for start of read --length LENGTH Number of bytes to read --record-nr RECORD_NR Number of record to read ----------------------------------------------------------------------
===> navigating the FS and reading files ---------------------------------------------------------------------- pySIM-shell (3f00)> select_file 7f20 ['622c8202782183027f20a509800171830400018d088a01058b032f0601c60f90017083010183018183010a83010b'] pySIM-shell (3f00/7f20)> read_binary --file-id 6f07 089910070000400310 ----------------------------------------------------------------------
===> interaction with local filesystem, i.e. I/O redirect + shell commands ---------------------------------------------------------------------- pySIM-shell (3f00)> select_adf a0000000871002 pySIM-shell (a0000000871002)> select_file 5f3b pySIM-shell (a0000000871002/5f3b)> read_binary --file-id 4f20 > /tmp/f pySIM-shell (a0000000871002/5f3b)> !cat /tmp/f ffffffffffffffff07 ----------------------------------------------------------------------
===> piping output through shell tools like grep ---------------------------------------------------------------------- pySIM-shell (3f00)> read_ust | grep 86 Service 86 - Allowed CSG Lists and corresponding indications ----------------------------------------------------------------------
===> enabling/disabling services ---------------------------------------------------------------------- pySIM-shell (3f00/7f20)> ust_service_activate 123 pySIM-shell (3f00/7f20)> ust_service_deactivate 123 ----------------------------------------------------------------------
It's a very first prototype, but it is really promising.
The major tasks I see to make this go anywhere is:
* have "File" class with encoder/decoder methods, which are registered automatically with a 'file system' layer that knows about the DF/ADF hierarchy ** this allows us to have a "read-decoded" command, which will call the decode method of the file, automatically resolved by the selected FID/path * automatic mapping of file-name -> FID and FID -> file name ** when printing (like in the path), use the human-readable names ** allow users to use human-readable names in SELECT * decode + display the TLVs / FCPs after a SELECT (like cyberflex-shell * ability to enable/disable APDU trace * dynamically register/deregster commands based on the path, i.e. offer USIM commands only when in ADF_USIM
We have quite a bit of that infrastructure in the c-language libosmosim, (part of libosmocore.git), but unfortunately not in python :/
Let me know if anyone is interested in joining this effort.
Regards, Harald
Harald,
I think this is a great idea and I'd love to help out, but I'm new to working on open source projects. What's the best way for me to get started in a way that would be helpful?
Best,
Bryan
On Fri, Jan 8, 2021 at 5:54 PM Harald Welte laforge@gnumonks.org wrote:
Dear Osmocom community,
pySim-prog was nice when there were only 5 parameters on a SIM that we could program, and where the use case was pretty limited. Today, we have SIM/USIM/ISIM cards with hundreds of files and even more parameters to program. We cannot add a command line argument for each file to pySim-prog.
Instead, this introduces an interactive command-line shell / REPL, in which one can navigate the file system of the card, read and update files both in raw format and in decoded/parsed format.
The idea is primarily inspired by Henryk Ploetz' venerable cyberflex-shell, but implemented on a more modern basis using the cmd2 python module.
You can see the very first prototype in the laforge/shell branch of pysim.git
You can do things with it like this:
===> Start-up and authenticate with adm pin
$ ./pysim-shell.py -p 0 Using PC/SC reader interface Autodetected card type: sysmoISIM-SJA2 AIDs on card: ['a0000000871002ffffffff8907090000', 'a0000000871004ffffffff8907090000'] Welcome to pySim-shell! pySIM-shell (3f00)> verify_adm 92990895
===> interactive help
pySIM-shell (3f00)> help
Documented commands (use 'help -v' for verbose/'help <topic>' for details):
ISO7816 Commands
read_binary select_adf select_file update_binary update_record verify_chv
pySim Commands
intro verify_adm
USIM Commands
read_ehplmn ust_service_activate ust_service_deactivate
pySim-shell built-in commands
alias help macro quit run_script shell edit history py run_pyscript set shortcuts
===> more interactive help
pySIM-shell (3f00)> help read_binary usage: read_binary [-h] [--file-id FILE_ID] [--offset OFFSET] [--length LENGTH] [--record-nr RECORD_NR]
Read binary data from a transparent EF
optional arguments: -h, --help show this help message and exit --file-id FILE_ID File ID --offset OFFSET Byte offset for start of read --length LENGTH Number of bytes to read --record-nr RECORD_NR Number of record to read
===> navigating the FS and reading files
pySIM-shell (3f00)> select_file 7f20
['622c8202782183027f20a509800171830400018d088a01058b032f0601c60f90017083010183018183010a83010b'] pySIM-shell (3f00/7f20)> read_binary --file-id 6f07 089910070000400310
===> interaction with local filesystem, i.e. I/O redirect + shell commands
pySIM-shell (3f00)> select_adf a0000000871002 pySIM-shell (a0000000871002)> select_file 5f3b pySIM-shell (a0000000871002/5f3b)> read_binary --file-id 4f20 > /tmp/f pySIM-shell (a0000000871002/5f3b)> !cat /tmp/f ffffffffffffffff07
===> piping output through shell tools like grep
pySIM-shell (3f00)> read_ust | grep 86 Service 86 - Allowed CSG Lists and corresponding indications
===> enabling/disabling services
pySIM-shell (3f00/7f20)> ust_service_activate 123 pySIM-shell (3f00/7f20)> ust_service_deactivate 123
It's a very first prototype, but it is really promising.
The major tasks I see to make this go anywhere is:
- have "File" class with encoder/decoder methods, which are registered automatically with a 'file system' layer that knows about the DF/ADF hierarchy
** this allows us to have a "read-decoded" command, which will call the decode method of the file, automatically resolved by the selected FID/path
- automatic mapping of file-name -> FID and FID -> file name
** when printing (like in the path), use the human-readable names ** allow users to use human-readable names in SELECT
- decode + display the TLVs / FCPs after a SELECT (like cyberflex-shell
- ability to enable/disable APDU trace
- dynamically register/deregster commands based on the path, i.e. offer USIM commands only when in ADF_USIM
We have quite a bit of that infrastructure in the c-language libosmosim, (part of libosmocore.git), but unfortunately not in python :/
Let me know if anyone is interested in joining this effort.
Regards, Harald --
- Harald Welte laforge@gnumonks.org
============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
Hi Bryan,
On Sat, Jan 09, 2021 at 10:00:58AM -0500, bryan coxwell wrote:
I think this is a great idea and I'd love to help out, but I'm new to working on open source projects. What's the best way for me to get started in a way that would be helpful?
thanks for reaching out. By now most of the APIs/architecture is sorted out, so you could
a) make sure that ts_51_011.py (sim) / ts_31_102.py (usim) / ts_31_103.py (isim) contain class definitions for all of the files specified in the latest Release 16 3GPP specifications
b) write encoder and decoder methods within the EF classes for the various files of interest. See the EF_IMSI() as an example. The Decoder method should return a python dict (possibly a hierarchy of dicts) and return the binary representation, while the encoder method should take the dict and produce the binary representation.
Regards, Harald
Sounds good to me. 51.011 looks to be unchanged, which I guess is expected. Working on updating 31.102 now.
Thanks,
Bryan Coxwell
On Jan 11, 2021, at 8:10 AM, Harald Welte laforge@gnumonks.org wrote:
Hi Bryan,
On Sat, Jan 09, 2021 at 10:00:58AM -0500, bryan coxwell wrote: I think this is a great idea and I'd love to help out, but I'm new to working on open source projects. What's the best way for me to get started in a way that would be helpful?
thanks for reaching out. By now most of the APIs/architecture is sorted out, so you could
a) make sure that ts_51_011.py (sim) / ts_31_102.py (usim) / ts_31_103.py (isim) contain class definitions for all of the files specified in the latest Release 16 3GPP specifications
b) write encoder and decoder methods within the EF classes for the various files of interest. See the EF_IMSI() as an example. The Decoder method should return a python dict (possibly a hierarchy of dicts) and return the binary representation, while the encoder method should take the dict and produce the binary representation.
Regards, Harald --
- Harald Welte laforge@gnumonks.org http://laforge.gnumonks.org/
============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
On Fri, Jan 08, 2021 at 11:53:51PM +0100, Harald Welte wrote:
You can see the very first prototype in the laforge/shell branch of pysim.git
Just a quick update: During past weeks I've been on and off spending a bit of time to bring this idea further along. It's already quite mature now.
* many bugs have been fixed
* SW1/SW2 are now parsed into human readable strings
* many file specific encoders/decoders added
* return to 'SELECT" command is now parsed into json:
---------------------------------------------------------------------- pySIM-shell (MF)> select ADF.ISIM { "file_descriptor": { "shareable": true, "file_type": "df", "structure": "no_info_given" }, "file_identifier": "FF01", "df_name": "A0000000871004FFFFFFFF8907090000", "proprietary_info": { "uicc_characteristics": "71", "available_memory": 101640 }, "life_cycle_status_int": "operational_activated" "security_attrib_compact": "00", "pin_status_template_do": "900170830101830181830 } pySIM-shell (MF/ADF.ISIM)> select EF.P-CSCF { "file_descriptor": { "shareable": true, "file_type": "working_ef", "structure": "linear_fixed", "record_len": 0, "num_of_rec": 0 }, "file_identifier": "6F09", "proprietary_info": { "proprietary_D0": "20", "proprietary_D2": "0F" }, "life_cycle_status_int": "operational_activated" "security_attrib_ref_expanded": "6F0603", "file_size": 1024, "short_file_id": "" } ----------------------------------------------------------------------
It's already reached a state where it can be used to perform useful tasks. Be warned: It's still very early and nothing has been tested with cards other than sysmoISIM-SJA2 at this point.
I think within the next weeks I'll probably try to clean up the patches and get the current state merged to master. There's still a lot of work to be done, including:
* option for storing per-ICCID ADM keys in some config file, so you don't have to enter them over and over again when frequently changing with the same cards * bulk read / write commands to read/write all records within one file * decide on some general rules on how to strucure the JSON output, such as including the file path and record number in some metadata? * automatically pad filss/records with fff to their size as determined from the card * ability to select/read/write arbitrary FID, e.g. for non-standard proprietary files that are not part of the ETSI/3GPP specs
Regards, Harald
Dear Osmocom community,
two months later, a lot of work has been going into pySim-shell. The following features are now available from pysim.git master:
* we now have a 'tree' command to list the filesystem hierarchy
* you can now select arbitrary files by FID, even those not specified in 3GPP
* there is now an 'export' command, which will iterate over all records in all files and dump them in an output format that can be used by pySim-shell itself to restore the data
* we have the start of a user manual including a reference to the various pySim-shell comamnds. A preview is available at https://people.osmocom.org/laforge/tmp/pysim-doc-test/html/
* you can now use python setuptools to build and install dependencies, etc.
* pySim-shell can retriev the card-specific ADM PIN from a CSV file in your home
The latest invention (just pushed to gerrit for review) is the integration of JSONpath support, wich allows you to specify certain parts of a decoded file or record[s] to be updated. If you've never heard of JSONpath: It is to JSON what XPath is to XML.
First example on how to use this:
------------------------------------------ pySIM-shell (MF/ADF.USIM/EF.FPLMN)> read_binary_decoded [ { "mcc": 262, "mnc": 42 }, { "mcc": 262, "mnc": 42 }, { "mcc": 262, "mnc": 42 }, { "mcc": 262, "mnc": 42 } ] pySIM-shell (MF/ADF.USIM/EF.FPLMN)> update_binary_decoded --json-path [*] null pySIM-shell (MF/ADF.USIM/EF.FPLMN)> read_binary_decoded [ null, null, null, null ] pySIM-shell (MF/ADF.USIM/EF.FPLMN)> update_binary_decoded --json-path [2] '{"mcc":"262", "mnc":"42"}' pySIM-shell (MF/ADF.USIM/EF.FPLMN)> read_binary_decoded [ null, null, { "mcc": 262, "mnc": 42 }, null ] ------------------------------------------
Second example, illustrating its use in more complex JSON types:
------------------------------------------ pySIM-shell (MF/ADF.USIM/EF.AD)> read_binary_decoded { "ms_operation_mode": "normal", "specific_facilities": { "ofm": true }, "len_of_mnc_in_imsi": 2 } pySIM-shell (MF/ADF.USIM/EF.AD)> update_binary_decoded --json-path specific_facilities.ofm false pySIM-shell (MF/ADF.USIM/EF.AD)> read_binary_decoded { "ms_operation_mode": "normal", "specific_facilities": { "ofm": false }, "len_of_mnc_in_imsi": 2 } ------------------------------------------
Stay tuned for more pySim developments coming up. Any feedback is welcome - as is help with writing encoder/decoder methods for all the files that don't have any yet.
Regards, Harald
Harald Welte wrote:
- we now have a 'tree' command to list the filesystem hierarchy
And just how do you get the card to tell you what selectable file IDs exist? I haven't seen anything like an ls operation in either the classic GSM 11.11 SIM protocol or the UICC protocol, thus the only way (that I know of) to find out what selectable file IDs exist is to do a brute force search of the 16-bit file ID space at every directory level. First select MF, then try selecting every possible 16-bit file ID from 0000 to FFFF (only skipping 3F00 for MF itself), and note which return something other than "not found" error. Follow up with a GET RESPONSE command for every SELECT which succeeded, parse the response, and report the findings. For all found file IDs which turn out to be DFs when the response is parsed, note those DF file IDs, and then repeat the brute force search inside every found DF - and then in any found nested DFs too.
This brute force search is implemented in fc-simtool and fc-uicc-tool programs in my fc-sim-tools suite, my competitor to pySim:
https://www.freecalypso.org/hg/fc-sim-tools/
As one would naturally expect, such brute force searches are painfully slow - IIRC, bfsearch-mf of sysmoISIM-SJA2 (just the MF tree, ADF trees have to be searched separately with bfsearch-adf) took about an hour, using HID Omnikey 3121 card reader, same model as the one currently sold in Sysmocom webshop - using an o'scope, I observed that it clocks the card at 4.8 MHz, almost up to the spec limit of 5 MHz.
Because these brute force searches are so slow, I collect the captures and check them into my source repository under the data directory - so if you are curious to see what undocumented proprietary files exist on both Sysmocom and Grcard SIMs (whose existence cannot be discovered in any other way than this bfsearch), just look in the repository linked above. :-)
I am not able to run pySim-shell on my Slackware system without expending more effort than I can currently justify, but I have glanced at the Python code, and I don't see anything like the just described brute force search - nor do I see it issuing any kind of secret undocumented ls-type APDU commands to the card - thus I am guessing that this 'tree' command displays nothing more than the tool's hard-coded knowledge of what files "should" exist at each given directory level, rather than what is actually found to exist. If I got this part wrong, then someone please explain what this command *actually* returns, and how it obtains this knowledge - I don't know of any way other than a brute force search.
M~
Hi Mychaela,
Without looking at the code I assume the way it works is that based on the service table the tool knows what files are supposed to be on the card and queries for them - just like an “ordinary phone” does. The point of the ST is exactly to avoid unnecessary bruteforcing/lookups of files that don’t exist. However it is correct to say that finding a file hidden on purpose (i.e. missing entry from the ST) could only be done via bruteforce imho.
In case I am wrong sorry, didn’t mean to mislead anybody.
Cheers, Domi
07.04.2021 dátummal, 2:41 időpontban Mychaela Falconia mychaela.falconia@gmail.com írta:
Harald Welte wrote:
- we now have a 'tree' command to list the filesystem hierarchy
And just how do you get the card to tell you what selectable file IDs exist? I haven't seen anything like an ls operation in either the classic GSM 11.11 SIM protocol or the UICC protocol, thus the only way (that I know of) to find out what selectable file IDs exist is to do a brute force search of the 16-bit file ID space at every directory level. First select MF, then try selecting every possible 16-bit file ID from 0000 to FFFF (only skipping 3F00 for MF itself), and note which return something other than "not found" error. Follow up with a GET RESPONSE command for every SELECT which succeeded, parse the response, and report the findings. For all found file IDs which turn out to be DFs when the response is parsed, note those DF file IDs, and then repeat the brute force search inside every found DF - and then in any found nested DFs too.
This brute force search is implemented in fc-simtool and fc-uicc-tool programs in my fc-sim-tools suite, my competitor to pySim:
https://www.freecalypso.org/hg/fc-sim-tools/
As one would naturally expect, such brute force searches are painfully slow - IIRC, bfsearch-mf of sysmoISIM-SJA2 (just the MF tree, ADF trees have to be searched separately with bfsearch-adf) took about an hour, using HID Omnikey 3121 card reader, same model as the one currently sold in Sysmocom webshop - using an o'scope, I observed that it clocks the card at 4.8 MHz, almost up to the spec limit of 5 MHz.
Because these brute force searches are so slow, I collect the captures and check them into my source repository under the data directory - so if you are curious to see what undocumented proprietary files exist on both Sysmocom and Grcard SIMs (whose existence cannot be discovered in any other way than this bfsearch), just look in the repository linked above. :-)
I am not able to run pySim-shell on my Slackware system without expending more effort than I can currently justify, but I have glanced at the Python code, and I don't see anything like the just described brute force search - nor do I see it issuing any kind of secret undocumented ls-type APDU commands to the card - thus I am guessing that this 'tree' command displays nothing more than the tool's hard-coded knowledge of what files "should" exist at each given directory level, rather than what is actually found to exist. If I got this part wrong, then someone please explain what this command *actually* returns, and how it obtains this knowledge - I don't know of any way other than a brute force search.
M~
-
bryan coxwell -
Harald Welte -
Harald Welte -
Mychaela Falconia -
Tomcsanyi, Domonkos