neels has uploaded this change for review. ( https://gerrit.osmocom.org/c/pysim/+/40199?usp=email )
Change subject: personalization: allow reading back multiple values from PES
......................................................................
personalization: allow reading back multiple values from PES
Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2
---
M pySim/esim/saip/personalization.py
1 file changed, 26 insertions(+), 18 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/pysim refs/changes/99/40199/1
diff --git a/pySim/esim/saip/personalization.py b/pySim/esim/saip/personalization.py
index ca37e3a..34d7df9 100644
--- a/pySim/esim/saip/personalization.py
+++ b/pySim/esim/saip/personalization.py
@@ -216,26 +216,24 @@
@classmethod
def get_values_from_pes(cls, pes: ProfileElementSequence) -> Generator:
- '''This is what subclasses implement: yield all values from a decoded profile package.
+ """This is what subclasses implement: yield all values from a decoded profile package.
Find all values in the pes, and yield them decoded to a valid cls.input_value format.
Should be a generator function, i.e. use 'yield' instead of 'return'.
- Usage example:
+ Yielded value must be a dict(). Usually, an implementation will return only one key, like
- cls = esim.saip.personalization.Iccid
- # use a set() to get a list of unique values from all results
- vals = set( cls.get_values_from_pes(pes) )
- if len(vals) != 1:
- raise ValueError(f'{cls.name}: need exactly one value, got {vals}')
- # the set contains a single value, return it
- return vals.pop()
+ { "ICCID": "1234567890123456789" }
+
+ Some implementations have more than one value to return, like
+
+ { "IMSI": "00101012345678", "IMSI-ACC" : "5" }
Implementation example:
for pe in pes:
if my_condition(pe):
- yield b2h(my_bin_value_from(pe))
- '''
+ yield { cls.name: b2h(my_bin_value_from(pe)) }
+ """
pass
@classmethod
@@ -364,12 +362,12 @@
def get_values_from_pes(cls, pes: ProfileElementSequence):
padded = b2h(pes.get_pe_for_type('header').decoded['iccid'])
iccid = unrpad(padded)
- yield iccid
+ yield { cls.name: iccid }
for pe in pes.get_pes_for_type('mf'):
iccid_pe = pe.decoded.get('ef-iccid', None)
if iccid_pe:
- yield dec_iccid(b2h(file_tuples_content_as_bytes(iccid_pe)))
+ yield { cls.name: dec_iccid(b2h(file_tuples_content_as_bytes(iccid_pe))) }
class Imsi(DecimalParam):
"""Configurable IMSI. Expects value to be a string of digits. Automatically sets the ACC to
@@ -396,8 +394,13 @@
def get_values_from_pes(cls, pes: ProfileElementSequence):
for pe in pes.get_pes_for_type('usim'):
imsi_pe = pe.decoded.get('ef-imsi', None)
+ acc_pe = pe.decoded.get('ef-acc', None)
+ y = {}
if imsi_pe:
- yield dec_imsi(b2h(file_tuples_content_as_bytes(imsi_pe)))
+ y[cls.name] = dec_imsi(b2h(file_tuples_content_as_bytes(imsi_pe)))
+ if acc_pe:
+ y[cls.name + '-ACC'] = b2h(file_tuples_content_as_bytes(acc_pe))
+ yield y
class SdKey(BinaryParam):
@@ -438,7 +441,7 @@
for key in pe.decoded['keyList']:
if key['keyIdentifier'][0] == cls.key_id and key['keyVersionNumber'][0] == cls.kvn:
if len(key['keyComponents']) >= 1:
- yield b2h(key['keyComponents'][0]['keyData'])
+ yield { cls.name: b2h(key['keyComponents'][0]['keyData']) }
class SdKeyScp80_01(SdKey):
kvn = 0x01
@@ -582,7 +585,7 @@
for pukCodes in obtain_all_pe_from_pelist(mf_pes, 'pukCodes'):
for pukCode in pukCodes.decoded['pukCodes']:
if pukCode['keyReference'] == cls.keyReference:
- yield cls.decimal_hex_to_str(pukCode['pukValue'])
+ yield { cls.name: cls.decimal_hex_to_str(pukCode['pukValue']) }
class Puk1(Puk):
is_abstract = False
@@ -623,13 +626,14 @@
@classmethod
def _read_all_pinvalues_from_pe(cls, pe: ProfileElement):
+ "This is a separate function because subclasses may feed different pe arguments."
for pinCodes in obtain_all_pe_from_pelist(pe, 'pinCodes'):
if pinCodes.decoded['pinCodes'][0] != 'pinconfig':
continue
for pinCode in pinCodes.decoded['pinCodes'][1]:
if pinCode['keyReference'] == cls.keyReference:
- yield cls.decimal_hex_to_str(pinCode['pinValue'])
+ yield { cls.name: cls.decimal_hex_to_str(pinCode['pinValue']) }
@classmethod
def get_values_from_pes(cls, pes: ProfileElementSequence):
@@ -698,7 +702,11 @@
algoConfiguration = pe.decoded['algoConfiguration']
if algoConfiguration[0] != 'algoParameter':
continue
- yield algoConfiguration[1][cls.algo_config_key]
+ val = algoConfiguration[1][cls.algo_config_key]
+ if isinstance(val, bytes):
+ val = b2h(val)
+ # if it is an int (algorithmID), just pass thru as int
+ yield { cls.name: val }
class AlgorithmID(DecimalParam, AlgoConfig):
--
To view, visit https://gerrit.osmocom.org/c/pysim/+/40199?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: pysim
Gerrit-Branch: master
Gerrit-Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2
Gerrit-Change-Number: 40199
Gerrit-PatchSet: 1
Gerrit-Owner: neels <nhofmeyr(a)sysmocom.de>
neels has uploaded this change for review. ( https://gerrit.osmocom.org/c/pysim/+/40198?usp=email )
Change subject: personalization: implement reading back values from a PES
......................................................................
personalization: implement reading back values from a PES
Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.
Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.
Implementation detail:
Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.
For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.
Rationales for reading back values:
This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).
Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.
Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463
---
M pySim/esim/saip/personalization.py
1 file changed, 121 insertions(+), 2 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/pysim refs/changes/98/40198/1
diff --git a/pySim/esim/saip/personalization.py b/pySim/esim/saip/personalization.py
index aaa4d8f..ca37e3a 100644
--- a/pySim/esim/saip/personalization.py
+++ b/pySim/esim/saip/personalization.py
@@ -18,13 +18,17 @@
import abc
import io
import copy
-from typing import List, Tuple, Generator
+from typing import List, Tuple, Generator, Optional
from osmocom.tlv import camel_to_snake
-from pySim.utils import enc_iccid, enc_imsi, h2b, rpad, sanitize_iccid, all_subclasses_of
+from osmocom.utils import hexstr
+from pySim.utils import enc_iccid, dec_iccid, enc_imsi, dec_imsi, h2b, b2h, rpad, sanitize_iccid, all_subclasses_of
from pySim.esim.saip import ProfileElement, ProfileElementSequence
from pySim.esim.saip import param_source
+def unrpad(s: hexstr, c='f') -> hexstr:
+ return hexstr(s.rstrip(c))
+
def remove_unwanted_tuples_from_list(l: List[Tuple], unwanted_keys: List[str]) -> List[Tuple]:
"""In a list of tuples, remove all tuples whose first part equals 'unwanted_key'."""
return list(filter(lambda x: x[0] not in unwanted_keys, l))
@@ -36,6 +40,22 @@
file.append(('fillFileContent', new_content))
return file
+def file_tuples_content_as_bytes(l: List[Tuple]) -> Optional[bytes]:
+ """linearize a list of fillFileContent / fillFileOffset tuples into a stream of bytes."""
+ stream = io.BytesIO()
+ for k, v in l:
+ if k == 'doNotCreate':
+ return None
+ if k == 'fileDescriptor':
+ pass
+ elif k == 'fillFileOffset':
+ stream.seek(v, os.SEEK_CUR)
+ elif k == 'fillFileContent':
+ stream.write(v)
+ else:
+ return ValueError("Unknown key '%s' in tuple list" % k)
+ return stream.getvalue()
+
class ConfigurableParameter:
r"""Base class representing a part of the eSIM profile that is configurable during the
personalization process (with dynamic data from elsewhere).
@@ -195,6 +215,30 @@
pass
@classmethod
+ def get_values_from_pes(cls, pes: ProfileElementSequence) -> Generator:
+ '''This is what subclasses implement: yield all values from a decoded profile package.
+ Find all values in the pes, and yield them decoded to a valid cls.input_value format.
+ Should be a generator function, i.e. use 'yield' instead of 'return'.
+
+ Usage example:
+
+ cls = esim.saip.personalization.Iccid
+ # use a set() to get a list of unique values from all results
+ vals = set( cls.get_values_from_pes(pes) )
+ if len(vals) != 1:
+ raise ValueError(f'{cls.name}: need exactly one value, got {vals}')
+ # the set contains a single value, return it
+ return vals.pop()
+
+ Implementation example:
+
+ for pe in pes:
+ if my_condition(pe):
+ yield b2h(my_bin_value_from(pe))
+ '''
+ pass
+
+ @classmethod
def get_len_range(cls):
"""considering all of min_len, max_len and allow_len, get a tuple of the resulting (min, max) of permitted
value length. For example, if an input value is an int, which needs to be represented with a minimum nr of
@@ -260,6 +304,17 @@
# a DecimalHexParam subclass expects the apply_val() input to be a bytes instance ready for the pes
return h2b(val)
+ @classmethod
+ def decimal_hex_to_str(cls, val):
+ 'useful for get_values_from_pes() implementations of subclasses'
+ if isinstance(val, bytes):
+ val = b2h(val)
+ assert isinstance(val, hexstr)
+ if cls.rpad is not None:
+ c = cls.rpad_char or 'f'
+ val = unrpad(val, c)
+ return val.to_bytes().decode('ascii')
+
class BinaryParam(ConfigurableParameter):
allow_types = (str, io.BytesIO, bytes, bytearray)
@@ -305,6 +360,17 @@
# patch MF/EF.ICCID
file_replace_content(pes.get_pe_for_type('mf').decoded['ef-iccid'], h2b(enc_iccid(val)))
+ @classmethod
+ def get_values_from_pes(cls, pes: ProfileElementSequence):
+ padded = b2h(pes.get_pe_for_type('header').decoded['iccid'])
+ iccid = unrpad(padded)
+ yield iccid
+
+ for pe in pes.get_pes_for_type('mf'):
+ iccid_pe = pe.decoded.get('ef-iccid', None)
+ if iccid_pe:
+ yield dec_iccid(b2h(file_tuples_content_as_bytes(iccid_pe)))
+
class Imsi(DecimalParam):
"""Configurable IMSI. Expects value to be a string of digits. Automatically sets the ACC to
the last digit of the IMSI."""
@@ -326,6 +392,13 @@
file_replace_content(pe.decoded['ef-acc'], acc.to_bytes(2, 'big'))
# TODO: DF.GSM_ACCESS if not linked?
+ @classmethod
+ def get_values_from_pes(cls, pes: ProfileElementSequence):
+ for pe in pes.get_pes_for_type('usim'):
+ imsi_pe = pe.decoded.get('ef-imsi', None)
+ if imsi_pe:
+ yield dec_imsi(b2h(file_tuples_content_as_bytes(imsi_pe)))
+
class SdKey(BinaryParam):
"""Configurable Security Domain (SD) Key. Value is presented as bytes."""
@@ -359,6 +432,14 @@
for pe in pes.get_pes_for_type('securityDomain'):
cls._apply_sd(pe, value)
+ @classmethod
+ def get_values_from_pes(cls, pes: ProfileElementSequence):
+ for pe in pes.get_pes_for_type('securityDomain'):
+ for key in pe.decoded['keyList']:
+ if key['keyIdentifier'][0] == cls.key_id and key['keyVersionNumber'][0] == cls.kvn:
+ if len(key['keyComponents']) >= 1:
+ yield b2h(key['keyComponents'][0]['keyData'])
+
class SdKeyScp80_01(SdKey):
kvn = 0x01
key_type = 0x88 # AES key type
@@ -495,6 +576,14 @@
raise ValueError("input template UPP has unexpected structure:"
f" cannot find pukCode with keyReference={cls.keyReference}")
+ @classmethod
+ def get_values_from_pes(cls, pes: ProfileElementSequence):
+ mf_pes = pes.pes_by_naa['mf'][0]
+ for pukCodes in obtain_all_pe_from_pelist(mf_pes, 'pukCodes'):
+ for pukCode in pukCodes.decoded['pukCodes']:
+ if pukCode['keyReference'] == cls.keyReference:
+ yield cls.decimal_hex_to_str(pukCode['pukValue'])
+
class Puk1(Puk):
is_abstract = False
name = 'PUK1'
@@ -532,6 +621,20 @@
raise ValueError('input template UPP has unexpected structure:'
+ f' {cls.get_name()} cannot find pinCode with keyReference={cls.keyReference}')
+ @classmethod
+ def _read_all_pinvalues_from_pe(cls, pe: ProfileElement):
+ for pinCodes in obtain_all_pe_from_pelist(pe, 'pinCodes'):
+ if pinCodes.decoded['pinCodes'][0] != 'pinconfig':
+ continue
+
+ for pinCode in pinCodes.decoded['pinCodes'][1]:
+ if pinCode['keyReference'] == cls.keyReference:
+ yield cls.decimal_hex_to_str(pinCode['pinValue'])
+
+ @classmethod
+ def get_values_from_pes(cls, pes: ProfileElementSequence):
+ yield from cls._read_all_pinvalues_from_pe(pes.pes_by_naa['mf'][0])
+
class Pin1(Pin):
is_abstract = False
name = 'PIN1'
@@ -555,6 +658,14 @@
raise ValueError('input template UPP has unexpected structure:'
+ f' {cls.get_name()} cannot find pinCode with keyReference={cls.keyReference} in {naa=}')
+ @classmethod
+ def get_values_from_pes(cls, pes: ProfileElementSequence):
+ for naa in pes.pes_by_naa:
+ if naa not in ['usim','isim','csim','telecom']:
+ continue
+ for pe in pes.pes_by_naa[naa]:
+ yield from cls._read_all_pinvalues_from_pe(pe)
+
class Adm1(Pin):
is_abstract = False
name = 'ADM1'
@@ -581,6 +692,14 @@
raise ValueError('input template UPP has unexpected structure:'
f' {cls.__name__} cannot find algoParameter with key={cls.algo_config_key}')
+ @classmethod
+ def get_values_from_pes(cls, pes: ProfileElementSequence):
+ for pe in pes.get_pes_for_type('akaParameter'):
+ algoConfiguration = pe.decoded['algoConfiguration']
+ if algoConfiguration[0] != 'algoParameter':
+ continue
+ yield algoConfiguration[1][cls.algo_config_key]
+
class AlgorithmID(DecimalParam, AlgoConfig):
is_abstract = False
--
To view, visit https://gerrit.osmocom.org/c/pysim/+/40198?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: pysim
Gerrit-Branch: master
Gerrit-Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463
Gerrit-Change-Number: 40198
Gerrit-PatchSet: 1
Gerrit-Owner: neels <nhofmeyr(a)sysmocom.de>
Attention is currently required from: dexter, fixeria.
Hello Jenkins Builder, dexter, laforge,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/c/pysim/+/39742?usp=email
to look at the new patch set (#10).
The following approvals got outdated and were removed:
Verified+1 by Jenkins Builder
The change is no longer submittable: Verified is unsatisfied now.
Change subject: [2/6] personalization: refactor ConfigurableParameter, Iccid, Imsi
......................................................................
[2/6] personalization: refactor ConfigurableParameter, Iccid, Imsi
Main points/rationales of the refactoring, details below:
1) common validation implementation
2) offer classmethods
The new features are optional, and will be heavily used by batch
personalization patches coming soon.
Implement Iccid and Imsi to use the new way, with a common abstract
DecimalParam implementation.
So far leave the other parameter classes working as they always did, to
follow suit in subsequent commits.
Details:
1) common validation implementation:
There are very common validation steps in the various parameter
implementations. It is more convenient and much more readable to
implement those once and set simple validation parameters per subclass.
So there now is a validate_val() classmethod, which subclasses can use
as-is to apply the validation parameters -- or subclasses can override
their cls.validate_val() for specialized validation.
(Those subclasses that this patch doesn't touch still override the
self.validate() instance method. Hence they still work as before this
patch, but don't use the new common features yet.)
2) offer stateless classmethods:
It is useful for...
- batch processing of multiple profiles (in upcoming patches) and
- user input validation
to be able to have classmethods that do what self.validate() and
self.apply() do, but do not modify any self.* members.
So far the paradigm was to create a class instance to keep state about
the value. This remains available, but in addition we make available the
paradigm of a singleton that is stateless (the classmethods).
Using self.validate() and self.apply() still work the same as before
this patch, i.e. via self.input_value and self.value -- but in addition,
there are now classmethods that don't touch self.* members.
Related: SYS#6768
Change-Id: I6522be4c463e34897ca9bff2309b3706a88b3ce8
---
M pySim/esim/saip/personalization.py
1 file changed, 179 insertions(+), 34 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/pysim refs/changes/42/39742/10
--
To view, visit https://gerrit.osmocom.org/c/pysim/+/39742?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: pysim
Gerrit-Branch: master
Gerrit-Change-Id: I6522be4c463e34897ca9bff2309b3706a88b3ce8
Gerrit-Change-Number: 39742
Gerrit-PatchSet: 10
Gerrit-Owner: neels <nhofmeyr(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: dexter <pmaier(a)sysmocom.de>
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Gerrit-Reviewer: neels <nhofmeyr(a)sysmocom.de>
Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Attention: fixeria <vyanitskiy(a)sysmocom.de>
Gerrit-Attention: dexter <pmaier(a)sysmocom.de>
neels has posted comments on this change by neels. ( https://gerrit.osmocom.org/c/pysim/+/39743?usp=email )
Change subject: [3/6] personalization: refactor Puk
......................................................................
Patch Set 6:
(1 comment)
Patchset:
PS6:
so far one +1 .. i would like to place a "trivial" +2 on this patch soon, because it naturally follows earlier patches
--
To view, visit https://gerrit.osmocom.org/c/pysim/+/39743?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email
Gerrit-MessageType: comment
Gerrit-Project: pysim
Gerrit-Branch: master
Gerrit-Change-Id: I271e6c030c890778ab7af9ab3bc7997e22018f6a
Gerrit-Change-Number: 39743
Gerrit-PatchSet: 6
Gerrit-Owner: neels <nhofmeyr(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Gerrit-Comment-Date: Mon, 05 May 2025 00:01:26 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
