[M] Change in pysim[master]: personalization: implement reading back values from a PES - gerrit-log

27 Apr 2026

laforge has submitted this change. ( https://gerrit.osmocom.org/c/pysim/+/40198?usp=email )
Change subject: personalization: implement reading back values from a PES
......................................................................
personalization: implement reading back values from a PES
Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.
Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.
Implementation detail:
Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.
For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.
Rationales for reading back values:
This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).
Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.
Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463
Jenkins: skip-card-test
---
M pySim/esim/saip/personalization.py
1 file changed, 179 insertions(+), 7 deletions(-)
Approvals:
  dexter: Looks good to me, approved
  Jenkins Builder: Verified

diff --git a/pySim/esim/saip/personalization.py b/pySim/esim/saip/personalization.py
index fb91cb5..2b99632 100644
--- a/pySim/esim/saip/personalization.py
+++ b/pySim/esim/saip/personalization.py
@@ -17,13 +17,17 @@
import abc
 import io
-from typing import List, Tuple
+from typing import List, Tuple, Generator, Optional
from osmocom.tlv import camel_to_snake
-from pySim.utils import enc_iccid, enc_imsi, h2b, rpad, sanitize_iccid
+from osmocom.utils import hexstr
+from pySim.utils import enc_iccid, dec_iccid, enc_imsi, dec_imsi, h2b, b2h, rpad, sanitize_iccid
 from pySim.esim.saip import ProfileElement, ProfileElementSequence
 from pySim.ts_51_011 import EF_SMSP
+def unrpad(s: hexstr, c='f') -> hexstr:
+    return hexstr(s.rstrip(c))
+
 def remove_unwanted_tuples_from_list(l: List[Tuple], unwanted_keys: List[str]) -> List[Tuple]:
     """In a list of tuples, remove all tuples whose first part equals 'unwanted_key'."""
     return list(filter(lambda x: x[0] not in unwanted_keys, l))
@@ -200,6 +204,29 @@
         pass
@classmethod
+    @abc.abstractmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence) -> Generator:
+        """This is what subclasses implement: yield all values from a decoded profile package.
+           Find all values in the pes, and yield them decoded to a valid cls.input_value format.
+           Should be a generator function, i.e. use 'yield' instead of 'return'.
+
+           Yielded value must be a dict(). Usually, an implementation will return only one key, like
+
+              { "ICCID": "1234567890123456789" }
+
+           Some implementations have more than one value to return, like
+
+              { "IMSI": "00101012345678", "IMSI-ACC" : "5" }
+
+           Implementation example:
+
+             for pe in pes:
+                if my_condition(pe):
+                    yield { cls.name: b2h(my_bin_value_from(pe)) }
+           """
+        pass
+
+    @classmethod
     def get_len_range(cls):
         """considering all of min_len, max_len and allow_len, get a tuple of the resulting (min, max) of permitted
         value length. For example, if an input value is an int, which needs to be represented with a minimum nr of
@@ -249,6 +276,7 @@
     @classmethod
     def validate_val(cls, val):
         val = super().validate_val(val)
+        assert isinstance(val, str)
         val = ''.join('%02x' % ord(x) for x in val)
         if cls.rpad is not None:
             c = cls.rpad_char
@@ -256,6 +284,17 @@
         # a DecimalHexParam subclass expects the apply_val() input to be a bytes instance ready for the pes
         return h2b(val)
+    @classmethod
+    def decimal_hex_to_str(cls, val):
+        """useful for get_values_from_pes() implementations of subclasses"""
+        if isinstance(val, bytes):
+            val = b2h(val)
+        assert isinstance(val, hexstr)
+        if cls.rpad is not None:
+            c = cls.rpad_char or 'f'
+            val = unrpad(val, c)
+        return val.to_bytes().decode('ascii')
+
 class IntegerParam(ConfigurableParameter):
     allow_types = (str, int)
     allow_chars = '0123456789'
@@ -279,6 +318,14 @@
             raise ValueError(f'Value {val} is out of range, must be [{cls.min_val}..{cls.max_val}]')
         return val
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        for valdict in super().get_values_from_pes(pes):
+            for key, val in valdict.items():
+                if isinstance(val, int):
+                    valdict[key] = str(val)
+            yield valdict
+
 class BinaryParam(ConfigurableParameter):
     allow_types = (str, io.BytesIO, bytes, bytearray)
     allow_chars = '0123456789abcdefABCDEF'
@@ -322,6 +369,17 @@
         # patch MF/EF.ICCID
         file_replace_content(pes.get_pe_for_type('mf').decoded['ef-iccid'], h2b(enc_iccid(val)))
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        padded = b2h(pes.get_pe_for_type('header').decoded['iccid'])
+        iccid = unrpad(padded)
+        yield { cls.name: iccid }
+
+        for pe in pes.get_pes_for_type('mf'):
+            iccid_f = pe.files.get('ef-iccid', None)
+            if iccid_f is not None:
+                yield { cls.name: dec_iccid(b2h(iccid_f.body)) }
+
 class Imsi(DecimalParam):
     """Configurable IMSI. Expects value to be a string of digits. Automatically sets the ACC to
     the last digit of the IMSI."""
@@ -342,6 +400,18 @@
             file_replace_content(pe.decoded['ef-acc'], acc.to_bytes(2, 'big'))
         # TODO: DF.GSM_ACCESS if not linked?
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        for pe in pes.get_pes_for_type('usim'):
+            imsi_f = pe.files.get('ef-imsi', None)
+            acc_f = pe.files.get('ef-acc', None)
+            y = {}
+            if imsi_f:
+                y[cls.name] = dec_imsi(b2h(imsi_f.body))
+            if acc_f:
+                y[cls.name + '-ACC'] = b2h(acc_f.body)
+            yield y
+
 class SmspTpScAddr(ConfigurableParameter):
     """Configurable SMSC (SMS Service Centre) TP-SC-ADDR. Expects to be a phone number in national or
     international format (designated by a leading +). Automatically sets the NPI to E.164 and the TON based on
@@ -354,21 +424,39 @@
     min_len = 1
     example_input = '+49301234567'
-    @classmethod
-    def validate_val(cls, val):
-        val = super().validate_val(val)
-        addr_str = str(val)
+    @staticmethod
+    def str_to_tuple(addr_str):
         if addr_str[0] == '+':
             digits = addr_str[1:]
             international = True
         else:
             digits = addr_str
             international = False
+        return (international, digits)
+
+    @staticmethod
+    def tuple_to_str(addr_tuple):
+        international, digits = addr_tuple
+        if international:
+            ret = '+'
+        else:
+            ret = ''
+        ret += digits
+        return ret
+
+    @classmethod
+    def validate_val(cls, val):
+        val = super().validate_val(val)
+
+        addr_tuple = cls.str_to_tuple(str(val))
+
+        international, digits = addr_tuple
         if len(digits) > 20:
             raise ValueError(f'TP-SC-ADDR must not exceed 20 digits: {digits!r}')
         if not digits.isdecimal():
             raise ValueError(f'TP-SC-ADDR must only contain decimal digits: {digits!r}')
-        return (international, digits)
+
+        return addr_tuple
@classmethod
     def apply_val(cls, pes: ProfileElementSequence, val):
@@ -398,6 +486,32 @@
             # re-generate the pe.decoded member from the File instance
             pe.file2pe(f_smsp)
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        for pe in pes.get_pes_for_type('usim'):
+            f_smsp = pe.files['ef-smsp']
+            ef_smsp = EF_SMSP()
+            ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1)
+
+            tp_sc_addr = ef_smsp_dec.get('tp_sc_addr', None)
+            if not tp_sc_addr:
+                continue
+
+            digits = tp_sc_addr.get('call_number', None)
+            if not digits:
+                continue
+
+            ton_npi = tp_sc_addr.get('ton_npi', None)
+            if not ton_npi:
+                continue
+            international = ton_npi.get('type_of_number', None)
+            if international is None:
+                continue
+            international = (international == 'international')
+
+            yield { cls.name: cls.tuple_to_str((international, digits)) }
+
+
 class SdKey(BinaryParam, metaclass=ClassVarMeta):
     """Configurable Security Domain (SD) Key.  Value is presented as bytes."""
     # these will be set by subclasses
@@ -430,6 +544,14 @@
         for pe in pes.get_pes_for_type('securityDomain'):
             cls._apply_sd(pe, value)
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        for pe in pes.get_pes_for_type('securityDomain'):
+            for key in pe.decoded['keyList']:
+                if key['keyIdentifier'][0] == cls.key_id and key['keyVersionNumber'][0] == cls.kvn:
+                    if len(key['keyComponents']) >= 1:
+                        yield { cls.name: b2h(key['keyComponents'][0]['keyData']) }
+
 class SdKeyScp80_01(SdKey, kvn=0x01, key_type=0x88, permitted_len=[16,24,32]): # AES key type
     pass
 class SdKeyScp80_01Kic(SdKeyScp80_01, key_id=0x01, key_usage_qual=0x18): # FIXME: ordering?
@@ -516,6 +638,14 @@
         raise ValueError("input template UPP has unexpected structure:"
                          f" cannot find pukCode with keyReference={cls.keyReference}")
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        mf_pes = pes.pes_by_naa['mf'][0]
+        for pukCodes in obtain_all_pe_from_pelist(mf_pes, 'pukCodes'):
+            for pukCode in pukCodes.decoded['pukCodes']:
+                if pukCode['keyReference'] == cls.keyReference:
+                    yield { cls.name: cls.decimal_hex_to_str(pukCode['pukValue']) }
+
 class Puk1(Puk):
     name = 'PUK1'
     keyReference = 0x01
@@ -551,6 +681,21 @@
             raise ValueError('input template UPP has unexpected structure:'
                              + f' {cls.get_name()} cannot find pinCode with keyReference={cls.keyReference}')
+    @classmethod
+    def _read_all_pinvalues_from_pe(cls, pe: ProfileElement):
+        "This is a separate function because subclasses may feed different pe arguments."
+        for pinCodes in obtain_all_pe_from_pelist(pe, 'pinCodes'):
+            if pinCodes.decoded['pinCodes'][0] != 'pinconfig':
+                continue
+
+            for pinCode in pinCodes.decoded['pinCodes'][1]:
+                if pinCode['keyReference'] == cls.keyReference:
+                    yield { cls.name: cls.decimal_hex_to_str(pinCode['pinValue']) }
+
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        yield from cls._read_all_pinvalues_from_pe(pes.pes_by_naa['mf'][0])
+
 class Pin1(Pin):
     name = 'PIN1'
     example_input = '0' * 4  # PIN are usually 4 digits
@@ -572,6 +717,14 @@
                     raise ValueError('input template UPP has unexpected structure:'
                             + f' {cls.get_name()} cannot find pinCode with keyReference={cls.keyReference} in {naa=}')
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        for naa in pes.pes_by_naa:
+            if naa not in ['usim','isim','csim','telecom']:
+                continue
+            for pe in pes.pes_by_naa[naa]:
+                yield from cls._read_all_pinvalues_from_pe(pe)
+
 class Adm1(Pin):
     name = 'ADM1'
     keyReference = 0x0A
@@ -596,6 +749,25 @@
             raise ValueError('input template UPP has unexpected structure:'
                              f' {cls.__name__} cannot find algoParameter with key={cls.algo_config_key}')
+    @classmethod
+    def get_values_from_pes(cls, pes: ProfileElementSequence):
+        for pe in pes.get_pes_for_type('akaParameter'):
+            algoConfiguration = pe.decoded['algoConfiguration']
+            if len(algoConfiguration) < 2:
+                continue
+            if algoConfiguration[0] != 'algoParameter':
+                continue
+            if not algoConfiguration[1]:
+                continue
+            val = algoConfiguration[1].get(cls.algo_config_key, None)
+            if val is None:
+                continue
+            if isinstance(val, bytes):
+                val = b2h(val)
+            # if it is an int (algorithmID), just pass thru as int
+            yield { cls.name: val }
+
+
 class AlgorithmID(DecimalParam, AlgoConfig):
     algo_config_key = 'algorithmID'
     allow_len = 1
-- 
To view, visit https://gerrit.osmocom.org/c/pysim/+/40198?usp=email
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email

Gerrit-MessageType: merged
Gerrit-Project: pysim
Gerrit-Branch: master
Gerrit-Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463
Gerrit-Change-Number: 40198
Gerrit-PatchSet: 16
Gerrit-Owner: neels nhofmeyr@sysmocom.de
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: dexter pmaier@sysmocom.de
Gerrit-Reviewer: laforge laforge@osmocom.org
Gerrit-Reviewer: neels nhofmeyr@sysmocom.de
Gerrit-CC: fixeria vyanitskiy@sysmocom.de



    