<p>laforge <strong>submitted</strong> this change.</p><p><a href="https://gerrit.osmocom.org/c/pysim/+/24452">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Jenkins Builder: Verified
laforge: Looks good to me, approved
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">construct: Recursive normalization of construct parse result<br><br>If we want to use construct parse results to generate JSON serializable<br>dicts, we need to<br><br>* apply the filter_dict() operation recursively, and<br>* simplify the construct Container and ListContainer classes to<br> a simple dict and/or list.<br><br>We introduce a pySim.construct.parse_construct() helper which is<br>subsequently used from all pySim.filesystem caller sites.<br><br>Change-Id: I319414eb69808ef65895293832bb30519f45949d<br>---<br>M pySim/construct.py<br>M pySim/filesystem.py<br>2 files changed, 36 insertions(+), 7 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/pySim/construct.py b/pySim/construct.py</span><br><span>index d0101e3..2a3efd3 100644</span><br><span>--- a/pySim/construct.py</span><br><span>+++ b/pySim/construct.py</span><br><span>@@ -1,3 +1,4 @@</span><br><span style="color: hsl(120, 100%, 40%);">+import typing</span><br><span> from construct import *</span><br><span> from pySim.utils import b2h, h2b, swap_nibbles</span><br><span> import gsm0338</span><br><span>@@ -84,6 +85,34 @@</span><br><span> res[key] = value</span><br><span> return res</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+from construct.lib.containers import Container, ListContainer</span><br><span style="color: hsl(120, 100%, 40%);">+from construct.core import EnumIntegerString</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def normalize_construct(c):</span><br><span style="color: hsl(120, 100%, 40%);">+ """Convert a construct specific type to a related base type, mostly useful</span><br><span style="color: hsl(120, 100%, 40%);">+ so we can serialize it."""</span><br><span style="color: hsl(120, 100%, 40%);">+ # we need to include the filter_dict as we otherwise get elements like this</span><br><span style="color: hsl(120, 100%, 40%);">+ # in the dict: '_io': <_io.BytesIO object at 0x7fdb64e05860> which we cannot json-serialize</span><br><span style="color: hsl(120, 100%, 40%);">+ c = filter_dict(c)</span><br><span style="color: hsl(120, 100%, 40%);">+ if isinstance(c, Container) or isinstance(c, dict):</span><br><span style="color: hsl(120, 100%, 40%);">+ r = {k : normalize_construct(v) for (k, v) in c.items()}</span><br><span style="color: hsl(120, 100%, 40%);">+ elif isinstance(c, ListContainer):</span><br><span style="color: hsl(120, 100%, 40%);">+ r = [normalize_construct(x) for x in c]</span><br><span style="color: hsl(120, 100%, 40%);">+ elif isinstance(c, list):</span><br><span style="color: hsl(120, 100%, 40%);">+ r = [normalize_construct(x) for x in c]</span><br><span style="color: hsl(120, 100%, 40%);">+ elif isinstance(c, EnumIntegerString):</span><br><span style="color: hsl(120, 100%, 40%);">+ r = str(c)</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ r = c</span><br><span style="color: hsl(120, 100%, 40%);">+ return r</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def parse_construct(c, raw_bin_data:bytes, length:typing.Optional[int]=None, exclude_prefix:str='_'):</span><br><span style="color: hsl(120, 100%, 40%);">+ """Helper function to wrap around normalize_construct() and filter_dict()."""</span><br><span style="color: hsl(120, 100%, 40%);">+ if not length:</span><br><span style="color: hsl(120, 100%, 40%);">+ length = len(raw_bin_data)</span><br><span style="color: hsl(120, 100%, 40%);">+ parsed = c.parse(raw_bin_data, total_len=length)</span><br><span style="color: hsl(120, 100%, 40%);">+ return normalize_construct(parsed)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> # here we collect some shared / common definitions of data types</span><br><span> LV = Prefixed(Int8ub, HexAdapter(GreedyBytes))</span><br><span> </span><br><span>diff --git a/pySim/filesystem.py b/pySim/filesystem.py</span><br><span>index 8c45ba1..9a2f4e1 100644</span><br><span>--- a/pySim/filesystem.py</span><br><span>+++ b/pySim/filesystem.py</span><br><span>@@ -35,7 +35,7 @@</span><br><span> from typing import cast, Optional, Iterable, List, Any, Dict, Tuple</span><br><span> </span><br><span> from pySim.utils import sw_match, h2b, b2h, i2h, is_hex, auto_int, bertlv_parse_one, Hexstr</span><br><span style="color: hsl(0, 100%, 40%);">-from pySim.construct import filter_dict</span><br><span style="color: hsl(120, 100%, 40%);">+from pySim.construct import filter_dict, parse_construct</span><br><span> from pySim.exceptions import *</span><br><span> from pySim.jsonpath import js_path_find, js_path_modify</span><br><span> </span><br><span>@@ -490,7 +490,7 @@</span><br><span> if callable(method):</span><br><span> return method(b2h(raw_bin_data))</span><br><span> if self._construct:</span><br><span style="color: hsl(0, 100%, 40%);">- return filter_dict(self._construct.parse(raw_bin_data, total_len=len(raw_bin_data)))</span><br><span style="color: hsl(120, 100%, 40%);">+ return parse_construct(self._construct, raw_bin_data)</span><br><span> return {'raw': raw_bin_data.hex()}</span><br><span> </span><br><span> def decode_hex(self, raw_hex_data:str) -> dict:</span><br><span>@@ -513,7 +513,7 @@</span><br><span> if callable(method):</span><br><span> return method(raw_bin_data)</span><br><span> if self._construct:</span><br><span style="color: hsl(0, 100%, 40%);">- return filter_dict(self._construct.parse(raw_bin_data, total_len=len(raw_bin_data)))</span><br><span style="color: hsl(120, 100%, 40%);">+ return parse_construct(self._construct, raw_bin_data)</span><br><span> return {'raw': raw_bin_data.hex()}</span><br><span> </span><br><span> def encode_bin(self, abstract_data:dict) -> bytearray:</span><br><span>@@ -712,7 +712,7 @@</span><br><span> if callable(method):</span><br><span> return method(raw_bin_data)</span><br><span> if self._construct:</span><br><span style="color: hsl(0, 100%, 40%);">- return filter_dict(self._construct.parse(raw_bin_data, total_len=len(raw_bin_data)))</span><br><span style="color: hsl(120, 100%, 40%);">+ return parse_construct(self._construct, raw_bin_data)</span><br><span> return {'raw': raw_bin_data.hex()}</span><br><span> </span><br><span> def decode_record_bin(self, raw_bin_data:bytearray) -> dict:</span><br><span>@@ -735,7 +735,7 @@</span><br><span> if callable(method):</span><br><span> return method(raw_hex_data)</span><br><span> if self._construct:</span><br><span style="color: hsl(0, 100%, 40%);">- return filter_dict(self._construct.parse(raw_bin_data, total_len=len(raw_bin_data)))</span><br><span style="color: hsl(120, 100%, 40%);">+ return parse_construct(self._construct, raw_bin_data)</span><br><span> return {'raw': raw_hex_data}</span><br><span> </span><br><span> def encode_record_hex(self, abstract_data:dict) -> str:</span><br><span>@@ -834,7 +834,7 @@</span><br><span> if callable(method):</span><br><span> return method(raw_bin_data)</span><br><span> if self._construct:</span><br><span style="color: hsl(0, 100%, 40%);">- return filter_dict(self._construct.parse(raw_bin_data, total_len=len(raw_bin_data)))</span><br><span style="color: hsl(120, 100%, 40%);">+ return parse_construct(self._construct, raw_bin_data)</span><br><span> return {'raw': raw_hex_data}</span><br><span> </span><br><span> def decode_record_bin(self, raw_bin_data:bytearray) -> dict:</span><br><span>@@ -857,7 +857,7 @@</span><br><span> if callable(method):</span><br><span> return method(raw_hex_data)</span><br><span> if self._construct:</span><br><span style="color: hsl(0, 100%, 40%);">- return filter_dict(self._construct.parse(raw_bin_data, total_len=len(raw_bin_data)))</span><br><span style="color: hsl(120, 100%, 40%);">+ return parse_construct(self._construct, raw_bin_data)</span><br><span> return {'raw': raw_hex_data}</span><br><span> </span><br><span> def encode_record_hex(self, abstract_data:dict) -> str:</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/pysim/+/24452">change 24452</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/pysim/+/24452"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: pysim </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I319414eb69808ef65895293832bb30519f45949d </div>
<div style="display:none"> Gerrit-Change-Number: 24452 </div>
<div style="display:none"> Gerrit-PatchSet: 5 </div>
<div style="display:none"> Gerrit-Owner: laforge <laforge@osmocom.org> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder </div>
<div style="display:none"> Gerrit-Reviewer: dexter <pmaier@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: fixeria <vyanitskiy@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: laforge <laforge@osmocom.org> </div>
<div style="display:none"> Gerrit-Reviewer: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>