dexter has uploaded this change for review.

View Change

tlv: improve flatten_dict_lists for multiple keys

when we flatten a list of dicts into a single dict we must make sure
that duplicate key names are not swallowed. This may happen when there
is an array of objects that share the same identifier. When those are
stored all following objects are stored at the same key so that only the
content of the last one survives.

One prominent example where this bug is triggered is the key_reference
in pin_status_template_do. When the MF is selected, there should be 4
key_reference values, but only the last appears.

Change I7f6d03bf323a153f3172853a3ef171cbec8aece7 fixes the problem, but
now the problematic dicts are no longer flattned, which results in a
strange looking output. When append an index to the key, we can still
flatten the dicts and the output looks compact again.

Related: OS#6211
Change-Id: Ie3b418ca3965b0221bbf5f85d7d0e0fbb39d9d87
---
M pySim/tlv.py
1 file changed, 46 insertions(+), 5 deletions(-)

git pull ssh://gerrit.osmocom.org:29418/pysim refs/changes/54/35254/1

diff --git a/pySim/tlv.py b/pySim/tlv.py
index c85d92b..d3c75e2 100644
--- a/pySim/tlv.py
+++ b/pySim/tlv.py
@@ -435,17 +435,33 @@
                 return False
         return True
 
-    def are_elements_unique(lod):
-        set_of_keys = set([list(x.keys())[0] for x in lod])
-        return len(lod) == len(set_of_keys)
+    def add_element_to_newdict(newdict:dict, key:str, element) -> str:
+        newkey = key
+        count = 1
+        while 1:
+            # Test if the new key (which is equal to key at the beginning) is
+            # already in the newdict. If this is a case we append an index
+            # number and try again.
+            if newkey not in newdict and newkey + "_0" not in newdict:
+                # Rename the already existing zeroth element to element_0, so
+                # that we maintain a consistent element counting.
+                if key in newdict and newkey != key:
+                    newdict[key + "_0"] = newdict.pop(key)
+                # Store new element under the new key and leave
+                newdict[newkey] = element
+                return
+
+            # Compute the next key to try
+            newkey = key + "_" + str(count)
+            count = count + 1
 
     if isinstance(inp, list):
-        if are_all_elements_dict(inp) and are_elements_unique(inp):
+        if are_all_elements_dict(inp):
             # flatten into one shared dict
             newdict = {}
             for e in inp:
                 key = list(e.keys())[0]
-                newdict[key] = e[key]
+                add_element_to_newdict(newdict, key, e[key])
             inp = newdict
             # process result as any native dict
             return {k:flatten_dict_lists(inp[k]) for k in inp.keys()}

To view, visit change 35254. To unsubscribe, or for help writing mail filters, visit settings.