<p>Neels Hofmeyr <strong>merged</strong> this change.</p><p><a href="https://gerrit.osmocom.org/11786">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Pau Espin Pedrol: Looks good to me, approved
Jenkins Builder: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">add contrib/struct_endianess.py<br><br>In libosmocore (and likely elsewhere) we have scores of packed structs with<br>sub-byte integer members that lack the necessary member reversal shims to be<br>able to work on big endian architectures.<br><br>Instead of manually editing each one of them and probably introduce errors in<br>the process, this script handles the change automatically, and in the future<br>allows us to verify correctness in gerrit verifications.<br><br>Change-Id: I8e75b17d8071c7b3a2a171ba776fb76854b28a53<br>---<br>A contrib/struct_endianess.py<br>1 file changed, 369 insertions(+), 0 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/contrib/struct_endianess.py b/contrib/struct_endianess.py</span><br><span>new file mode 100755</span><br><span>index 0000000..be73fbe</span><br><span>--- /dev/null</span><br><span>+++ b/contrib/struct_endianess.py</span><br><span>@@ -0,0 +1,369 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/usr/bin/env python3</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+'''Using mad regexes, automatically make sure that all structs with sub-byte</span><br><span style="color: hsl(120, 100%, 40%);">+integers have matching big-endian definitions. The idea is to save a lot of</span><br><span style="color: hsl(120, 100%, 40%);">+manual effort, and to automatically verify that there are no errors.</span><br><span style="color: hsl(120, 100%, 40%);">+This script most certainly has numerous holes and shortcomings, but actually,</span><br><span style="color: hsl(120, 100%, 40%);">+if you hit problems with it, rather adjust your coding style so that this</span><br><span style="color: hsl(120, 100%, 40%);">+script can deal with it...'''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+import re</span><br><span style="color: hsl(120, 100%, 40%);">+import sys</span><br><span style="color: hsl(120, 100%, 40%);">+import codecs</span><br><span style="color: hsl(120, 100%, 40%);">+import os.path</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+re_struct_start = re.compile(r'^struct\s*[a-zA-Z_][a-zA-Z_0-9]*\s*{\s*$')</span><br><span style="color: hsl(120, 100%, 40%);">+re_struct_end = re.compile(r'^}[^;]*;\s*$')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+re_substruct_start = re.compile(r'^\s+struct\s*{\s*$')</span><br><span style="color: hsl(120, 100%, 40%);">+re_substruct_end = re.compile(r'^\s+}\s*([^;]*\s)[a-zA-Z_][a-zA-Z_0-9]*\s*;\s*$')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+re_int_def = re.compile(r'(^\s*((const|unsigned|signed|char|int|long|int[0-9]+_t|uint[0-9]_t)\s+)+\s*)([^;]*;)',</span><br><span style="color: hsl(120, 100%, 40%);">+ re.DOTALL | re.MULTILINE)</span><br><span style="color: hsl(120, 100%, 40%);">+re_int_members = re.compile(r'([a-zA-Z_][a-zA-Z_0-9]*|[a-zA-Z_][a-zA-Z_0-9]*\s*:\s*[0-9]+)\s*[,;]\s*', re.DOTALL | re.MULTILINE)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+re_little_endian_ifdef = re.compile(r'#\s*(if|elif)\s+OSMO_IS_LITTLE_ENDIAN\s*(==\s*1\s*|)');</span><br><span style="color: hsl(120, 100%, 40%);">+re_big_endian_ifdef = re.compile(r'#\s*(if|elif)\s+OSMO_IS_BIG_ENDIAN\s*');</span><br><span style="color: hsl(120, 100%, 40%);">+re_else = re.compile(r'#\s*else\s*');</span><br><span style="color: hsl(120, 100%, 40%);">+re_endif = re.compile(r'#\s*endif\s*');</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+re_c_comment = re.compile(r'(/\*[^*]+\*/|//.?$)')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def remove_c_comments(code_str):</span><br><span style="color: hsl(120, 100%, 40%);">+ return ''.join(re_c_comment.split(code_str)[::2])</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def section_struct_body(struct_body_lines):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''divide a top-level-struct body into sections of</span><br><span style="color: hsl(120, 100%, 40%);">+ ['arbitrary string', ['body;\n', 'lines;\n'], 'arbitrary string', ...]</span><br><span style="color: hsl(120, 100%, 40%);">+ Aim: handle each sub-struct on its own, and if there already are ifdefs for</span><br><span style="color: hsl(120, 100%, 40%);">+ little and big endian, keep just the little endian bit and derive big</span><br><span style="color: hsl(120, 100%, 40%);">+ endian from it.</span><br><span style="color: hsl(120, 100%, 40%);">+ An arbitrary string is anything other than struct member definitions, like</span><br><span style="color: hsl(120, 100%, 40%);">+ a 'struct {', '} sub_name;', ...</span><br><span style="color: hsl(120, 100%, 40%);">+ "body lines" are lines that define struct members (possibly with comments).</span><br><span style="color: hsl(120, 100%, 40%);">+ Return: list of alternate arbitrary strings and variable definitions.</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # these globals are needed so that end_def() can change them from inside</span><br><span style="color: hsl(120, 100%, 40%);">+ # the function. Not very nice style, but easiest implementation.</span><br><span style="color: hsl(120, 100%, 40%);">+ global struct_body_parts</span><br><span style="color: hsl(120, 100%, 40%);">+ global arbitrary_part</span><br><span style="color: hsl(120, 100%, 40%);">+ global def_part</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ struct_body_parts = []</span><br><span style="color: hsl(120, 100%, 40%);">+ arbitrary_part = []</span><br><span style="color: hsl(120, 100%, 40%);">+ def_part = []</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def end_def():</span><br><span style="color: hsl(120, 100%, 40%);">+ '''if there is any content, flush out recorded parts (def_part,</span><br><span style="color: hsl(120, 100%, 40%);">+ arbitrary_part) and start a new part. In short, cut a section</span><br><span style="color: hsl(120, 100%, 40%);">+ boundary.'''</span><br><span style="color: hsl(120, 100%, 40%);">+ global struct_body_parts</span><br><span style="color: hsl(120, 100%, 40%);">+ global arbitrary_part</span><br><span style="color: hsl(120, 100%, 40%);">+ global def_part</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if def_part:</span><br><span style="color: hsl(120, 100%, 40%);">+ struct_body_parts.append(arbitrary_part)</span><br><span style="color: hsl(120, 100%, 40%);">+ arbitrary_part = []</span><br><span style="color: hsl(120, 100%, 40%);">+ struct_body_parts.append(def_part)</span><br><span style="color: hsl(120, 100%, 40%);">+ def_part = []</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ j = 0</span><br><span style="color: hsl(120, 100%, 40%);">+ while j < len(struct_body_lines):</span><br><span style="color: hsl(120, 100%, 40%);">+ line = struct_body_lines[j]</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (re_substruct_start.fullmatch(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ or re_substruct_end.fullmatch(line)):</span><br><span style="color: hsl(120, 100%, 40%);">+ end_def()</span><br><span style="color: hsl(120, 100%, 40%);">+ arbitrary_part.append(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ j += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if re_big_endian_ifdef.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ end_def()</span><br><span style="color: hsl(120, 100%, 40%);">+ # discard big endian section</span><br><span style="color: hsl(120, 100%, 40%);">+ j += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ while j < len(struct_body_lines):</span><br><span style="color: hsl(120, 100%, 40%);">+ line = struct_body_lines[j]</span><br><span style="color: hsl(120, 100%, 40%);">+ if re_endif.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ end_def()</span><br><span style="color: hsl(120, 100%, 40%);">+ j += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ break;</span><br><span style="color: hsl(120, 100%, 40%);">+ if re_little_endian_ifdef.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ end_def()</span><br><span style="color: hsl(120, 100%, 40%);">+ # keep that start of little endian section, not j++</span><br><span style="color: hsl(120, 100%, 40%);">+ break;</span><br><span style="color: hsl(120, 100%, 40%);">+ if re_else.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ # there's an '#else' after big-endian. Shim a little-endian header in just for the loop.</span><br><span style="color: hsl(120, 100%, 40%);">+ struct_body_lines[j] = '#if OSMO_IS_LITTLE_ENDIAN\n'</span><br><span style="color: hsl(120, 100%, 40%);">+ break;</span><br><span style="color: hsl(120, 100%, 40%);">+ j += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if re_little_endian_ifdef.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ end_def()</span><br><span style="color: hsl(120, 100%, 40%);">+ j += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ while j < len(struct_body_lines):</span><br><span style="color: hsl(120, 100%, 40%);">+ line = struct_body_lines[j]</span><br><span style="color: hsl(120, 100%, 40%);">+ if re_endif.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ end_def()</span><br><span style="color: hsl(120, 100%, 40%);">+ j += 1</span><br><span style="color: hsl(120, 100%, 40%);">+ break;</span><br><span style="color: hsl(120, 100%, 40%);">+ if re_big_endian_ifdef.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ end_def()</span><br><span style="color: hsl(120, 100%, 40%);">+ # keep that start of big endian section, not j++</span><br><span style="color: hsl(120, 100%, 40%);">+ break;</span><br><span style="color: hsl(120, 100%, 40%);">+ if re_else.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ # there's an '#else' after little-endian. Shim a big-endian header in just for the loop.</span><br><span style="color: hsl(120, 100%, 40%);">+ struct_body_lines[j] = '#if OSMO_IS_BIG_ENDIAN\n'</span><br><span style="color: hsl(120, 100%, 40%);">+ break;</span><br><span style="color: hsl(120, 100%, 40%);">+ def_part.append(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ j += 1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def_part.append(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ j += 1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # flush the last section remaining that didn't see an explicit end</span><br><span style="color: hsl(120, 100%, 40%);">+ end_def()</span><br><span style="color: hsl(120, 100%, 40%);">+ # end_def() only flushes arbitrary_part if there was a def_part, so:</span><br><span style="color: hsl(120, 100%, 40%);">+ if arbitrary_part:</span><br><span style="color: hsl(120, 100%, 40%);">+ struct_body_parts.append(arbitrary_part)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return struct_body_parts</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def struct_body_to_big_endian(body_str):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''Input: a multi-line string containing the body of a struct, i.e. without</span><br><span style="color: hsl(120, 100%, 40%);">+ sub-structs and without #if OSMO_IS_BIG_ENDIAN. like</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ '\tconst char *foo;\n\tuint8_t moo:3, goo:2;\n\tuint8_t loo:3;\n\tvoid *baz;\n'</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ Return None to indicate that there is no little/big endian split</span><br><span style="color: hsl(120, 100%, 40%);">+ required, or return a multi-line string of the big-endian version of this</span><br><span style="color: hsl(120, 100%, 40%);">+ same struct body, where sub-byte ints are reversed at byte boundaries, and</span><br><span style="color: hsl(120, 100%, 40%);">+ all others are copied 1:1. If there are no sub-byte integers, return None,</span><br><span style="color: hsl(120, 100%, 40%);">+ to indicate that there is no little/big endian split required.'''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # kick comments out of the code analysis. They will end up being stripped</span><br><span style="color: hsl(120, 100%, 40%);">+ # from big-endian only.</span><br><span style="color: hsl(120, 100%, 40%);">+ body_str = remove_c_comments(body_str)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def_strs = body_str.split(';')</span><br><span style="color: hsl(120, 100%, 40%);">+ def_strs = ('%s;' % def_str for def_str in def_strs if def_str.strip())</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # classify defs as containing sub-byte members or not</span><br><span style="color: hsl(120, 100%, 40%);">+ # defs = [ (true, 'uint8_t ', ('foo:3', 'bar:5')),</span><br><span style="color: hsl(120, 100%, 40%);">+ # (false, 'int baz;'),...]</span><br><span style="color: hsl(120, 100%, 40%);">+ defs = []</span><br><span style="color: hsl(120, 100%, 40%);">+ any_sub_byte_ints = False</span><br><span style="color: hsl(120, 100%, 40%);">+ for one_def in def_strs:</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # does it have sub-string integers?</span><br><span style="color: hsl(120, 100%, 40%);">+ int_def = re_int_def.fullmatch(one_def)</span><br><span style="color: hsl(120, 100%, 40%);">+ if not int_def:</span><br><span style="color: hsl(120, 100%, 40%);">+ # not even a number, same for big and little endian</span><br><span style="color: hsl(120, 100%, 40%);">+ defs.append((False, one_def))</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ int_type = int_def.group(1)</span><br><span style="color: hsl(120, 100%, 40%);">+ members_str = int_def.groups()[-1]</span><br><span style="color: hsl(120, 100%, 40%);">+ has_sub_byte_ints = False</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ members = []</span><br><span style="color: hsl(120, 100%, 40%);">+ for int_member in re_int_members.finditer(members_str):</span><br><span style="color: hsl(120, 100%, 40%);">+ member = int_member.group(1)</span><br><span style="color: hsl(120, 100%, 40%);">+ members.append(member)</span><br><span style="color: hsl(120, 100%, 40%);">+ if ':' in member:</span><br><span style="color: hsl(120, 100%, 40%);">+ has_sub_byte_ints = True</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if not has_sub_byte_ints:</span><br><span style="color: hsl(120, 100%, 40%);">+ defs.append((False, one_def))</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ defs.append((True, one_def, int_type, members))</span><br><span style="color: hsl(120, 100%, 40%);">+ any_sub_byte_ints = True</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if not any_sub_byte_ints:</span><br><span style="color: hsl(120, 100%, 40%);">+ return None</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # now the interesting part, go over the defs, and reverse the sub-byte ints</span><br><span style="color: hsl(120, 100%, 40%);">+ # at byte boundaries.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ i = 0</span><br><span style="color: hsl(120, 100%, 40%);">+ got_bits = 0</span><br><span style="color: hsl(120, 100%, 40%);">+ byte_type = None</span><br><span style="color: hsl(120, 100%, 40%);">+ members_within_a_byte = []</span><br><span style="color: hsl(120, 100%, 40%);">+ big_endian_defs = []</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ big_defs = []</span><br><span style="color: hsl(120, 100%, 40%);">+ for classified_def in defs:</span><br><span style="color: hsl(120, 100%, 40%);">+ has_sub_byte_ints = classified_def[0]</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # now the big endian part</span><br><span style="color: hsl(120, 100%, 40%);">+ if has_sub_byte_ints:</span><br><span style="color: hsl(120, 100%, 40%);">+ _, one_def, int_type, members = classified_def</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if byte_type and byte_type.strip() != int_type.strip():</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('mismatching type continuation after incomplete byte: %r %r to %r'</span><br><span style="color: hsl(120, 100%, 40%);">+ % (byte_type, members_within_a_byte, int_type))</span><br><span style="color: hsl(120, 100%, 40%);">+ byte_type = int_type</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for member in members:</span><br><span style="color: hsl(120, 100%, 40%);">+ member_name, bits_str = member.split(':')</span><br><span style="color: hsl(120, 100%, 40%);">+ member_name = member_name.strip()</span><br><span style="color: hsl(120, 100%, 40%);">+ bits = int(bits_str)</span><br><span style="color: hsl(120, 100%, 40%);">+ member = '%s:%d' % (member_name, bits)</span><br><span style="color: hsl(120, 100%, 40%);">+ members_within_a_byte.append(member)</span><br><span style="color: hsl(120, 100%, 40%);">+ got_bits += bits</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if got_bits == 8:</span><br><span style="color: hsl(120, 100%, 40%);">+ # reverse these.</span><br><span style="color: hsl(120, 100%, 40%);">+ big_endian_defs.append('%s%s;' % (byte_type, ', '.join(reversed(members_within_a_byte))))</span><br><span style="color: hsl(120, 100%, 40%);">+ members_within_a_byte = []</span><br><span style="color: hsl(120, 100%, 40%);">+ byte_type = None</span><br><span style="color: hsl(120, 100%, 40%);">+ got_bits = 0</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ elif got_bits > 8:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('sub-byte int breaks clean byte bounds: %s -- %d + %d = %d bits'</span><br><span style="color: hsl(120, 100%, 40%);">+ % (member, got_bits - bits, bits, got_bits))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ elif not has_sub_byte_ints:</span><br><span style="color: hsl(120, 100%, 40%);">+ if got_bits:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('sub-byte members do not add up to clean byte bounds: %r' % members_within_a_byte)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ big_endian_defs.append(classified_def[1])</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # strip empty lines</span><br><span style="color: hsl(120, 100%, 40%);">+ lines = [l for l in (''.join(big_endian_defs).split('\n')) if l.strip()]</span><br><span style="color: hsl(120, 100%, 40%);">+ # clean lines' whitespace errors we might have taken in with the type names</span><br><span style="color: hsl(120, 100%, 40%);">+ for i in range(len(lines)):</span><br><span style="color: hsl(120, 100%, 40%);">+ line = lines[i]</span><br><span style="color: hsl(120, 100%, 40%);">+ while len(line) and line[-1] in ' \t':</span><br><span style="color: hsl(120, 100%, 40%);">+ line = line[:-1]</span><br><span style="color: hsl(120, 100%, 40%);">+ lines[i] = line</span><br><span style="color: hsl(120, 100%, 40%);">+ return '\n'.join(lines)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def handle_struct_body(body_str):</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ big_endian_body_str = struct_body_to_big_endian(body_str)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if big_endian_body_str:</span><br><span style="color: hsl(120, 100%, 40%);">+ new_lines = ['#if OSMO_IS_LITTLE_ENDIAN\n']</span><br><span style="color: hsl(120, 100%, 40%);">+ new_lines.append(body_str)</span><br><span style="color: hsl(120, 100%, 40%);">+ new_lines.append('#elif OSMO_IS_BIG_ENDIAN\n'</span><br><span style="color: hsl(120, 100%, 40%);">+ '/* auto-generated from the little endian part above (libosmocore/contrib/struct_endianess.py) */\n')</span><br><span style="color: hsl(120, 100%, 40%);">+ new_lines.append(big_endian_body_str)</span><br><span style="color: hsl(120, 100%, 40%);">+ new_lines.append('\n#endif\n')</span><br><span style="color: hsl(120, 100%, 40%);">+ return ''.join(new_lines)</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ return body_str</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def _check_file(f):</span><br><span style="color: hsl(120, 100%, 40%);">+ if not (f.endswith('.h') or f.endswith('.c') or f.endswith('.cpp')):</span><br><span style="color: hsl(120, 100%, 40%);">+ return</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # section the file into</span><br><span style="color: hsl(120, 100%, 40%);">+ # [ ["no struct def"], ["struct {...};"], ["no struct def"], ... ]</span><br><span style="color: hsl(120, 100%, 40%);">+ sections = []</span><br><span style="color: hsl(120, 100%, 40%);">+ in_struct = False</span><br><span style="color: hsl(120, 100%, 40%);">+ buf = []</span><br><span style="color: hsl(120, 100%, 40%);">+ for line in codecs.open(f, "r", "utf-8").readlines():</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if not in_struct and re_struct_start.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ # flush whatever might still be in buf from before</span><br><span style="color: hsl(120, 100%, 40%);">+ sections.append(buf)</span><br><span style="color: hsl(120, 100%, 40%);">+ # start an in_struct section</span><br><span style="color: hsl(120, 100%, 40%);">+ buf = [line]</span><br><span style="color: hsl(120, 100%, 40%);">+ in_struct = True</span><br><span style="color: hsl(120, 100%, 40%);">+ elif in_struct and re_struct_end.fullmatch(line):</span><br><span style="color: hsl(120, 100%, 40%);">+ # add this end to the in_struct section and then start a non-struct section</span><br><span style="color: hsl(120, 100%, 40%);">+ buf.append(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ sections.append(buf)</span><br><span style="color: hsl(120, 100%, 40%);">+ in_struct = False</span><br><span style="color: hsl(120, 100%, 40%);">+ buf = []</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ buf.append(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ # flush any leftovers in buf</span><br><span style="color: hsl(120, 100%, 40%);">+ if buf:</span><br><span style="color: hsl(120, 100%, 40%);">+ sections.append(buf)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # examine each struct, i.e. every second item in 'sections'</span><br><span style="color: hsl(120, 100%, 40%);">+ for i in range(len(sections)):</span><br><span style="color: hsl(120, 100%, 40%);">+ if not (i & 1):</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ struct = sections[i]</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # If the struct isn't packed, we need not bother.</span><br><span style="color: hsl(120, 100%, 40%);">+ # The practical use of this: in some structs we have booleans in the</span><br><span style="color: hsl(120, 100%, 40%);">+ # form of</span><br><span style="color: hsl(120, 100%, 40%);">+ # integer flag:1;</span><br><span style="color: hsl(120, 100%, 40%);">+ # and these don't add up to bytes, and cause errors. So let's skip all</span><br><span style="color: hsl(120, 100%, 40%);">+ # non-packed structs, then all of those are out of the picture.</span><br><span style="color: hsl(120, 100%, 40%);">+ if not 'packed' in struct[-1]:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ try:</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # assume the 'struct foo {' is on the first line, the closing brace</span><br><span style="color: hsl(120, 100%, 40%);">+ # '} __attribute...;' on the last, and the rest are individual</span><br><span style="color: hsl(120, 100%, 40%);">+ # definitions split by ';'.</span><br><span style="color: hsl(120, 100%, 40%);">+ struct_body_lines = struct[1:-1]</span><br><span style="color: hsl(120, 100%, 40%);">+ struct_body_parts = section_struct_body(struct_body_lines)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ new_struct_body_parts = []</span><br><span style="color: hsl(120, 100%, 40%);">+ for j in range(len(struct_body_parts)):</span><br><span style="color: hsl(120, 100%, 40%);">+ part = ''.join(struct_body_parts[j])</span><br><span style="color: hsl(120, 100%, 40%);">+ if not (j & 1):</span><br><span style="color: hsl(120, 100%, 40%);">+ new_struct_body_parts.append(part)</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ new_struct_body_parts.append(handle_struct_body(part))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ new_struct = [struct[0], ''.join(new_struct_body_parts), struct[-1]]</span><br><span style="color: hsl(120, 100%, 40%);">+ sections[i] = new_struct</span><br><span style="color: hsl(120, 100%, 40%);">+ except Exception as e:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('ERROR in struct %r' % struct[0])</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # phew. result.</span><br><span style="color: hsl(120, 100%, 40%);">+ result = ''.join((''.join(s) for s in sections))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # see if osmocom/core/endian.h is needed and included.</span><br><span style="color: hsl(120, 100%, 40%);">+ if (not f.endswith('endian.h')</span><br><span style="color: hsl(120, 100%, 40%);">+ and 'OSMO_IS_LITTLE_ENDIAN' in result</span><br><span style="color: hsl(120, 100%, 40%);">+ and '#include <osmocom/core/endian.h>' not in result):</span><br><span style="color: hsl(120, 100%, 40%);">+ # add the include after the last 'osmocom/core' include</span><br><span style="color: hsl(120, 100%, 40%);">+ last_include_start = result.rfind('#include <osmocom/core/')</span><br><span style="color: hsl(120, 100%, 40%);">+ if last_include_start < 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ last_include_start = result.rfind('#include <osmocom/')</span><br><span style="color: hsl(120, 100%, 40%);">+ if last_include_start < 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ last_include_start = result.rfind('#include')</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if last_include_start < 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('do not know where to include osmocom/core/endian.h in %r' % f)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ insert_at = result.find('\n', last_include_start)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ result = result[:insert_at] + '\n#include <osmocom/core/endian.h>' + result[insert_at:]</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ with codecs.open(f, "w", "utf-8") as fd:</span><br><span style="color: hsl(120, 100%, 40%);">+ fd.write(result)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+def check_file(f):</span><br><span style="color: hsl(120, 100%, 40%);">+ try:</span><br><span style="color: hsl(120, 100%, 40%);">+ _check_file(f)</span><br><span style="color: hsl(120, 100%, 40%);">+ except Exception as e:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception('ERROR IN FILE %r' % f)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+args = sys.argv[1:]</span><br><span style="color: hsl(120, 100%, 40%);">+if not args:</span><br><span style="color: hsl(120, 100%, 40%);">+ args = ['.']</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+for f in args:</span><br><span style="color: hsl(120, 100%, 40%);">+ if os.path.isdir(f):</span><br><span style="color: hsl(120, 100%, 40%);">+ for parent_path, subdirs, files in os.walk(f, None, None):</span><br><span style="color: hsl(120, 100%, 40%);">+ for ff in files:</span><br><span style="color: hsl(120, 100%, 40%);">+ check_file(os.path.join(parent_path, ff))</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ check_file(f)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# vim: tabstop=4 shiftwidth=4 expandtab</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/11786">change 11786</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/11786"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: libosmocore </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: merged </div>
<div style="display:none"> Gerrit-Change-Id: I8e75b17d8071c7b3a2a171ba776fb76854b28a53 </div>
<div style="display:none"> Gerrit-Change-Number: 11786 </div>
<div style="display:none"> Gerrit-PatchSet: 4 </div>
<div style="display:none"> Gerrit-Owner: Neels Hofmeyr <nhofmeyr@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder (1000002) </div>
<div style="display:none"> Gerrit-Reviewer: Neels Hofmeyr <nhofmeyr@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Pau Espin Pedrol <pespin@sysmocom.de> </div>