April 2025 - gerrit-log - lists.osmocom.org

[S] Change in osmocom-bb[master]: trx_toolkit: Move FakeTRX-related performance-sensitive modules to Cy...

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40052?usp=email ) Change subject: trx_toolkit: Move FakeTRX-related performance-sensitive modules to Cython ...................................................................... trx_toolkit: Move FakeTRX-related performance-sensitive modules to Cython Cython (https://cython.org/ , https://cython.readthedocs.io/en/latest/) is Python-compatible compiled language, that extends Python and allows to add type annotations which help to optimize generated code. It is based on Pyrex (https://www.csse.canterbury.ac.nz/greg.ewing/python/Pyrex/) which is no longer actively maintained this days. Using Cython allows one to take Python codebase and improve its speed incrementally step by step instead of doing full rewrite from scratch. Here we are only hooking Cython build system into the project and move *.py to *.pyx files for the following modules: - burst_fwd.c - _clck_gen.c - data_if.c - data_msg.c - _fake_trx.c - udp_link.c - transceiver.c Those modules turned up to require being optimized because their functionality is used by fake_trx loop non-stop at the very least ~2000 times a second if there is only one BTS attached to fake_trx. We will optimize those modules step by step in the follow-up patches. For now it is only plain code movement with adding # cython: language_level=3 comment to the top of each file because else Cython uses 3str language mode and emits a warning. We use language_level=3 because it has the same semantic as of regular Python3. The project now needs to be build before running: either `pip install -e .` or `python setup.py build_ext -i` when doing development, or simple `pip install .` for non-develop build. There is no performance increase just from the move to Cython. We will add optimizations step by step in the follow-up patches. Change-Id: I2159a07bece13bda4f6ccd957063d4644d8b5e4f --- M src/target/trx_toolkit/.gitignore R src/target/trx_toolkit/_clck_gen.pyx R src/target/trx_toolkit/_fake_trx.pyx R src/target/trx_toolkit/burst_fwd.pyx R src/target/trx_toolkit/data_if.pyx R src/target/trx_toolkit/data_msg.pyx A src/target/trx_toolkit/pyproject.toml A src/target/trx_toolkit/setup.py R src/target/trx_toolkit/transceiver.pyx R src/target/trx_toolkit/udp_link.pyx 10 files changed, 35 insertions(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/52/40052/1 diff --git a/src/target/trx_toolkit/.gitignore b/src/target/trx_toolkit/.gitignore index dc634e2..3ea0ef1 100644 --- a/src/target/trx_toolkit/.gitignore +++ b/src/target/trx_toolkit/.gitignore @@ -2,5 +2,14 @@ __pycache__/ *.py[cod] *$py.class +/build/ +/*.so /perf.data* /flamegraph.html +/burst_fwd.c +/_clck_gen.c +/data_if.c +/data_msg.c +/_fake_trx.c +/udp_link.c +/transceiver.c diff --git a/src/target/trx_toolkit/_clck_gen.py b/src/target/trx_toolkit/_clck_gen.pyx similarity index 98% rename from src/target/trx_toolkit/_clck_gen.py rename to src/target/trx_toolkit/_clck_gen.pyx index 9d9fd15..914b821 100644 --- a/src/target/trx_toolkit/_clck_gen.py +++ b/src/target/trx_toolkit/_clck_gen.pyx @@ -1,3 +1,5 @@ +# cython: language_level=3 + # TRX Toolkit # Simple TDMA frame clock generator # diff --git a/src/target/trx_toolkit/_fake_trx.py b/src/target/trx_toolkit/_fake_trx.pyx similarity index 99% rename from src/target/trx_toolkit/_fake_trx.py rename to src/target/trx_toolkit/_fake_trx.pyx index 87a352b..92573f8 100644 --- a/src/target/trx_toolkit/_fake_trx.py +++ b/src/target/trx_toolkit/_fake_trx.pyx @@ -1,3 +1,5 @@ +# cython: language_level=3 + # TRX Toolkit # Virtual Um-interface (fake transceiver) # diff --git a/src/target/trx_toolkit/burst_fwd.py b/src/target/trx_toolkit/burst_fwd.pyx similarity index 98% rename from src/target/trx_toolkit/burst_fwd.py rename to src/target/trx_toolkit/burst_fwd.pyx index 2824e0a..2737232 100644 --- a/src/target/trx_toolkit/burst_fwd.py +++ b/src/target/trx_toolkit/burst_fwd.pyx @@ -1,4 +1,5 @@ # -*- coding: utf-8 -*- +# cython: language_level=3 # TRX Toolkit # Burst forwarding between transceivers diff --git a/src/target/trx_toolkit/data_if.py b/src/target/trx_toolkit/data_if.pyx similarity index 98% rename from src/target/trx_toolkit/data_if.py rename to src/target/trx_toolkit/data_if.pyx index f59ca17..396987e 100644 --- a/src/target/trx_toolkit/data_if.py +++ b/src/target/trx_toolkit/data_if.pyx @@ -1,4 +1,5 @@ # -*- coding: utf-8 -*- +# cython: language_level=3 # TRX Toolkit # DATA interface implementation diff --git a/src/target/trx_toolkit/data_msg.py b/src/target/trx_toolkit/data_msg.pyx similarity index 99% rename from src/target/trx_toolkit/data_msg.py rename to src/target/trx_toolkit/data_msg.pyx index b1673f5..af0aac6 100644 --- a/src/target/trx_toolkit/data_msg.py +++ b/src/target/trx_toolkit/data_msg.pyx @@ -1,4 +1,5 @@ # -*- coding: utf-8 -*- +# cython: language_level=3 # TRX Toolkit # DATA interface message definitions and helpers diff --git a/src/target/trx_toolkit/pyproject.toml b/src/target/trx_toolkit/pyproject.toml new file mode 100644 index 0000000..e9f294d --- /dev/null +++ b/src/target/trx_toolkit/pyproject.toml @@ -0,0 +1,2 @@ +[build-system] +requires = ["setuptools", "wheel", "cython"] diff --git a/src/target/trx_toolkit/setup.py b/src/target/trx_toolkit/setup.py new file mode 100644 index 0000000..78de7ff --- /dev/null +++ b/src/target/trx_toolkit/setup.py @@ -0,0 +1,15 @@ +from setuptools import setup +from Cython.Build import cythonize + +setup( + name='trx_toolkit', + ext_modules = cythonize([ + "burst_fwd.pyx", + "_clck_gen.pyx", + "data_if.pyx", + "data_msg.pyx", + "_fake_trx.pyx", + "udp_link.pyx", + "transceiver.pyx" + ]) +) diff --git a/src/target/trx_toolkit/transceiver.py b/src/target/trx_toolkit/transceiver.pyx similarity index 99% rename from src/target/trx_toolkit/transceiver.py rename to src/target/trx_toolkit/transceiver.pyx index 2ebb29a..aaf3a90 100644 --- a/src/target/trx_toolkit/transceiver.py +++ b/src/target/trx_toolkit/transceiver.pyx @@ -1,4 +1,5 @@ # -*- coding: utf-8 -*- +# cython: language_level=3 # TRX Toolkit # Transceiver implementation diff --git a/src/target/trx_toolkit/udp_link.py b/src/target/trx_toolkit/udp_link.pyx similarity index 97% rename from src/target/trx_toolkit/udp_link.py rename to src/target/trx_toolkit/udp_link.pyx index 53a0bfb..779c0fc 100644 --- a/src/target/trx_toolkit/udp_link.py +++ b/src/target/trx_toolkit/udp_link.pyx @@ -1,4 +1,5 @@ # -*- coding: utf-8 -*- +# cython: language_level=3 # TRX Toolkit # UDP link implementation -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40052?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: I2159a07bece13bda4f6ccd957063d4644d8b5e4f Gerrit-Change-Number: 40052 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[S] Change in osmocom-bb[master]: trx_toolkit/udp_link: Switch UDPLink to cdef class

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40054?usp=email ) Change subject: trx_toolkit/udp_link: Switch UDPLink to cdef class ...................................................................... trx_toolkit/udp_link: Switch UDPLink to cdef class - Put fields into the object struct; fields are now accessed directly via that C-level struct instead of via __dict__ lookup - cimport instead of import UDPLink at the users Change-Id: I4768d06450873c3052ea29420848cf7a2752e474 --- M src/target/trx_toolkit/data_if.pyx M src/target/trx_toolkit/transceiver.pyx A src/target/trx_toolkit/udp_link.pxd M src/target/trx_toolkit/udp_link.pyx 4 files changed, 10 insertions(+), 4 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/54/40054/1 diff --git a/src/target/trx_toolkit/data_if.pyx b/src/target/trx_toolkit/data_if.pyx index 396987e..8dcc4dc 100644 --- a/src/target/trx_toolkit/data_if.pyx +++ b/src/target/trx_toolkit/data_if.pyx @@ -20,7 +20,7 @@ import logging as log -from udp_link import UDPLink +from udp_link cimport UDPLink from data_msg import * class DATAInterface(UDPLink): diff --git a/src/target/trx_toolkit/transceiver.pyx b/src/target/trx_toolkit/transceiver.pyx index aaf3a90..519eb4b 100644 --- a/src/target/trx_toolkit/transceiver.pyx +++ b/src/target/trx_toolkit/transceiver.pyx @@ -23,7 +23,7 @@ from ctrl_if_trx import CTRLInterfaceTRX from data_if import DATAInterface -from udp_link import UDPLink +from udp_link cimport UDPLink from trx_list import TRXList from gsm_shared import HoppingParams diff --git a/src/target/trx_toolkit/udp_link.pxd b/src/target/trx_toolkit/udp_link.pxd new file mode 100644 index 0000000..2d02497 --- /dev/null +++ b/src/target/trx_toolkit/udp_link.pxd @@ -0,0 +1,6 @@ +# cython: language_level=3 + +cdef class UDPLink: + cdef readonly object sock + cdef str remote_addr + cdef int remote_port diff --git a/src/target/trx_toolkit/udp_link.pyx b/src/target/trx_toolkit/udp_link.pyx index 779c0fc..ae90021 100644 --- a/src/target/trx_toolkit/udp_link.pyx +++ b/src/target/trx_toolkit/udp_link.pyx @@ -20,8 +20,8 @@ import socket -class UDPLink: - def __init__(self, remote_addr, remote_port, bind_addr = '0.0.0.0', bind_port = 0): +cdef class UDPLink: + def __init__(self, str remote_addr, int remote_port, bind_addr = '0.0.0.0', bind_port = 0): self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.sock.bind((bind_addr, bind_port)) -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40054?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: I4768d06450873c3052ea29420848cf7a2752e474 Gerrit-Change-Number: 40054 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[XS] Change in osmocom-bb[master]: trx_toolkit/burst_fwd: Do not inherit BurstForwarder from TRXList

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40053?usp=email ) Change subject: trx_toolkit/burst_fwd: Do not inherit BurstForwarder from TRXList ...................................................................... trx_toolkit/burst_fwd: Do not inherit BurstForwarder from TRXList That is simply unneeded as TRXList is used primarly by fake_trx.Application when doing initial setup of TRX list, while for BurstForwarder it is enough to have just a regular list of TRX instances. We need to remove TRXList from BurstForwarder base to be able to switch BurstForwarder to cdef class in a follow-up patch as that is possible only with cdef class basesm, while TRXList is plain py class. Change-Id: I9ce8ca1531eea5d59358fb41cc99d8c349757abb --- M src/target/trx_toolkit/burst_fwd.pyx 1 file changed, 4 insertions(+), 3 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/53/40053/1 diff --git a/src/target/trx_toolkit/burst_fwd.pyx b/src/target/trx_toolkit/burst_fwd.pyx index 2737232..36e5095 100644 --- a/src/target/trx_toolkit/burst_fwd.pyx +++ b/src/target/trx_toolkit/burst_fwd.pyx @@ -21,9 +21,7 @@ import logging as log -from trx_list import TRXList - -class BurstForwarder(TRXList): +class BurstForwarder: """ Performs burst forwarding between transceivers. BurstForwarder distributes bursts between the list of given @@ -42,6 +40,9 @@ """ + def __init__(self, trx_list): + self.trx_list = trx_list + def forward_msg(self, src_trx, rx_msg): # Originating Transceiver may use frequency hopping, # so let's precalculate its Tx frequency in advance -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40053?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: I9ce8ca1531eea5d59358fb41cc99d8c349757abb Gerrit-Change-Number: 40053 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[M] Change in osmocom-bb[master]: trx_toolkit/udp_link: Optimize UDPLink.send

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40055?usp=email ) Change subject: trx_toolkit/udp_link: Optimize UDPLink.send ...................................................................... trx_toolkit/udp_link: Optimize UDPLink.send As can be seen from http://navytux.spb.ru/~kirr/osmo/fake_trx/pyx-base.html (UDPLink_11send) UDPLink.send spends ~ 30% of its time in python overhead, with doing only ~ 70% of the time in the sendto syscall. As fake_trx invokes send a lot it makes sense to cut that overhead. -> Do that: - prepare destination address in parsed form only once instead of parsing it in py runtime at every call - avoid doing PyGetBuffer... dance - just support bytes|bytearray|str for data and use fast CAPI macros to retrieve underlying data pointer - invoke sendto syscall directly ourselves - do not release/reacquire GIL to avoid corresponding performance penalty. As Iaa675c95059ec8ccfad667f69984d5a7f608c249 (trx_toolkit/clck_gen: Don't use threads because Python GIL is latency killer) shown GIL-related functions can be little in the profile, but harm latency a lot. We can skip doing release/reacquire GIL because we know UDPLink's socket is non-blocking. Change-Id: I83204545066a925dadbcd0b72cbbc2e3407129fe --- M src/target/trx_toolkit/udp_link.pxd M src/target/trx_toolkit/udp_link.pyx 2 files changed, 64 insertions(+), 9 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/55/40055/1 diff --git a/src/target/trx_toolkit/udp_link.pxd b/src/target/trx_toolkit/udp_link.pxd index 2d02497..d50fe5f 100644 --- a/src/target/trx_toolkit/udp_link.pxd +++ b/src/target/trx_toolkit/udp_link.pxd @@ -1,6 +1,21 @@ # cython: language_level=3 +from libc.stdint cimport uint16_t, uint32_t + +cdef extern from "<arpa/inet.h>": + struct sockaddr: + pass + struct in_addr: + uint32_t s_addr + struct sockaddr_in: + int sin_family + uint16_t sin_port + in_addr sin_addr + cdef class UDPLink: cdef readonly object sock - cdef str remote_addr - cdef int remote_port + cdef int sock_fd + cdef sockaddr_in remote_addr + + +cdef _raise_oserr() diff --git a/src/target/trx_toolkit/udp_link.pyx b/src/target/trx_toolkit/udp_link.pyx index ae90021..d75e058 100644 --- a/src/target/trx_toolkit/udp_link.pyx +++ b/src/target/trx_toolkit/udp_link.pyx @@ -20,38 +20,78 @@ import socket +from cpython cimport PyBytes_AS_STRING, PyBytes_GET_SIZE, PyUnicode_AsUTF8AndSize +from cpython.bytearray cimport PyByteArray_FromStringAndSize, PyByteArray_AS_STRING, PyByteArray_GET_SIZE + +from libc.errno cimport errno +from libc.string cimport strerror + +cdef extern from "<arpa/inet.h>": + int inet_pton(int af, const char *src, void *dst) + char *inet_ntoa(in_addr) + uint16_t htons(uint16_t) + +cdef extern from "<sys/socket.h>": + ssize_t sendto(int fd, const void *buf, size_t len, int flags, const sockaddr *dst_addr, int addrlen) + + cdef class UDPLink: def __init__(self, str remote_addr, int remote_port, bind_addr = '0.0.0.0', bind_port = 0): + # Save remote address in parsed form + self.remote_addr.sin_family = socket.AF_INET + self.remote_addr.sin_port = htons(remote_port) + err = inet_pton(socket.AF_INET, remote_addr.encode(), &self.remote_addr.sin_addr) + if err <= 0: + _raise_oserr() + self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.sock.bind((bind_addr, bind_port)) self.sock.setblocking(False) - # Save remote info - self.remote_addr = remote_addr - self.remote_port = remote_port + self.sock_fd = self.sock.fileno() def __del__(self): self.sock.close() + self.sock_fd = -1 def desc_local(self): (bind_addr, bind_port) = self.sock.getsockname() return "%s:%u" % (bind_addr, bind_port) def desc_remote(self): - return "%s:%u" % (self.remote_addr, self.remote_port) + return "%s:%u" % (inet_ntoa(self.remote_addr.sin_addr), self.remote_addr.sin_port) def desc_link(self): return "L:%s <-> R:%s" % (self.desc_local(), self.desc_remote()) def send(self, data): - if type(data) not in [bytearray, bytes]: - data = data.encode() + cdef const char *buf + cdef Py_ssize_t buflen + if type(data) is bytearray: + buf = PyByteArray_AS_STRING(data) + buflen = PyByteArray_GET_SIZE(data) + elif type(data) is bytes: + buf = PyBytes_AS_STRING(data) + buflen = PyBytes_GET_SIZE(data) + elif type(data) is str: + buf = PyUnicode_AsUTF8AndSize(data, &buflen) + else: + raise TypeError("send: accept only bytes|bytearray|str ; got %r" % type(data)) - self.sock.sendto(data, (self.remote_addr, self.remote_port)) + # NOTE we do not release/reacquire gil to save us from gil ping-pong performance penalty + # we can do that because the socket is non-blocking + n = sendto(self.sock_fd, buf, buflen, + 0, <sockaddr*>&self.remote_addr, sizeof(self.remote_addr)) + if n == -1: + _raise_oserr() def sendto(self, data, remote): if type(data) not in [bytearray, bytes]: data = data.encode() self.sock.sendto(data, remote) + + +cdef _raise_oserr(): + raise OSError(errno, strerror(errno)) -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40055?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: I83204545066a925dadbcd0b72cbbc2e3407129fe Gerrit-Change-Number: 40055 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[S] Change in osmocom-bb[master]: trx_toolkit/udp_link: Factor code to describe remote into .desc_remot...

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40046?usp=email ) Change subject: trx_toolkit/udp_link: Factor code to describe remote into .desc_remote() function ...................................................................... trx_toolkit/udp_link: Factor code to describe remote into .desc_remote() function And use that utility everywhere where remote of UDPLink is logged. The reason we are doing this is that with upcoming switch to Cython the way remote address is stored will change to `struct sockaddr_in` and instead of updating all users, we will need to only change UDPLink.desc_remote() in one place. Add .desc_local() for symmetry. Change-Id: I1e2fa560ada7a8de4c9b9150058c2a1c73874fbe --- M src/target/trx_toolkit/data_if.py M src/target/trx_toolkit/udp_link.py 2 files changed, 9 insertions(+), 5 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/46/40046/1 diff --git a/src/target/trx_toolkit/data_if.py b/src/target/trx_toolkit/data_if.py index 5bc243f..f59ca17 100644 --- a/src/target/trx_toolkit/data_if.py +++ b/src/target/trx_toolkit/data_if.py @@ -70,7 +70,7 @@ msg.parse_msg(data) except: log.error("Failed to parse a TRXD Tx message " - "from R:%s:%u" % (self.remote_addr, self.remote_port)) + "from R:%s" % self.desc_remote()) return None # Make sure the header version matches @@ -90,7 +90,7 @@ msg.parse_msg(bytearray(data)) except: log.error("Failed to parse a TRXD Rx message " - "from R:%s:%u" % (self.remote_addr, self.remote_port)) + "from R:%s" % self.desc_remote()) return None # Make sure the header version matches diff --git a/src/target/trx_toolkit/udp_link.py b/src/target/trx_toolkit/udp_link.py index c3f1476..53a0bfb 100644 --- a/src/target/trx_toolkit/udp_link.py +++ b/src/target/trx_toolkit/udp_link.py @@ -33,11 +33,15 @@ def __del__(self): self.sock.close() - def desc_link(self): + def desc_local(self): (bind_addr, bind_port) = self.sock.getsockname() + return "%s:%u" % (bind_addr, bind_port) - return "L:%s:%u <-> R:%s:%u" \ - % (bind_addr, bind_port, self.remote_addr, self.remote_port) + def desc_remote(self): + return "%s:%u" % (self.remote_addr, self.remote_port) + + def desc_link(self): + return "L:%s <-> R:%s" % (self.desc_local(), self.desc_remote()) def send(self, data): if type(data) not in [bytearray, bytes]: -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40046?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: I1e2fa560ada7a8de4c9b9150058c2a1c73874fbe Gerrit-Change-Number: 40046 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[XS] Change in osmocom-bb[master]: trx_toolkit/burst_fwd: Use 'is' instead of '==' when checking if trx ...

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40044?usp=email ) Change subject: trx_toolkit/burst_fwd: Use 'is' instead of '==' when checking if trx is src_trx ...................................................................... trx_toolkit/burst_fwd: Use 'is' instead of '==' when checking if trx is src_trx In python `a is b` is just a pointer comparison of locations of objects a and b, while `a == b` can involve doing arbitrary code invoking __eq__ and is generally slower even without __eq__ defined: In [1]: class A: ...: pass ...: In [2]: a = A() In [4]: b = A() In [5]: a == a Out[5]: True In [6]: a == b Out[6]: False In [7]: a is a Out[7]: True In [8]: a is b Out[8]: False In [9]: %timeit a is a 84.2 ns ± 0.133 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) In [10]: %timeit a is b 87.5 ns ± 0.0736 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) In [11]: %timeit a == a 100 ns ± 0.659 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) In [12]: %timeit a == b 116 ns ± 0.399 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) BurstForwarder.forward_msg is one of the hottest places, as e.g. for every received packet from BTS it forwards it to multiple Ms receivers. It makes sense to be careful and save cycles here. Change-Id: Ic9e16720daeb348b5f9c535c24a682db53a93529 --- M src/target/trx_toolkit/burst_fwd.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/44/40044/1 diff --git a/src/target/trx_toolkit/burst_fwd.py b/src/target/trx_toolkit/burst_fwd.py index 6924531..e3da9c2 100644 --- a/src/target/trx_toolkit/burst_fwd.py +++ b/src/target/trx_toolkit/burst_fwd.py @@ -52,7 +52,7 @@ # Iterate over all known transceivers for trx in self.trx_list: - if trx == src_trx: + if trx is src_trx: continue # Check transceiver state -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40044?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: Ic9e16720daeb348b5f9c535c24a682db53a93529 Gerrit-Change-Number: 40044 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[XS] Change in osmocom-bb[master]: trx_toolkit/*: Don't use `del x; x = None` idiom

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40045?usp=email ) Change subject: trx_toolkit/*: Don't use `del x; x = None` idiom ...................................................................... trx_toolkit/*: Don't use `del x; x = None` idiom In Python it is enough to do `x = None` to release the object that was pointed to by x previously. So at Python level doing this extra del only spends more CPU time, but when we will switch to Cython, and e.g. Msg.burst will be a C-level attribute, the following will not work at all del msg.burst msg.burst = None because at runtime it will complain that del msg.burst AttributeError: 'Msg' object has no attribute 'burst' and no __dict__ for setting new attributes -> Remove unneeded del to save some time and avoid problems with upcoming switch to Cython. Change-Id: I7a83bdd52fb9318bd8b975f85ce37c7144873f61 --- M src/target/trx_toolkit/burst_fwd.py M src/target/trx_toolkit/clck_gen.py M src/target/trx_toolkit/fake_trx.py 3 files changed, 2 insertions(+), 5 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/45/40045/1 diff --git a/src/target/trx_toolkit/burst_fwd.py b/src/target/trx_toolkit/burst_fwd.py index e3da9c2..2824e0a 100644 --- a/src/target/trx_toolkit/burst_fwd.py +++ b/src/target/trx_toolkit/burst_fwd.py @@ -47,8 +47,7 @@ tx_freq = src_trx.get_tx_freq(rx_msg.fn) if src_trx.rf_muted: - del rx_msg.burst # burst bits are omited - rx_msg.burst = None + rx_msg.burst = None # burst bits are omited # Iterate over all known transceivers for trx in self.trx_list: diff --git a/src/target/trx_toolkit/clck_gen.py b/src/target/trx_toolkit/clck_gen.py index ae7a899..f769c3e 100755 --- a/src/target/trx_toolkit/clck_gen.py +++ b/src/target/trx_toolkit/clck_gen.py @@ -82,7 +82,6 @@ self._thread.join() # Free memory, reset breaker - del self._thread self._thread = None self._breaker.clear() diff --git a/src/target/trx_toolkit/fake_trx.py b/src/target/trx_toolkit/fake_trx.py index 8f187ac..711ad21 100755 --- a/src/target/trx_toolkit/fake_trx.py +++ b/src/target/trx_toolkit/fake_trx.py @@ -226,8 +226,7 @@ return # Since TRXDv1, we should send a NOPE.ind - del msg.burst # burst bits are omited - msg.burst = None + msg.burst = None # burst bits are omited # TODO: shoud we make these values configurable? msg.toa256 = self.TOA256_NOISE_DEFAULT -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40045?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: I7a83bdd52fb9318bd8b975f85ce37c7144873f61 Gerrit-Change-Number: 40045 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[S] Change in osmocom-bb[master]: trx_toolkit/transceiver: Remove _tx_queue_lock

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40048?usp=email ) Change subject: trx_toolkit/transceiver: Remove _tx_queue_lock ...................................................................... trx_toolkit/transceiver: Remove _tx_queue_lock After Iaa675c95059ec8ccfad667f69984d5a7f608c249 (trx_toolkit/clck_gen: Don't use threads because Python GIL is latency killer) fake_trx.py is single-threaded architecture. There is no need to use mutex when adding/removing entries to/from TX queue. Change-Id: I77762564326b370479845a347fc512cb5f643bda --- M src/target/trx_toolkit/transceiver.py 1 file changed, 12 insertions(+), 23 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/48/40048/1 diff --git a/src/target/trx_toolkit/transceiver.py b/src/target/trx_toolkit/transceiver.py index ce71bb1..2ebb29a 100644 --- a/src/target/trx_toolkit/transceiver.py +++ b/src/target/trx_toolkit/transceiver.py @@ -19,7 +19,6 @@ # GNU General Public License for more details. import logging as log -import threading from ctrl_if_trx import CTRLInterfaceTRX from data_if import DATAInterface @@ -134,7 +133,7 @@ a L12TRX message on the TRXD (data) interface, so it gets queued by this function. Then, to ensure the timeous transmission, the user of this implementation needs to call clck_tick() on each TDMA - frame. Both functions are thread-safe (queue mutex). + frame. In a multi-trx configuration, the use of queue additionally ensures proper burst aggregation on multiple TRXD connections, so all L12TRX @@ -193,8 +192,7 @@ # List of child transceivers self.child_trx_list = TRXList() - # Tx (L12TRX) burst queue and mutex - self._tx_queue_lock = threading.Lock() + # Tx (L12TRX) burst queue self._tx_queue = [] def __str__(self): @@ -256,7 +254,7 @@ for trx in trx_list: trx.running = poweron if not poweron: - trx.tx_queue_clear() + trx._tx_queue.clear() trx.disable_fh() # Trigger clock generator if required @@ -289,21 +287,13 @@ return None # Enqueue the message, it will be sent later - self.tx_queue_append(msg) + self._tx_queue.append(msg) return msg def handle_data_msg(self, msg): # TODO: make legacy mode configurable (via argv?) self.data_if.send_msg(msg, legacy = True) - def tx_queue_append(self, msg): - with self._tx_queue_lock: - self._tx_queue.append(msg) - - def tx_queue_clear(self): - with self._tx_queue_lock: - self._tx_queue.clear() - def clck_tick(self, fwd, fn): if not self.running: return @@ -312,16 +302,15 @@ emit = [] wait = [] - with self._tx_queue_lock: - for msg in self._tx_queue: - if msg.fn < fn: - drop.append(msg) - elif msg.fn == fn: - emit.append(msg) - else: - wait.append(msg) + for msg in self._tx_queue: + if msg.fn < fn: + drop.append(msg) + elif msg.fn == fn: + emit.append(msg) + else: + wait.append(msg) - self._tx_queue = wait + self._tx_queue = wait for msg in emit: fwd.forward_msg(self, msg) -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40048?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: I77762564326b370479845a347fc512cb5f643bda Gerrit-Change-Number: 40048 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[M] Change in osmocom-bb[master]: trx_toolkit/clck_gen: Don't use threads because Python GIL is latency...

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40047?usp=email ) Change subject: trx_toolkit/clck_gen: Don't use threads because Python GIL is latency killer ...................................................................... trx_toolkit/clck_gen: Don't use threads because Python GIL is latency killer Currently running fake_trx.py with just 1 BTS and 1 ccch_scan shows a bit high CPU usage on 2 cores and regular time overrun in CLCKgen: 1 BTS + 1 ccch_scan: 2 threads: first ~ 85-90% CPU, second ~ 50% CPU; regular time overrun in CLCKgen Trying to run fake_trx.py with 1 BTS and 2 ccch_scan shows even more CPU usage and non-stop CLCKGen time overruns: 1 BTS + 2 ccch_scan: 2 threads: first ~ 100% CPU, second ~ 70% CPU; time overrun all the time with BTS also emitting something like the following all the time: 20250416102139774 <0006> scheduler_trx.c:591 We were 42 FN faster than TRX, compensating 20250416102140358 <0006> scheduler_trx.c:591 We were 24 FN faster than TRX, compensating Profiling fake_trx.py for 1 BTS + 1 ccch_scan case via perf (see https://docs.python.org/3/howto/perf_profiling.html) as $ perf record -g python -X perf fake_trx.py --trx M2@127.0.0.1:7700 --trx M3@127.0.0.1:8700 shows that besides in-kernel work related to send/recv/select syscalls the system is also releasing/reacquiring GIL frequently http://navytux.spb.ru/~kirr/osmo/fake_trx/gil-pingpong.html (search for "gil" and "Thread" there) Which is understandable because in current architecture there are 2 threads: - T1 is running CLCKGen loop + TX work - T2 is running FakeTRX loop + RX work Unfortunately, even though the functions that take/release the GIL are not very huge themselves in the profile, still the GIL creates a huge problem about latency, and fake_trx problems of having CLCKGen time overrun are directly related to latency, because it means that CLCKGen could not be woken up in time and missed 2 GSM frames to forward queued burst timely. The GIL creates latency problems because of the following: in py2 the GIL was working like this: the main interpreter evaluator was checking whether there is a GIL taking request every 100 instructions, and if there was such a request from another python thread, it was making the switch. However in py3 they did change the way how GIL works and instead of switching every 100 instructions, the GIL was made to preemptively switch every 5 _milliseconds_ to avoid unnecessary switches. This helped throughput of computation-heavy workloads, but harmed the workloads that are latency sensitive. Please see the following presentation by David Beazley with overview on how GIL works on py2 and py3: https://speakerdeck.com/dabeaz/understanding-the-python-gil https://dabeaz.com/GIL/ That presentation actually mentions the problem of GIL latency on py3 and further explains it on https://www.dabeaz.com/blog/2010/02/revisiting-thread-priorities-and-new.ht… that problem was further filed to Python issue tracker at the same 2010 in https://bugs.python.org/issue7946 but was never addressed despite several propositional patches. In the end it was canceled by the original reported ten years later https://bugs.python.org/issue7946#msg377865 So despite the many years of this problem being known and not addressed despite several patches proposed, there is, practically, no hope that it will be addressed at all. And what happens in fake_trx case is that CLCKGen thread can be prevented from timely woken up if main loop thread is e.g. busy in decoding received bursts. That's why we see lots of time overruns, because even due to the Python GIL only we can be practically wasting 5 milliseconds. Not to mention that there are more problems related to threads and frequently switching in between them, because of e.g. 2 cores need to ping pong data in between their caches and similar things. -> cancel all that threading to avoid latency hits and organize CLCKGen timer to be hooked into the sole IO loop by the way of timerfd system call. This help latency and CPU usage significantly. After this patch: 1 BTS + 1 ccch_scan: 1 thread ~ 55% CPU; no CLCKGen time overruns 1 BTS + 2 ccch_scan: 1 thread ~ 65% CPU; seldom CLCKGen time overruns Unfortunately os.timerfd_create() & friends are only available starting from Python 3.13 . If this poses a problem it can be easily solved by doing those timerfd related system calls in Cython, after we switch most of the codebase to Cython later. Change-Id: Iaa675c95059ec8ccfad667f69984d5a7f608c249 --- M src/target/trx_toolkit/.gitignore M src/target/trx_toolkit/clck_gen.py M src/target/trx_toolkit/fake_trx.py M src/target/trx_toolkit/test_clck_gen.py 4 files changed, 63 insertions(+), 65 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/47/40047/1 diff --git a/src/target/trx_toolkit/.gitignore b/src/target/trx_toolkit/.gitignore index 749ccda..dc634e2 100644 --- a/src/target/trx_toolkit/.gitignore +++ b/src/target/trx_toolkit/.gitignore @@ -2,3 +2,5 @@ __pycache__/ *.py[cod] *$py.class +/perf.data* +/flamegraph.html diff --git a/src/target/trx_toolkit/clck_gen.py b/src/target/trx_toolkit/clck_gen.py index f769c3e..a223572 100755 --- a/src/target/trx_toolkit/clck_gen.py +++ b/src/target/trx_toolkit/clck_gen.py @@ -21,25 +21,26 @@ APP_CR_HOLDERS = [("2017-2019", "Vadim Yanitskiy <axilirator(a)gmail.com>")] import logging as log -import threading import time import signal import os +import sys from app_common import ApplicationBase from udp_link import UDPLink from gsm_shared import * + +ns = 1e-9 +us = 1e-6 + + class CLCKGen: # GSM TDMA definitions SEC_DELAY_US = 1000 * 1000 GSM_FRAME_US = 4615.0 - def __init__(self, clck_links, clck_start = 0, ind_period = 102, sched_rr_prio = None): - # This event is needed to control the thread - self._breaker = threading.Event() - self._thread = None - + def __init__(self, clck_links, clck_start = 0, ind_period = 102): self.clck_links = clck_links self.ind_period = ind_period self.clck_start = clck_start @@ -47,71 +48,56 @@ # Calculate counter time self.ctr_interval = self.GSM_FRAME_US self.ctr_interval /= self.SEC_DELAY_US + self._t_tick = int(self.ctr_interval // ns) + + # Initialize timer fd + self._timerfd = os.timerfd_create(time.CLOCK_MONOTONIC) # (Optional) clock consumer self.clck_handler = None - # RR Scheduler priority of thread. None = don't set it. - self.sched_rr_prio = sched_rr_prio + def __del__(self): + os.close(self._timerfd) @property def running(self): - if self._thread is None: - return False - return self._thread.is_alive() + t_next, _ = os.timerfd_gettime_ns(self._timerfd) + return (t_next != 0) def start(self): - # Make sure we won't start two threads - assert(self._thread is None) - # (Re)set the clock counter self.clck_src = self.clck_start - # Initialize and start a new thread - self._thread = threading.Thread(target = self._worker) - self._thread.daemon = True - self._thread.start() + # start timer fd + os.timerfd_settime_ns(self._timerfd, initial=self._t_tick, interval=self._t_tick) def stop(self): - # No thread, no problem ;) - if self._thread is None: - return + # stop timer fd + os.timerfd_settime_ns(self._timerfd, initial=0, interval=0) - # Stop the thread first - self._breaker.set() - self._thread.join() - - # Free memory, reset breaker - self._thread = None - self._breaker.clear() - - def _worker(self): - if self.sched_rr_prio is not None: - sched_param = os.sched_param(self.sched_rr_prio) - try: - log.info("CLCKGen: Setting real time process scheduler to SCHED_RR, priority %u" % (self.sched_rr_prio)) - os.sched_setscheduler(0, os.SCHED_RR, sched_param) - except OSError: - log.error("CLCKGen: Failed to set real time process scheduler to SCHED_RR, priority %u" % (self.sched_rr_prio)) + # tick must be called periodically by CLCKGen user. + # + # It waits for the next GSM frame to happen and emits corresponding clock indication at that time. + # It also runs attached .clck_handler if there is one. + # + # It is possible to use .tick in both blocking and non-blocking ways: + # + # - without extra care .tick will block waiting for the next GSM frame as explained above, + # - client code can also poll/select on ._timerfd to wait for GSM frame. + # After ._timerfd becomes ready it is guaranteed that the next .tick call will not block. + def tick(self): # run .send_clck_ind() every .ctr_interval - # be careful not to accumulate timing error when organizing the clock loop - ns = 1e-9 - us = 1e-6 - t_tick = int(self.ctr_interval // ns) - t_next = time.monotonic_ns() - while 1: - t_next += t_tick - t = time.monotonic_ns() - dt = (t_next - t) - if dt < 0: - log.warning("CLCKGen: time overrun by %dus; resetting the clock" % (dt * ns // us)) - t_next = time.monotonic_ns() - dt = 0 + # NOTE timerfd is careful not to accumulate timing error when organizing the clock loop - if self._breaker.wait(dt * ns): - break + _ = os.read(self._timerfd, 8) + assert len(_) == 8, len(_) + ticks = int.from_bytes(_, byteorder=sys.byteorder) + assert ticks > 0, ticks + if ticks > 1: + log.warning("CLCKGen: time overrun by %dus; resetting the clock" % ((ticks-1)*self._t_tick * ns // us)) + # (the kernel does clock reset by itself) - self.send_clck_ind() + self.send_clck_ind() def send_clck_ind(self): # We don't need to send so often @@ -135,6 +121,8 @@ # Just a wrapper for independent usage class Application(ApplicationBase): def __init__(self): + self.stop = False + # Print copyright self.app_print_copyright(APP_CR_HOLDERS) @@ -150,13 +138,16 @@ self.clck = CLCKGen([self.link], ind_period = 51) self.clck.start() - # Block unless we receive a signal - self.clck._thread.join() + while not self.stop: + self.clck.tick() + + self.clck.stop() + def sig_handler(self, signum, frame): log.info("Signal %d received" % signum) if signum == signal.SIGINT: - self.clck.stop() + self.stop = True if __name__ == '__main__': app = Application() diff --git a/src/target/trx_toolkit/fake_trx.py b/src/target/trx_toolkit/fake_trx.py index 711ad21..12d5b77 100755 --- a/src/target/trx_toolkit/fake_trx.py +++ b/src/target/trx_toolkit/fake_trx.py @@ -397,7 +397,7 @@ self.trx_list = TRXList() # Init shared clock generator - self.clck_gen = CLCKGen([], sched_rr_prio = None if self.argv.sched_rr_prio is None else self.argv.sched_rr_prio + 1) + self.clck_gen = CLCKGen([]) # This method will be called on each TDMA frame self.clck_gen.clck_handler = self.clck_handler @@ -459,7 +459,7 @@ log.error("Failed to set real time process scheduler to SCHED_RR, priority %u" % (self.argv.sched_rr_prio)) # Compose list of to be monitored sockets - sock_list = [] + sock_list = [self.clck_gen._timerfd] for trx in self.trx_list.trx_list: sock_list.append(trx.ctrl_if.sock) sock_list.append(trx.data_if.sock) @@ -469,6 +469,10 @@ # Wait until we get any data on any socket r_event, _, _ = select.select(sock_list, [], []) + # clock is priority + if self.clck_gen._timerfd in r_event: + self.clck_gen.tick() + # Iterate over all transceivers for trx in self.trx_list.trx_list: # DATA interface @@ -479,7 +483,7 @@ if trx.ctrl_if.sock in r_event: trx.ctrl_if.handle_rx() - # This method will be called by the clock thread + # This method will be called by the clock generator def clck_handler(self, fn): # We assume that this list is immutable at run-time for trx in self.trx_list.trx_list: diff --git a/src/target/trx_toolkit/test_clck_gen.py b/src/target/trx_toolkit/test_clck_gen.py index 92e5e55..3eaa4de 100644 --- a/src/target/trx_toolkit/test_clck_gen.py +++ b/src/target/trx_toolkit/test_clck_gen.py @@ -14,7 +14,6 @@ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -import threading import time import unittest @@ -32,23 +31,25 @@ ntick = 200 # ~ 1 s tick = 0 tstart = tend = None - done = threading.Event() + done = False def _(fn): - nonlocal tick, tstart, tend + nonlocal tick, tstart, tend, done if tick == 0: tstart = time.monotonic() if tick == ntick: tend = time.monotonic() - done.set() + done = True tick += 1 time.sleep(clck.ctr_interval / 2) clck.clck_handler = _ clck.start() try: - ok = done.wait(10) - self.assertTrue(ok, "clck_gen stuck") + t0 = time.monotonic() + while not done: + self.assertTrue(time.monotonic() - t0 < 10, "clck_gen stuck") + clck.tick() self.assertIsNotNone(tstart) self.assertIsNotNone(tend) -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40047?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: Iaa675c95059ec8ccfad667f69984d5a7f608c249 Gerrit-Change-Number: 40047 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

[M] Change in osmocom-bb[master]: trx_toolkit/clck_gen: Split it into clck_gen and _clck_gen modules

by kirr

kirr has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmocom-bb/+/40049?usp=email ) Change subject: trx_toolkit/clck_gen: Split it into clck_gen and _clck_gen modules ...................................................................... trx_toolkit/clck_gen: Split it into clck_gen and _clck_gen modules clck_gen will remain at Python while _clck_gen will be later converted to Cython for speed to avoid py-related overhead. This patch does only plain code movement as a preparatory step for that. Change-Id: If90f5b6718d6a24eda6221e52dc4626197f57356 --- A src/target/trx_toolkit/_clck_gen.py M src/target/trx_toolkit/clck_gen.py 2 files changed, 112 insertions(+), 91 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmocom-bb refs/changes/49/40049/1 diff --git a/src/target/trx_toolkit/_clck_gen.py b/src/target/trx_toolkit/_clck_gen.py new file mode 100644 index 0000000..9d9fd15 --- /dev/null +++ b/src/target/trx_toolkit/_clck_gen.py @@ -0,0 +1,111 @@ +# TRX Toolkit +# Simple TDMA frame clock generator +# +# (C) 2017-2019 by Vadim Yanitskiy <axilirator(a)gmail.com> +# +# All Rights Reserved +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +import logging as log +import time +import os +import sys + +from gsm_shared import * + + +ns = 1e-9 +us = 1e-6 + + +class CLCKGen: + # GSM TDMA definitions + SEC_DELAY_US = 1000 * 1000 + GSM_FRAME_US = 4615.0 + + def __init__(self, clck_links, clck_start = 0, ind_period = 102): + self.clck_links = clck_links + self.ind_period = ind_period + self.clck_start = clck_start + + # Calculate counter time + self.ctr_interval = self.GSM_FRAME_US + self.ctr_interval /= self.SEC_DELAY_US + self._t_tick = int(self.ctr_interval // ns) + + # Initialize timer fd + self._timerfd = os.timerfd_create(time.CLOCK_MONOTONIC) + + # (Optional) clock consumer + self.clck_handler = None + + def __del__(self): + os.close(self._timerfd) + + @property + def running(self): + t_next, _ = os.timerfd_gettime_ns(self._timerfd) + return (t_next != 0) + + def start(self): + # (Re)set the clock counter + self.clck_src = self.clck_start + + # start timer fd + os.timerfd_settime_ns(self._timerfd, initial=self._t_tick, interval=self._t_tick) + + def stop(self): + # stop timer fd + os.timerfd_settime_ns(self._timerfd, initial=0, interval=0) + + # tick must be called periodically by CLCKGen user. + # + # It waits for the next GSM frame to happen and emits corresponding clock indication at that time. + # It also runs attached .clck_handler if there is one. + # + # It is possible to use .tick in both blocking and non-blocking ways: + # + # - without extra care .tick will block waiting for the next GSM frame as explained above, + # - client code can also poll/select on ._timerfd to wait for GSM frame. + # After ._timerfd becomes ready it is guaranteed that the next .tick call will not block. + def tick(self): + # run .send_clck_ind() every .ctr_interval + # NOTE timerfd is careful not to accumulate timing error when organizing the clock loop + + _ = os.read(self._timerfd, 8) + assert len(_) == 8, len(_) + ticks = int.from_bytes(_, byteorder=sys.byteorder) + assert ticks > 0, ticks + if ticks > 1: + log.warning("CLCKGen: time overrun by %dus; resetting the clock" % ((ticks-1)*self._t_tick * ns // us)) + # (the kernel does clock reset by itself) + + self.send_clck_ind() + + def send_clck_ind(self): + # We don't need to send so often + if self.clck_src % self.ind_period == 0: + # Create UDP payload + payload = "IND CLOCK %u\0" % self.clck_src + + # Send indication to all UDP links + for link in self.clck_links: + link.send(payload) + + # Debug print + log.debug(payload.rstrip("\0")) + + if self.clck_handler is not None: + self.clck_handler(self.clck_src) + + # Increase frame count (modular arithmetic) + self.clck_src = (self.clck_src + 1) % GSM_HYPERFRAME diff --git a/src/target/trx_toolkit/clck_gen.py b/src/target/trx_toolkit/clck_gen.py index a223572..c40446b 100755 --- a/src/target/trx_toolkit/clck_gen.py +++ b/src/target/trx_toolkit/clck_gen.py @@ -21,103 +21,13 @@ APP_CR_HOLDERS = [("2017-2019", "Vadim Yanitskiy <axilirator(a)gmail.com>")] import logging as log -import time import signal -import os -import sys +from _clck_gen import CLCKGen from app_common import ApplicationBase from udp_link import UDPLink -from gsm_shared import * -ns = 1e-9 -us = 1e-6 - - -class CLCKGen: - # GSM TDMA definitions - SEC_DELAY_US = 1000 * 1000 - GSM_FRAME_US = 4615.0 - - def __init__(self, clck_links, clck_start = 0, ind_period = 102): - self.clck_links = clck_links - self.ind_period = ind_period - self.clck_start = clck_start - - # Calculate counter time - self.ctr_interval = self.GSM_FRAME_US - self.ctr_interval /= self.SEC_DELAY_US - self._t_tick = int(self.ctr_interval // ns) - - # Initialize timer fd - self._timerfd = os.timerfd_create(time.CLOCK_MONOTONIC) - - # (Optional) clock consumer - self.clck_handler = None - - def __del__(self): - os.close(self._timerfd) - - @property - def running(self): - t_next, _ = os.timerfd_gettime_ns(self._timerfd) - return (t_next != 0) - - def start(self): - # (Re)set the clock counter - self.clck_src = self.clck_start - - # start timer fd - os.timerfd_settime_ns(self._timerfd, initial=self._t_tick, interval=self._t_tick) - - def stop(self): - # stop timer fd - os.timerfd_settime_ns(self._timerfd, initial=0, interval=0) - - # tick must be called periodically by CLCKGen user. - # - # It waits for the next GSM frame to happen and emits corresponding clock indication at that time. - # It also runs attached .clck_handler if there is one. - # - # It is possible to use .tick in both blocking and non-blocking ways: - # - # - without extra care .tick will block waiting for the next GSM frame as explained above, - # - client code can also poll/select on ._timerfd to wait for GSM frame. - # After ._timerfd becomes ready it is guaranteed that the next .tick call will not block. - def tick(self): - # run .send_clck_ind() every .ctr_interval - # NOTE timerfd is careful not to accumulate timing error when organizing the clock loop - - _ = os.read(self._timerfd, 8) - assert len(_) == 8, len(_) - ticks = int.from_bytes(_, byteorder=sys.byteorder) - assert ticks > 0, ticks - if ticks > 1: - log.warning("CLCKGen: time overrun by %dus; resetting the clock" % ((ticks-1)*self._t_tick * ns // us)) - # (the kernel does clock reset by itself) - - self.send_clck_ind() - - def send_clck_ind(self): - # We don't need to send so often - if self.clck_src % self.ind_period == 0: - # Create UDP payload - payload = "IND CLOCK %u\0" % self.clck_src - - # Send indication to all UDP links - for link in self.clck_links: - link.send(payload) - - # Debug print - log.debug(payload.rstrip("\0")) - - if self.clck_handler is not None: - self.clck_handler(self.clck_src) - - # Increase frame count (modular arithmetic) - self.clck_src = (self.clck_src + 1) % GSM_HYPERFRAME - # Just a wrapper for independent usage class Application(ApplicationBase): def __init__(self): -- To view, visit https://gerrit.osmocom.org/c/osmocom-bb/+/40049?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: osmocom-bb Gerrit-Branch: master Gerrit-Change-Id: If90f5b6718d6a24eda6221e52dc4626197f57356 Gerrit-Change-Number: 40049 Gerrit-PatchSet: 1 Gerrit-Owner: kirr <kirr(a)nexedi.com> Gerrit-CC: fixeria <vyanitskiy(a)sysmocom.de> Gerrit-CC: osmith <osmith(a)sysmocom.de> Gerrit-CC: pespin <pespin(a)sysmocom.de>

4 months

1
0
0 0

2025

2024

2023

2022

gerrit-log April 2025