10 Jan
2018
10 Jan
'18
1:24 a.m.
Hi,
the lua binding code was added to be able to automate OpenBSC tests. In theory we should
be able to do this for SMS and UpdateLocation (call handling with MNCC exposing is left as
a todo) but in practice we miss a piece of software to coordinate this and run the test.
We miss it because it is an interesting problem but also I lost time on switching
countries, learning new tricks at a project...
The basic testing structure looks easy as well. We want to define the number of concurrent
subscribers (0, 10, 100, 1000, n) and to make it simple a single test (UL, send SMS, t)
and execute the same test for each subscriber and call it a success if y% of tests succeed
within time T. The way to measure this is easy as well. The lua script would print some
data (e.g. the name of the ms) when it starts and completes.
For some degrees of freedom I don't have a good idea.. and feedback is welcome.
I am not sure if I should spawn, configure, add subscribers, a flavor of Osmocom cellular?
I look into having some set of templates for the config, the stack to launch and in
concept it looks awfully similar to something the GSM tester is doing. Shall we leave
virtbts/cellular to the Osmocom tester and just focus on coordinating mobile? My feeling
is to leave this to the Osmo GSM tester.
If we have n subscribers I would launch m copies of "mobile" (but run multiple
MS in a single binary). So with 4 MS per mobile process and 10k subs we would end with
2.5k processes + many log messages coming from each. Would that scale with python? Should
we look into doing this one in Go? Or can some of GSM tester be used (the template part)?
I would probably design this concurrently with Go(besides being the first).
any ideas/comments?
holger
10 Jan
10 Jan
10:50 a.m.
Hi Holger,
I never used virtbts but I think it should be quite easy adding support
for it in osmo-gsm-tester. The only big issue I can think of is that we
need to find a way to differentiate between virtbs and other bts since
they use a different communication medium and then that means that the
requested modems to run for the test with virtbts need to be selected
accordingly (ie. don't allocate an ofono modem from the object pool if
we are using a virbts). That can probably accomplished easily too, by
adding new attributes to objects to describe that characteristic.
It can be a good idea to support this at some point if we want to test
for compatibility between "mobile" app + virtbs and other BTS and
modems, for instance see if we can place a call between a regular modem
with a sysmobts and a "mobile" app + virtbts. Then we can make sure we
don't add regressions in the future.
Regarding the performance, I imagine osmo-gsm-tester will have a hard
time running that amount of objects/processes, as we didn't have this
kind of scenarios in mind since it was developed having real hardware in
mind. Furthermore, the unit running osmo-gsm-tester is not a big machine
so I guess it would have problems running that amount of processes
alone. I don't think the language is going to be that important, it
depends more on how do you plan intend to manage all of those processes
in a efficient way from system OS point of view.
Regards,
--
- Pau Espin Pedrol <pespin(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
12:55 p.m.
Hi Holger,
On Wed, Jan 10, 2018 at 12:24:47AM +0000, Holger Freyther wrote:
the lua binding code was added to be able to automate
OpenBSC tests.
In theory we should be able to do this for SMS and UpdateLocation
(call handling with MNCC exposing is left as a todo) but in practice
we miss a piece of software to coordinate this and run the test. We
miss it because it is an interesting problem but also I lost time on
switching countries, learning new tricks at a project...
Sure, I understand. However, it is definitely a part that we're very
much looking forward to have :)
The basic testing structure looks easy as well. We
want to define the
number of concurrent subscribers (0, 10, 100, 1000, n) and to make it
simple a single test (UL, send SMS, t) and execute the same test for
each subscriber and call it a success if y% of tests succeed within
time T. The way to measure this is easy as well. The lua script would
print some data (e.g. the name of the ms) when it starts and
completes.
One might also think of a more structured format to return the data, but
that could always added later. One could e.g. print a XML or JSON
snippet that's easier to parse/consume by whoever processes it.
What I also believe is very important is some kind of rate limiting /
staggering when starting up. We know a single-BTS setup will for sure
fail lots of LU if you stat 1k MS at the same time. So there should be
some kind of provision to say something "start 1000 MS at a rate of 10
per second". I wouldn't go for more elaborate schemes, but simply a
single linear rate/slope.
I am not sure if I should spawn, configure, add
subscribers, a flavor
of Osmocom cellular? I look into having some set of templates for the
config, the stack to launch and in concept it looks awfully similar to
something the GSM tester is doing. Shall we leave virtbts/cellular to
the Osmocom tester and just focus on coordinating mobile? My feeling
is to leave this to the Osmo GSM tester.
Yes, I think it's ok to focus on the "tester" side and not on the IUT
(implementation under test) side. So we assume that the user will
somehow bring up the [virtual] cellular network before excuting the load
test. One preferred way of doing this is - I agree - by reusing those
parts from osmo-gsm-tester.
If we have n subscribers I would launch m copies of
"mobile" (but run
multiple MS in a single binary).
I would argue the number of MS per 'mobile' should be configurable from
1-N.
So with 4 MS per mobile process and 10k subs we would
end with 2.5k
processes + many log messages coming from each.
The question is how many of those log messages do we need/want. In
order to avoid the risk of 'mobile' blocking on writing to
stdout/stderr, I think it would be best not to pipe that into other
processes but write to files (could even be tmpfs!) and process the
files after the run?
Would that scale with python? Should we look into
doing this one in
Go?
Or can some of GSM tester be used (the template part)?
I'm not sufficiently familiar with osmo-gsm-tester to say if we can use
it. On an abstract level, I would think the "defining resources and
generating configuration files" part should be reusable, but then it
also just uses (jinja2?) templates that anyone can use in python. And
whether it's sufficiently scalable to generate thousands of config
files, I don't know either.
I would probably design this concurrently with
Go(besides being the
first).
I would suggest we keep not further the number of programming languages
one needs to understand. But then, it's "just" a tool for load testing,
so probably not that critical after all.
My naive assumption would be that starting 2.5k processes (and
processing the SIGCHILD from python should be possible without causing
a performance/scalability problem? As indicated, log file processing
could be handled later, or one could configure stdio logging to be
absolutely minimal (with verbose logs going to files)?
My attached test program (not using python 'subprocess' as I couldn't find
a way to make it do non-blocking wait for the child to terminate) runs
perfectly fine here, even without any rate limiting I get the following
on my laptop:
$ time ./subproc.py
2018-01-10 12:44:14,811 INFO Beginning starting of processes
2018-01-10 12:44:15,603 INFO Started 2500 processes
2018-01-10 12:44:18,607 INFO Waited for all processes
./subproc.py 2.74s user 1.46s system 108% cpu 3.881 total
So 2500 processes could be forked in less than one second, and the
starting/reaping in python needed onyl very few seconds of system time -
compared with the amount of resources required to run the 'mobile'
programs including the GSMTAP socket traffic etc. for sure neglectable?
Now of course '/bin/sleep' is a much simpler program to start, but the
overhead of the python "orchestration" doesn't change with the resource
footprint of the program started.
Just my thoughs, as usual. The decision is yours...
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
3:13 p.m.
On 10.01.2018 12:55, Harald Welte wrote:
The question is how many of those log messages do we
need/want. In
order to avoid the risk of 'mobile' blocking on writing to
stdout/stderr, I think it would be best not to pipe that into other
processes but write to files (could even be tmpfs!) and process the
files after the run?
Another option would be to change "log gsmtap" to not duplicate it to
stdout/stderr
and use it from mobile.
My attached test program (not using python 'subprocess' as I couldn't find
a way to make it do non-blocking wait for the child to terminate) runs
perfectly fine here
Alternative approach would be using asyncio module - one example can be found in
osmo-python-tests/tests/test_py3.py
Just a suggestion though - have not tried either with thousands of processes.
--
Max Suraev <msuraev(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
11:52 p.m.
On 10. Jan 2018, at 11:55, Harald Welte
<laforge(a)gnumonks.org> wrote:
Hi Holger,
Sure, I understand. However, it is definitely a part
that we're very
much looking forward to have :)
me too... I dislike not having made progress here.
One might also think of a more structured format to
return the data, but
that could always added later. One could e.g. print a XML or JSON
snippet that's easier to parse/consume by whoever processes it.
Good point. There is nothing that prevents us opening a file/socket/pipe
from the lua code and writing out something. We would need to write out
complete XML/JSON documents (and likely multiple of them) or define a
framing ourselves..
What I also believe is very important is some kind of
rate limiting /
staggering when starting up. We know a single-BTS setup will for sure
fail lots of LU if you stat 1k MS at the same time. So there should be
some kind of provision to say something "start 1000 MS at a rate of 10
per second". I wouldn't go for more elaborate schemes, but simply a
single linear rate/slope.
Makes sense.
Yes, I think it's ok to focus on the
"tester" side and not on the IUT
(implementation under test) side. So we assume that the user will
somehow bring up the [virtual] cellular network before excuting the load
test. One preferred way of doing this is - I agree - by reusing those
parts from osmo-gsm-tester.
Good. I will leave this part.
Now of course '/bin/sleep' is a much simpler
program to start, but the
overhead of the python "orchestration" doesn't change with the resource
footprint of the program started.
Thanks for looking at the process creation. I was concerned of Python's
IO model to read from 2.5k FDs but apparently it has epoll. Computers are
powerful and a single python process could be fine here...
okay. That reduces some degrees of the freedom. Where to put it? Into the
OsmocomBB sources? Osmo GSM tester even if it might not share much code?
holger
11 Jan
11 Jan
12:39 a.m.
Hi Holger,
On Wed, Jan 10, 2018 at 10:52:59PM +0000, Holger Freyther wrote:
okay. That reduces some degrees of the freedom. Where
to put it? Into the
OsmocomBB sources? Osmo GSM tester even if it might not share much code?
If you reuse osmo-gsm-tester code for templates or the like, then it should
probably go there. If not, OsmocomBB seems like the more logical place.
Would be great if in that case it is some kind of python library/module
and a small 'main' wrapper around. This way the library could later be
used by whatever is starting both the network and the 'mobile's.
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
21 Jan
21 Jan
9:16 p.m.
On 10. Jan 2018, at 23:39, Harald Welte
<laforge(a)gnumonks.org> wrote:
Hi!
If you reuse osmo-gsm-tester code for templates or the
like, then it should
probably go there. If not, OsmocomBB seems like the more logical place.
Would be great if in that case it is some kind of python library/module
and a small 'main' wrapper around. This way the library could later be
used by whatever is starting both the network and the 'mobile's.
small progress update. I will use "luasocket" to open a af_unix dgram socket
from the testcase, a small json library, the template, log and maybe the
process code from osmo-gsm-tester. I will give Python's asyncio a try as
well. It has unix socket and subprocess support.
Right now there a lot of loose ends but I should be able to make progress
from here.
holger
28 Jan
28 Jan
11:41 p.m.
On 21. Jan 2018, at 20:16, Holger Freyther
<holger(a)freyther.de> wrote:
Hey!
some more progress. I had to learn a bit about asyncio and in the end will
be forced to use a stream socket[1] and will end up using the IPA and add
a reservation for JSON events. I will probably push my unfinished code to
gerrit but it is too early to review.
holger
small progress update. I will use
"luasocket" to open a af_unix dgram socket
from the testcase, a small json library, the template, log and maybe the
process code from osmo-gsm-tester. I will give Python's asyncio a try as
well. It has unix socket and subprocess support.
Right now there a lot of loose ends but I should be able to make progress
from here.
holger
[1] There is a asyncio.start_unix_server to create a AF_UNIX socket but
only with SOCK_STREAM. The API has (thought not documented) a way to pass
a file descriptor but when passing a socket they will check if it is a
stream socket and if not fail. This is quite annoying in some ways.. First
I want reliable delivery (excludes UDP) and have messages/datagrams (don't
want to do an IPA style header..). Second thanks to the BSD socket API the
accept will behave exactly the same for TCP, SCTP, UNIX, Bluetooth, etc
and the python API allows to pass a socket factory.
29 Jan
29 Jan
11:50 a.m.
Hi Holger,
thanks for your status update. Looking forward to the related code.
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
10:52 p.m.
On 29. Jan 2018, at 10:50, Harald Welte
<laforge(a)gnumonks.org> wrote:
Hi Holger,
thanks for your status update. Looking forward to the related code.
hehe. It is just a couple of hundred lines so far. It should bring us far enough. Will
continue with it in a few minutes. For simplicity I am not segmenting mobile instances
(one lua script for one mobile process).
holger
4 Feb
4 Feb
11:35 p.m.
On 29. Jan 2018, at 21:52, Holger Freyther
<holger(a)freyther.de> wrote:
Hey,
I wish I had better news. Instead of an end to end test I realize that picking
"asyncio" was a grave mistake. Besides the lack AF_UNIX SOCK_DGRAM support, the
process support is dangerous and doesn't scale. Today I stumbled into SIGCHLD handling
of spawned processes. While normally neither virthphy/mobile would prematurely exit, a few
exits could take the entire test down.
Speculating from the behavior of strace:
* SIGCHLD arrives
* Something will be written into one end of a socketpair[1]
* In the python code on wait(2) will be called on every registered process
(https://github.com/python/cpython/blob/3.6/Lib/asyncio/unix_events.py#L819)
As more processes exit the main python code is still in the for loop and the buffer runs
full. It also means that the main application will be busy executing Xk syscalls instead
of launching processes or events.
It is unfortunate that I only notice now but probably still better than having to deal
mysterious failures of processes not starting and all tests failing. As not many people
here know Go. I think I will stay with python but use the lower level epoll API directly.
So we end up with a single "select" system... I should finally have something
during the week.
I hope all of you had a nice fosdem!
holger
[1] A common trick to notify one thread or part of the application of an event. There is
not much that is allowed in a signal handler so using a buffer is a reasonable approach..
It should forward the PID of the process that exited though.
7 Feb
7 Feb
10:31 a.m.
On 4. Feb 2018, at 22:35, Holger Freyther
<holger(a)freyther.de> wrote:
Hi,
* SIGCHLD arrives
* Something will be written into one end of a socketpair[1]
* In the python code on wait(2) will be called on every registered process
(https://github.com/python/cpython/blob/3.6/Lib/asyncio/unix_events.py#L819)
there is a way to switch to a child reaper that is using wait(-1.. but even
then we have two issues.
1.) If I use await asyncio.create_subprocess_exec... then we can't tick
with 50ms. On a 60s schedule to start 10k processes there is a delay of +60s.
2.) If I delay the process creation then I can almost keep the schedule most
of the time but we run into buffering exceptions and/or not all tasks are
started within a reasonable amount of time.
Knowing what I know now I would have gone for Go. It is made for the kind of
concurrency where python seems unusable. I hesitated to build my own event
loop but will build something that looks like our C apps in python (single
select, no async/await).
holger
25 Feb
25 Feb
11:08 p.m.
On 7. Feb 2018, at 09:31, Holger Freyther
<holger(a)freyther.de> wrote:
Hey,
Knowing what I know now I would have gone for Go. It
is made for the kind of
concurrency where python seems unusable. I hesitated to build my own event
loop but will build something that looks like our C apps in python (single
select, no async/await).
finally... still plenty of things to improve and do... The next thing is to
improve "mobile" to not add the >30s timeout for first cell selection.
python3 -mosmo_ms_driver
22:59:56.255200 --- -: Going to store files in
{tmp_dir='/tmp/tmp5wbisu2hosmo-ms-driver'}
22:59:56.256180 run lu_test: Starting testcase
22:59:56.256813 run lu_test: Pre-launching all virtphy's
22:59:56.257391 run osmo-ms/00000: Starting virtphy process
[lu_test↪osmo-ms/00000]
22:59:56.258454 run osmo-ms/00000: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00000 [lu_test↪osmo-ms/00000]
22:59:56.262654 run osmo-ms/00001: Starting virtphy process
[lu_test↪osmo-ms/00001]
22:59:56.262793 run osmo-ms/00001: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00001 [lu_test↪osmo-ms/00001]
22:59:56.269087 run osmo-ms/00002: Starting virtphy process
[lu_test↪osmo-ms/00002]
22:59:56.269241 run osmo-ms/00002: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00002 [lu_test↪osmo-ms/00002]
22:59:56.274869 run osmo-ms/00003: Starting virtphy process
[lu_test↪osmo-ms/00003]
22:59:56.275031 run osmo-ms/00003: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00003 [lu_test↪osmo-ms/00003]
22:59:56.280416 run osmo-ms/00004: Starting virtphy process
[lu_test↪osmo-ms/00004]
22:59:56.280570 run osmo-ms/00004: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00004 [lu_test↪osmo-ms/00004]
22:59:56.285699 run osmo-ms/00005: Starting virtphy process
[lu_test↪osmo-ms/00005]
22:59:56.285809 run osmo-ms/00005: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00005 [lu_test↪osmo-ms/00005]
22:59:56.291308 run osmo-ms/00006: Starting virtphy process
[lu_test↪osmo-ms/00006]
22:59:56.291488 run osmo-ms/00006: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00006 [lu_test↪osmo-ms/00006]
22:59:56.300046 run osmo-ms/00007: Starting virtphy process
[lu_test↪osmo-ms/00007]
22:59:56.300352 run osmo-ms/00007: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00007 [lu_test↪osmo-ms/00007]
22:59:56.306980 run osmo-ms/00008: Starting virtphy process
[lu_test↪osmo-ms/00008]
22:59:56.307134 run osmo-ms/00008: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00008 [lu_test↪osmo-ms/00008]
22:59:56.312373 run osmo-ms/00009: Starting virtphy process
[lu_test↪osmo-ms/00009]
22:59:56.312526 run osmo-ms/00009: virtphy
--l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00009 [lu_test↪osmo-ms/00009]
22:59:56.319714 run lu_test: Checking if sockets are in the
filesystem
22:59:56.539503 run osmo-ms/00000: Starting process
[lu_test↪osmo-ms/00000]
22:59:56.539977 run osmo-ms/00000: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00000.cfg --vty-port=0 [lu_test↪osmo-ms/00000]
22:59:56.607714 run lu_test: MS start registered
{at=282743.473796635, delay=0.06289598299190402, ms=00000}
23:00:09.926047 run osmo-ms/00001: Starting process
[lu_test↪osmo-ms/00001]
23:00:09.926563 run osmo-ms/00001: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00001.cfg --vty-port=0 [lu_test↪osmo-ms/00001]
23:00:09.994826 run lu_test: MS start registered
{at=282756.860905894, delay=0.06362233200343326, ms=00001}
23:00:15.424151 run osmo-ms/00002: Starting process
[lu_test↪osmo-ms/00002]
23:00:15.425052 run osmo-ms/00002: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00002.cfg --vty-port=0 [lu_test↪osmo-ms/00002]
23:00:15.488956 run lu_test: MS start registered
{at=282762.355019533, delay=0.05711088899988681, ms=00002}
23:00:19.724246 run osmo-ms/00003: Starting process
[lu_test↪osmo-ms/00003]
23:00:19.724996 run osmo-ms/00003: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00003.cfg --vty-port=0 [lu_test↪osmo-ms/00003]
23:00:19.785684 run lu_test: MS start registered
{at=282766.651760042, delay=0.05416889599291608, ms=00003}
23:00:23.325812 run osmo-ms/00004: Starting process
[lu_test↪osmo-ms/00004]
23:00:23.326760 run osmo-ms/00004: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00004.cfg --vty-port=0 [lu_test↪osmo-ms/00004]
23:00:23.393305 run lu_test: MS start registered
{at=282770.259381962, delay=0.06159330299124122, ms=00004}
23:00:26.427752 run osmo-ms/00005: Starting process
[lu_test↪osmo-ms/00005]
23:00:26.428336 run osmo-ms/00005: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00005.cfg --vty-port=0 [lu_test↪osmo-ms/00005]
23:00:26.494263 run lu_test: MS start registered
{at=282773.360336693, delay=0.059708705986849964, ms=00005}
23:00:29.627894 run osmo-ms/00006: Starting process
[lu_test↪osmo-ms/00006]
23:00:29.628553 run osmo-ms/00006: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00006.cfg --vty-port=0 [lu_test↪osmo-ms/00006]
23:00:29.683842 run lu_test: MS start registered
{at=282776.54991861, delay=0.04898471699561924, ms=00006}
23:00:33.226115 run osmo-ms/00007: Starting process
[lu_test↪osmo-ms/00007]
23:00:33.226723 run osmo-ms/00007: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00007.cfg --vty-port=0 [lu_test↪osmo-ms/00007]
23:00:33.288238 run lu_test: MS start registered
{at=282780.154311069, delay=0.055712109024170786, ms=00007}
23:00:34.052813 run lu_test: MS performed LU
{at=282780.918918862, lu_delay=37.44512222701451, ms=00000}
23:00:37.525812 run osmo-ms/00008: Starting process
[lu_test↪osmo-ms/00008]
23:00:37.526370 run osmo-ms/00008: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00008.cfg --vty-port=0 [lu_test↪osmo-ms/00008]
23:00:37.590594 run lu_test: MS start registered
{at=282784.456672404, delay=0.05611523304833099, ms=00008}
23:00:43.028214 run osmo-ms/00009: Starting process
[lu_test↪osmo-ms/00009]
23:00:43.028898 run osmo-ms/00009: mobile -c
/tmp/tmp5wbisu2hosmo-ms-driver/mob_00009.cfg --vty-port=0 [lu_test↪osmo-ms/00009]
23:00:43.087549 run lu_test: MS start registered
{at=282789.95362104, delay=0.05322436196729541, ms=00009}
23:00:43.087727 run lu_test: All started...
{duration=46.56737156602321, too_slow=0}
23:00:47.231257 run lu_test: MS performed LU
{at=282794.097388755, lu_delay=37.23648286098614, ms=00001}
23:00:52.878843 run lu_test: MS performed LU
{at=282799.744935023, lu_delay=37.38991548999911, ms=00002}
23:00:56.661687 run lu_test: MS performed LU
{at=282803.527769994, lu_delay=36.87600995198591, ms=00003}
23:01:00.411273 run lu_test: MS performed LU
{at=282807.277505226, lu_delay=37.01812326401705, ms=00004}
23:01:04.194030 run lu_test: MS performed LU
{at=282811.06022426, lu_delay=37.699887567025144, ms=00005}
23:01:06.060154 run lu_test: MS performed LU
{at=282812.926287126, lu_delay=36.37636851600837, ms=00006}
23:01:09.847484 run lu_test: MS performed LU
{at=282816.713489132, lu_delay=36.559178063005675, ms=00007}
23:01:15.475567 run lu_test: MS performed LU
{at=282822.341696909, lu_delay=37.8850245049689, ms=00008}
23:01:21.124497 run lu_test: MS performed LU
{at=282827.990630504, lu_delay=38.037009463994764, ms=00009}
23:01:56.525619 run lu_test: Tests done {all_completed=True,
max=38.037009463994764, min=36.37636851600837}
2673
days inactive
2719
days old
12 comments
4 participants
participants (4)
-
Harald Welte -
Holger Freyther -
Max -
Pau Espin Pedrol