Approach to system testing for Osmocom stack
Hi, the lua binding code was added to be able to automate OpenBSC tests. In theory we should be able to do this for SMS and UpdateLocation (call handling with MNCC exposing is left as a todo) but in practice we miss a piece of software to coordinate this and run the test. We miss it because it is an interesting problem but also I lost time on switching countries, learning new tricks at a project... The basic testing structure looks easy as well. We want to define the number of concurrent subscribers (0, 10, 100, 1000, n) and to make it simple a single test (UL, send SMS, t) and execute the same test for each subscriber and call it a success if y% of tests succeed within time T. The way to measure this is easy as well. The lua script would print some data (e.g. the name of the ms) when it starts and completes. For some degrees of freedom I don't have a good idea.. and feedback is welcome. I am not sure if I should spawn, configure, add subscribers, a flavor of Osmocom cellular? I look into having some set of templates for the config, the stack to launch and in concept it looks awfully similar to something the GSM tester is doing. Shall we leave virtbts/cellular to the Osmocom tester and just focus on coordinating mobile? My feeling is to leave this to the Osmo GSM tester. If we have n subscribers I would launch m copies of "mobile" (but run multiple MS in a single binary). So with 4 MS per mobile process and 10k subs we would end with 2.5k processes + many log messages coming from each. Would that scale with python? Should we look into doing this one in Go? Or can some of GSM tester be used (the template part)? I would probably design this concurrently with Go(besides being the first). any ideas/comments? holger
Hi Holger, I never used virtbts but I think it should be quite easy adding support for it in osmo-gsm-tester. The only big issue I can think of is that we need to find a way to differentiate between virtbs and other bts since they use a different communication medium and then that means that the requested modems to run for the test with virtbts need to be selected accordingly (ie. don't allocate an ofono modem from the object pool if we are using a virbts). That can probably accomplished easily too, by adding new attributes to objects to describe that characteristic. It can be a good idea to support this at some point if we want to test for compatibility between "mobile" app + virtbs and other BTS and modems, for instance see if we can place a call between a regular modem with a sysmobts and a "mobile" app + virtbts. Then we can make sure we don't add regressions in the future. Regarding the performance, I imagine osmo-gsm-tester will have a hard time running that amount of objects/processes, as we didn't have this kind of scenarios in mind since it was developed having real hardware in mind. Furthermore, the unit running osmo-gsm-tester is not a big machine so I guess it would have problems running that amount of processes alone. I don't think the language is going to be that important, it depends more on how do you plan intend to manage all of those processes in a efficient way from system OS point of view. Regards, -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
On 10. Jan 2018, at 09:49, Pau Espin Pedrol <pespin@sysmocom.de> wrote:
Hey Pau,
I never used virtbts but I think it should be quite easy adding support for it in osmo-gsm-tester. The only big issue I can think of is that we need to find a way to differentiate between virtbs and other bts since they use a different communication medium and then that means that the requested modems to run for the test with virtbts need to be selected accordingly (ie. don't allocate an ofono modem from the object pool if we are using a virbts). That can probably accomplished easily too, by adding new attributes to objects to describe that characteristic.
It can be a good idea to support this at some point if we want to test for compatibility between "mobile" app + virtbs and other BTS and modems, for instance see if we can place a call between a regular modem with a sysmobts and a "mobile" app + virtbts. Then we can make sure we don't add regressions in the future.
I think I have finally a routine that gives me some time to move this forward. We should have the ability for bi-directional communication between mobile and the test driver and can start implementing the mobile as a non-ofono "modem". From a mobile lua/point of view: The mobile will start, register over unix dgram to a server. At this point the server can send commands (as JSON) and the lua script can dispatch them (e.g. on/off, cell selection, send sms, place call) Can you help me to see what needs to be done from a osmo-gsm-tester point of view? What amount of work is necessary? The benefit might be to have a quick(er) test on Jenkins. regards holger
Hi Holger Since I wrote my last e-mail in this thread, we developed some more stuff for osmo-gsm-tester which may be interesting for this topic. I recently introduced some code to be able to run TTCN3 tests using real HW and a motorola c213 phone running osmocom-bb, since we originally only run TTCN3 BTS tests using osmo-bts-virtual with virtphy, see [1] for more information. In order to drive all HW and software processes, I am using osmo-gsm-tester with specific config and testsuite which you can find in osmo-gsm-teser.git/ttcn/. You can find the jenkins job running tests with that setup in [2]. The setup is running mostly fine but I didn't find time yet to look at the list of tests still failing (which doesn't fail with the regular virt TTCN3 setup). So, as a result, it means we now support using a osmocom-bb capable phone using real RF (wired) communicating against already available osmo-bts such as osmo-bts-sysmo, osmo-bts-trx, nanobts, etc., so no need to add support for osmo-bts-virt and also a way to differentiate networks that are not compatible (for instance, you don't want to pick a mobile running through virtphy and connecting it to osmo-bts-trx right?). Current working setup for TTCN3 implies having a motorola phone automatically powered on when the corresponding serial device is opened (using [3]). So osmo-gsm-tester starts and controls the "osmocon" application which opens and uses the serial device. (You can find it in ./src/osmo_gsm_tester/osmocon.py). In The TTCN3 testcase, we then connect TTCN3 code to the L1CTL iface provided by "osmocon", but in your case what you want to do I guess is connecting the "mobile" application to it. So what I think it's required in your case: * Move src/osmo_gsm_tester/modem.py to src/osmo_gsm_tester/modem_ofono.py (and class Modem to class ModemOfono * To make code more easy to follow and extend, create an abstract class "Modem" (see class Bts in src/osmo_gsm_tester/bts.py), and make ModemOfono inheit from it. * Create a new class ModemOsmocomLuaMobile (or similar) in a new file src/osmo_gsm_tester/modem_osmocom_mobile.py and inherit from modem.py:Modem. In there implement all required methods used by Modem interface users. To integrate it with the rest of the system you need to: ** Create a Modem "type" system, like we do for BTS (see resources.conf.prod how it's used), and add a method similar to suite.py:bts_obj() which creates a subclass based on that type, then use it in suite.py:modem() (similar to suite.py:bts()). ** Add any new required extra config parameters for this type of modems in resource.py:RESOURCES_SCHEMA ** I think to make it compatible with ofono modems, you should start both "mobile" and "osmocon" processes in the implementation of your Modem class, since the user of the modem class in the tests doesn't need to set up a osmocon process based on the type of modem (we do similar stuff with osmo-trx in bts_osmotrx.py). I think that's mostly it. Don't hesitate to ping me here or in IRC if you have any question or require some help. [1] https://osmocom.org/issues/3155 [2] https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester... [3] https://wiki.cuvoodoo.info/doku.php?id=osmotoserial -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
Hi Holger, I never used virtbts but I think it should be quite easy adding support for it in osmo-gsm-tester. The only big issue I can think of is that we need to find a way to differentiate between virtbs and other bts since they use a different communication medium and then that means that the requested modems to run for the test with virtbts need to be selected accordingly (ie. don't allocate an ofono modem from the object pool if we are using a virbts). That can probably accomplished easily too, by adding new attributes to objects to describe that characteristic. It can be a good idea to support this at some point if we want to test for compatibility between "mobile" app + virtbs and other BTS and modems, for instance see if we can place a call between a regular modem with a sysmobts and a "mobile" app + virtbts. Then we can make sure we don't add regressions in the future. Regarding the performance, I imagine osmo-gsm-tester will have a hard time running that amount of objects/processes, as we didn't have this kind of scenarios in mind since it was developed having real hardware in mind. Furthermore, the unit running osmo-gsm-tester is not a big machine so I guess it would have problems running that amount of processes alone. I don't think the language is going to be that important, it depends more on how do you plan intend to manage all of those processes in a efficient way from system OS point of view. Regards, -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
Hi Holger, On Wed, Jan 10, 2018 at 12:24:47AM +0000, Holger Freyther wrote:
the lua binding code was added to be able to automate OpenBSC tests. In theory we should be able to do this for SMS and UpdateLocation (call handling with MNCC exposing is left as a todo) but in practice we miss a piece of software to coordinate this and run the test. We miss it because it is an interesting problem but also I lost time on switching countries, learning new tricks at a project...
Sure, I understand. However, it is definitely a part that we're very much looking forward to have :)
The basic testing structure looks easy as well. We want to define the number of concurrent subscribers (0, 10, 100, 1000, n) and to make it simple a single test (UL, send SMS, t) and execute the same test for each subscriber and call it a success if y% of tests succeed within time T. The way to measure this is easy as well. The lua script would print some data (e.g. the name of the ms) when it starts and completes.
One might also think of a more structured format to return the data, but that could always added later. One could e.g. print a XML or JSON snippet that's easier to parse/consume by whoever processes it. What I also believe is very important is some kind of rate limiting / staggering when starting up. We know a single-BTS setup will for sure fail lots of LU if you stat 1k MS at the same time. So there should be some kind of provision to say something "start 1000 MS at a rate of 10 per second". I wouldn't go for more elaborate schemes, but simply a single linear rate/slope.
I am not sure if I should spawn, configure, add subscribers, a flavor of Osmocom cellular? I look into having some set of templates for the config, the stack to launch and in concept it looks awfully similar to something the GSM tester is doing. Shall we leave virtbts/cellular to the Osmocom tester and just focus on coordinating mobile? My feeling is to leave this to the Osmo GSM tester.
Yes, I think it's ok to focus on the "tester" side and not on the IUT (implementation under test) side. So we assume that the user will somehow bring up the [virtual] cellular network before excuting the load test. One preferred way of doing this is - I agree - by reusing those parts from osmo-gsm-tester.
If we have n subscribers I would launch m copies of "mobile" (but run multiple MS in a single binary).
I would argue the number of MS per 'mobile' should be configurable from 1-N.
So with 4 MS per mobile process and 10k subs we would end with 2.5k processes + many log messages coming from each.
The question is how many of those log messages do we need/want. In order to avoid the risk of 'mobile' blocking on writing to stdout/stderr, I think it would be best not to pipe that into other processes but write to files (could even be tmpfs!) and process the files after the run?
Would that scale with python? Should we look into doing this one in Go?
Or can some of GSM tester be used (the template part)?
I'm not sufficiently familiar with osmo-gsm-tester to say if we can use it. On an abstract level, I would think the "defining resources and generating configuration files" part should be reusable, but then it also just uses (jinja2?) templates that anyone can use in python. And whether it's sufficiently scalable to generate thousands of config files, I don't know either.
I would probably design this concurrently with Go(besides being the first).
I would suggest we keep not further the number of programming languages one needs to understand. But then, it's "just" a tool for load testing, so probably not that critical after all. My naive assumption would be that starting 2.5k processes (and processing the SIGCHILD from python should be possible without causing a performance/scalability problem? As indicated, log file processing could be handled later, or one could configure stdio logging to be absolutely minimal (with verbose logs going to files)? My attached test program (not using python 'subprocess' as I couldn't find a way to make it do non-blocking wait for the child to terminate) runs perfectly fine here, even without any rate limiting I get the following on my laptop: $ time ./subproc.py 2018-01-10 12:44:14,811 INFO Beginning starting of processes 2018-01-10 12:44:15,603 INFO Started 2500 processes 2018-01-10 12:44:18,607 INFO Waited for all processes ./subproc.py 2.74s user 1.46s system 108% cpu 3.881 total So 2500 processes could be forked in less than one second, and the starting/reaping in python needed onyl very few seconds of system time - compared with the amount of resources required to run the 'mobile' programs including the GSMTAP socket traffic etc. for sure neglectable? Now of course '/bin/sleep' is a much simpler program to start, but the overhead of the python "orchestration" doesn't change with the resource footprint of the program started. Just my thoughs, as usual. The decision is yours... -- - Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
On 10. Jan 2018, at 11:55, Harald Welte <laforge@gnumonks.org> wrote:
Hi Holger,
Sure, I understand. However, it is definitely a part that we're very much looking forward to have :)
me too... I dislike not having made progress here.
One might also think of a more structured format to return the data, but that could always added later. One could e.g. print a XML or JSON snippet that's easier to parse/consume by whoever processes it.
Good point. There is nothing that prevents us opening a file/socket/pipe from the lua code and writing out something. We would need to write out complete XML/JSON documents (and likely multiple of them) or define a framing ourselves..
What I also believe is very important is some kind of rate limiting / staggering when starting up. We know a single-BTS setup will for sure fail lots of LU if you stat 1k MS at the same time. So there should be some kind of provision to say something "start 1000 MS at a rate of 10 per second". I wouldn't go for more elaborate schemes, but simply a single linear rate/slope.
Makes sense.
Yes, I think it's ok to focus on the "tester" side and not on the IUT (implementation under test) side. So we assume that the user will somehow bring up the [virtual] cellular network before excuting the load test. One preferred way of doing this is - I agree - by reusing those parts from osmo-gsm-tester.
Good. I will leave this part.
Now of course '/bin/sleep' is a much simpler program to start, but the overhead of the python "orchestration" doesn't change with the resource footprint of the program started.
Thanks for looking at the process creation. I was concerned of Python's IO model to read from 2.5k FDs but apparently it has epoll. Computers are powerful and a single python process could be fine here... okay. That reduces some degrees of the freedom. Where to put it? Into the OsmocomBB sources? Osmo GSM tester even if it might not share much code? holger
Hi Holger, On Wed, Jan 10, 2018 at 10:52:59PM +0000, Holger Freyther wrote:
okay. That reduces some degrees of the freedom. Where to put it? Into the OsmocomBB sources? Osmo GSM tester even if it might not share much code?
If you reuse osmo-gsm-tester code for templates or the like, then it should probably go there. If not, OsmocomBB seems like the more logical place. Would be great if in that case it is some kind of python library/module and a small 'main' wrapper around. This way the library could later be used by whatever is starting both the network and the 'mobile's. -- - Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
On 10. Jan 2018, at 23:39, Harald Welte <laforge@gnumonks.org> wrote:
Hi!
If you reuse osmo-gsm-tester code for templates or the like, then it should probably go there. If not, OsmocomBB seems like the more logical place. Would be great if in that case it is some kind of python library/module and a small 'main' wrapper around. This way the library could later be used by whatever is starting both the network and the 'mobile's.
small progress update. I will use "luasocket" to open a af_unix dgram socket from the testcase, a small json library, the template, log and maybe the process code from osmo-gsm-tester. I will give Python's asyncio a try as well. It has unix socket and subprocess support. Right now there a lot of loose ends but I should be able to make progress from here. holger
On 21. Jan 2018, at 20:16, Holger Freyther <holger@freyther.de> wrote:
Hey! some more progress. I had to learn a bit about asyncio and in the end will be forced to use a stream socket[1] and will end up using the IPA and add a reservation for JSON events. I will probably push my unfinished code to gerrit but it is too early to review. holger
small progress update. I will use "luasocket" to open a af_unix dgram socket from the testcase, a small json library, the template, log and maybe the process code from osmo-gsm-tester. I will give Python's asyncio a try as well. It has unix socket and subprocess support.
Right now there a lot of loose ends but I should be able to make progress from here.
holger
[1] There is a asyncio.start_unix_server to create a AF_UNIX socket but only with SOCK_STREAM. The API has (thought not documented) a way to pass a file descriptor but when passing a socket they will check if it is a stream socket and if not fail. This is quite annoying in some ways.. First I want reliable delivery (excludes UDP) and have messages/datagrams (don't want to do an IPA style header..). Second thanks to the BSD socket API the accept will behave exactly the same for TCP, SCTP, UNIX, Bluetooth, etc and the python API allows to pass a socket factory.
Hi Holger, thanks for your status update. Looking forward to the related code. -- - Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
On 29. Jan 2018, at 10:50, Harald Welte <laforge@gnumonks.org> wrote:
Hi Holger,
thanks for your status update. Looking forward to the related code.
hehe. It is just a couple of hundred lines so far. It should bring us far enough. Will continue with it in a few minutes. For simplicity I am not segmenting mobile instances (one lua script for one mobile process). holger
On 29. Jan 2018, at 21:52, Holger Freyther <holger@freyther.de> wrote:
Hey, I wish I had better news. Instead of an end to end test I realize that picking "asyncio" was a grave mistake. Besides the lack AF_UNIX SOCK_DGRAM support, the process support is dangerous and doesn't scale. Today I stumbled into SIGCHLD handling of spawned processes. While normally neither virthphy/mobile would prematurely exit, a few exits could take the entire test down. Speculating from the behavior of strace: * SIGCHLD arrives * Something will be written into one end of a socketpair[1] * In the python code on wait(2) will be called on every registered process (https://github.com/python/cpython/blob/3.6/Lib/asyncio/unix_events.py#L819) As more processes exit the main python code is still in the for loop and the buffer runs full. It also means that the main application will be busy executing Xk syscalls instead of launching processes or events. It is unfortunate that I only notice now but probably still better than having to deal mysterious failures of processes not starting and all tests failing. As not many people here know Go. I think I will stay with python but use the lower level epoll API directly. So we end up with a single "select" system... I should finally have something during the week. I hope all of you had a nice fosdem! holger [1] A common trick to notify one thread or part of the application of an event. There is not much that is allowed in a signal handler so using a buffer is a reasonable approach.. It should forward the PID of the process that exited though.
On 4. Feb 2018, at 22:35, Holger Freyther <holger@freyther.de> wrote:
Hi,
* SIGCHLD arrives * Something will be written into one end of a socketpair[1] * In the python code on wait(2) will be called on every registered process (https://github.com/python/cpython/blob/3.6/Lib/asyncio/unix_events.py#L819)
there is a way to switch to a child reaper that is using wait(-1.. but even then we have two issues. 1.) If I use await asyncio.create_subprocess_exec... then we can't tick with 50ms. On a 60s schedule to start 10k processes there is a delay of +60s. 2.) If I delay the process creation then I can almost keep the schedule most of the time but we run into buffering exceptions and/or not all tasks are started within a reasonable amount of time. Knowing what I know now I would have gone for Go. It is made for the kind of concurrency where python seems unusable. I hesitated to build my own event loop but will build something that looks like our C apps in python (single select, no async/await). holger
On 7. Feb 2018, at 09:31, Holger Freyther <holger@freyther.de> wrote:
Hey,
Knowing what I know now I would have gone for Go. It is made for the kind of concurrency where python seems unusable. I hesitated to build my own event loop but will build something that looks like our C apps in python (single select, no async/await).
finally... still plenty of things to improve and do... The next thing is to improve "mobile" to not add the >30s timeout for first cell selection. python3 -mosmo_ms_driver 22:59:56.255200 --- -: Going to store files in {tmp_dir='/tmp/tmp5wbisu2hosmo-ms-driver'} 22:59:56.256180 run lu_test: Starting testcase 22:59:56.256813 run lu_test: Pre-launching all virtphy's 22:59:56.257391 run osmo-ms/00000: Starting virtphy process [lu_test↪osmo-ms/00000] 22:59:56.258454 run osmo-ms/00000: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00000 [lu_test↪osmo-ms/00000] 22:59:56.262654 run osmo-ms/00001: Starting virtphy process [lu_test↪osmo-ms/00001] 22:59:56.262793 run osmo-ms/00001: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00001 [lu_test↪osmo-ms/00001] 22:59:56.269087 run osmo-ms/00002: Starting virtphy process [lu_test↪osmo-ms/00002] 22:59:56.269241 run osmo-ms/00002: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00002 [lu_test↪osmo-ms/00002] 22:59:56.274869 run osmo-ms/00003: Starting virtphy process [lu_test↪osmo-ms/00003] 22:59:56.275031 run osmo-ms/00003: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00003 [lu_test↪osmo-ms/00003] 22:59:56.280416 run osmo-ms/00004: Starting virtphy process [lu_test↪osmo-ms/00004] 22:59:56.280570 run osmo-ms/00004: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00004 [lu_test↪osmo-ms/00004] 22:59:56.285699 run osmo-ms/00005: Starting virtphy process [lu_test↪osmo-ms/00005] 22:59:56.285809 run osmo-ms/00005: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00005 [lu_test↪osmo-ms/00005] 22:59:56.291308 run osmo-ms/00006: Starting virtphy process [lu_test↪osmo-ms/00006] 22:59:56.291488 run osmo-ms/00006: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00006 [lu_test↪osmo-ms/00006] 22:59:56.300046 run osmo-ms/00007: Starting virtphy process [lu_test↪osmo-ms/00007] 22:59:56.300352 run osmo-ms/00007: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00007 [lu_test↪osmo-ms/00007] 22:59:56.306980 run osmo-ms/00008: Starting virtphy process [lu_test↪osmo-ms/00008] 22:59:56.307134 run osmo-ms/00008: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00008 [lu_test↪osmo-ms/00008] 22:59:56.312373 run osmo-ms/00009: Starting virtphy process [lu_test↪osmo-ms/00009] 22:59:56.312526 run osmo-ms/00009: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00009 [lu_test↪osmo-ms/00009] 22:59:56.319714 run lu_test: Checking if sockets are in the filesystem 22:59:56.539503 run osmo-ms/00000: Starting process [lu_test↪osmo-ms/00000] 22:59:56.539977 run osmo-ms/00000: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00000.cfg --vty-port=0 [lu_test↪osmo-ms/00000] 22:59:56.607714 run lu_test: MS start registered {at=282743.473796635, delay=0.06289598299190402, ms=00000} 23:00:09.926047 run osmo-ms/00001: Starting process [lu_test↪osmo-ms/00001] 23:00:09.926563 run osmo-ms/00001: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00001.cfg --vty-port=0 [lu_test↪osmo-ms/00001] 23:00:09.994826 run lu_test: MS start registered {at=282756.860905894, delay=0.06362233200343326, ms=00001} 23:00:15.424151 run osmo-ms/00002: Starting process [lu_test↪osmo-ms/00002] 23:00:15.425052 run osmo-ms/00002: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00002.cfg --vty-port=0 [lu_test↪osmo-ms/00002] 23:00:15.488956 run lu_test: MS start registered {at=282762.355019533, delay=0.05711088899988681, ms=00002} 23:00:19.724246 run osmo-ms/00003: Starting process [lu_test↪osmo-ms/00003] 23:00:19.724996 run osmo-ms/00003: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00003.cfg --vty-port=0 [lu_test↪osmo-ms/00003] 23:00:19.785684 run lu_test: MS start registered {at=282766.651760042, delay=0.05416889599291608, ms=00003} 23:00:23.325812 run osmo-ms/00004: Starting process [lu_test↪osmo-ms/00004] 23:00:23.326760 run osmo-ms/00004: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00004.cfg --vty-port=0 [lu_test↪osmo-ms/00004] 23:00:23.393305 run lu_test: MS start registered {at=282770.259381962, delay=0.06159330299124122, ms=00004} 23:00:26.427752 run osmo-ms/00005: Starting process [lu_test↪osmo-ms/00005] 23:00:26.428336 run osmo-ms/00005: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00005.cfg --vty-port=0 [lu_test↪osmo-ms/00005] 23:00:26.494263 run lu_test: MS start registered {at=282773.360336693, delay=0.059708705986849964, ms=00005} 23:00:29.627894 run osmo-ms/00006: Starting process [lu_test↪osmo-ms/00006] 23:00:29.628553 run osmo-ms/00006: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00006.cfg --vty-port=0 [lu_test↪osmo-ms/00006] 23:00:29.683842 run lu_test: MS start registered {at=282776.54991861, delay=0.04898471699561924, ms=00006} 23:00:33.226115 run osmo-ms/00007: Starting process [lu_test↪osmo-ms/00007] 23:00:33.226723 run osmo-ms/00007: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00007.cfg --vty-port=0 [lu_test↪osmo-ms/00007] 23:00:33.288238 run lu_test: MS start registered {at=282780.154311069, delay=0.055712109024170786, ms=00007} 23:00:34.052813 run lu_test: MS performed LU {at=282780.918918862, lu_delay=37.44512222701451, ms=00000} 23:00:37.525812 run osmo-ms/00008: Starting process [lu_test↪osmo-ms/00008] 23:00:37.526370 run osmo-ms/00008: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00008.cfg --vty-port=0 [lu_test↪osmo-ms/00008] 23:00:37.590594 run lu_test: MS start registered {at=282784.456672404, delay=0.05611523304833099, ms=00008} 23:00:43.028214 run osmo-ms/00009: Starting process [lu_test↪osmo-ms/00009] 23:00:43.028898 run osmo-ms/00009: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00009.cfg --vty-port=0 [lu_test↪osmo-ms/00009] 23:00:43.087549 run lu_test: MS start registered {at=282789.95362104, delay=0.05322436196729541, ms=00009} 23:00:43.087727 run lu_test: All started... {duration=46.56737156602321, too_slow=0} 23:00:47.231257 run lu_test: MS performed LU {at=282794.097388755, lu_delay=37.23648286098614, ms=00001} 23:00:52.878843 run lu_test: MS performed LU {at=282799.744935023, lu_delay=37.38991548999911, ms=00002} 23:00:56.661687 run lu_test: MS performed LU {at=282803.527769994, lu_delay=36.87600995198591, ms=00003} 23:01:00.411273 run lu_test: MS performed LU {at=282807.277505226, lu_delay=37.01812326401705, ms=00004} 23:01:04.194030 run lu_test: MS performed LU {at=282811.06022426, lu_delay=37.699887567025144, ms=00005} 23:01:06.060154 run lu_test: MS performed LU {at=282812.926287126, lu_delay=36.37636851600837, ms=00006} 23:01:09.847484 run lu_test: MS performed LU {at=282816.713489132, lu_delay=36.559178063005675, ms=00007} 23:01:15.475567 run lu_test: MS performed LU {at=282822.341696909, lu_delay=37.8850245049689, ms=00008} 23:01:21.124497 run lu_test: MS performed LU {at=282827.990630504, lu_delay=38.037009463994764, ms=00009} 23:01:56.525619 run lu_test: Tests done {all_completed=True, max=38.037009463994764, min=36.37636851600837}
participants (3)
-
Harald Welte -
Holger Freyther -
Pau Espin Pedrol