Hi,
the lua binding code was added to be able to automate OpenBSC tests. In theory we should be able to do this for SMS and UpdateLocation (call handling with MNCC exposing is left as a todo) but in practice we miss a piece of software to coordinate this and run the test. We miss it because it is an interesting problem but also I lost time on switching countries, learning new tricks at a project...
The basic testing structure looks easy as well. We want to define the number of concurrent subscribers (0, 10, 100, 1000, n) and to make it simple a single test (UL, send SMS, t) and execute the same test for each subscriber and call it a success if y% of tests succeed within time T. The way to measure this is easy as well. The lua script would print some data (e.g. the name of the ms) when it starts and completes.
For some degrees of freedom I don't have a good idea.. and feedback is welcome.
I am not sure if I should spawn, configure, add subscribers, a flavor of Osmocom cellular? I look into having some set of templates for the config, the stack to launch and in concept it looks awfully similar to something the GSM tester is doing. Shall we leave virtbts/cellular to the Osmocom tester and just focus on coordinating mobile? My feeling is to leave this to the Osmo GSM tester.
If we have n subscribers I would launch m copies of "mobile" (but run multiple MS in a single binary). So with 4 MS per mobile process and 10k subs we would end with 2.5k processes + many log messages coming from each. Would that scale with python? Should we look into doing this one in Go? Or can some of GSM tester be used (the template part)? I would probably design this concurrently with Go(besides being the first).
any ideas/comments?
holger
Hi Holger,
I never used virtbts but I think it should be quite easy adding support for it in osmo-gsm-tester. The only big issue I can think of is that we need to find a way to differentiate between virtbs and other bts since they use a different communication medium and then that means that the requested modems to run for the test with virtbts need to be selected accordingly (ie. don't allocate an ofono modem from the object pool if we are using a virbts). That can probably accomplished easily too, by adding new attributes to objects to describe that characteristic.
It can be a good idea to support this at some point if we want to test for compatibility between "mobile" app + virtbs and other BTS and modems, for instance see if we can place a call between a regular modem with a sysmobts and a "mobile" app + virtbts. Then we can make sure we don't add regressions in the future.
Regarding the performance, I imagine osmo-gsm-tester will have a hard time running that amount of objects/processes, as we didn't have this kind of scenarios in mind since it was developed having real hardware in mind. Furthermore, the unit running osmo-gsm-tester is not a big machine so I guess it would have problems running that amount of processes alone. I don't think the language is going to be that important, it depends more on how do you plan intend to manage all of those processes in a efficient way from system OS point of view.
Regards,
Hi Holger,
On Wed, Jan 10, 2018 at 12:24:47AM +0000, Holger Freyther wrote:
the lua binding code was added to be able to automate OpenBSC tests. In theory we should be able to do this for SMS and UpdateLocation (call handling with MNCC exposing is left as a todo) but in practice we miss a piece of software to coordinate this and run the test. We miss it because it is an interesting problem but also I lost time on switching countries, learning new tricks at a project...
Sure, I understand. However, it is definitely a part that we're very much looking forward to have :)
The basic testing structure looks easy as well. We want to define the number of concurrent subscribers (0, 10, 100, 1000, n) and to make it simple a single test (UL, send SMS, t) and execute the same test for each subscriber and call it a success if y% of tests succeed within time T. The way to measure this is easy as well. The lua script would print some data (e.g. the name of the ms) when it starts and completes.
One might also think of a more structured format to return the data, but that could always added later. One could e.g. print a XML or JSON snippet that's easier to parse/consume by whoever processes it.
What I also believe is very important is some kind of rate limiting / staggering when starting up. We know a single-BTS setup will for sure fail lots of LU if you stat 1k MS at the same time. So there should be some kind of provision to say something "start 1000 MS at a rate of 10 per second". I wouldn't go for more elaborate schemes, but simply a single linear rate/slope.
I am not sure if I should spawn, configure, add subscribers, a flavor of Osmocom cellular? I look into having some set of templates for the config, the stack to launch and in concept it looks awfully similar to something the GSM tester is doing. Shall we leave virtbts/cellular to the Osmocom tester and just focus on coordinating mobile? My feeling is to leave this to the Osmo GSM tester.
Yes, I think it's ok to focus on the "tester" side and not on the IUT (implementation under test) side. So we assume that the user will somehow bring up the [virtual] cellular network before excuting the load test. One preferred way of doing this is - I agree - by reusing those parts from osmo-gsm-tester.
If we have n subscribers I would launch m copies of "mobile" (but run multiple MS in a single binary).
I would argue the number of MS per 'mobile' should be configurable from 1-N.
So with 4 MS per mobile process and 10k subs we would end with 2.5k processes + many log messages coming from each.
The question is how many of those log messages do we need/want. In order to avoid the risk of 'mobile' blocking on writing to stdout/stderr, I think it would be best not to pipe that into other processes but write to files (could even be tmpfs!) and process the files after the run?
Would that scale with python? Should we look into doing this one in Go?
Or can some of GSM tester be used (the template part)?
I'm not sufficiently familiar with osmo-gsm-tester to say if we can use it. On an abstract level, I would think the "defining resources and generating configuration files" part should be reusable, but then it also just uses (jinja2?) templates that anyone can use in python. And whether it's sufficiently scalable to generate thousands of config files, I don't know either.
I would probably design this concurrently with Go(besides being the first).
I would suggest we keep not further the number of programming languages one needs to understand. But then, it's "just" a tool for load testing, so probably not that critical after all.
My naive assumption would be that starting 2.5k processes (and processing the SIGCHILD from python should be possible without causing a performance/scalability problem? As indicated, log file processing could be handled later, or one could configure stdio logging to be absolutely minimal (with verbose logs going to files)?
My attached test program (not using python 'subprocess' as I couldn't find a way to make it do non-blocking wait for the child to terminate) runs perfectly fine here, even without any rate limiting I get the following on my laptop:
$ time ./subproc.py 2018-01-10 12:44:14,811 INFO Beginning starting of processes 2018-01-10 12:44:15,603 INFO Started 2500 processes 2018-01-10 12:44:18,607 INFO Waited for all processes ./subproc.py 2.74s user 1.46s system 108% cpu 3.881 total
So 2500 processes could be forked in less than one second, and the starting/reaping in python needed onyl very few seconds of system time - compared with the amount of resources required to run the 'mobile' programs including the GSMTAP socket traffic etc. for sure neglectable?
Now of course '/bin/sleep' is a much simpler program to start, but the overhead of the python "orchestration" doesn't change with the resource footprint of the program started.
Just my thoughs, as usual. The decision is yours...
On 10.01.2018 12:55, Harald Welte wrote:
The question is how many of those log messages do we need/want. In order to avoid the risk of 'mobile' blocking on writing to stdout/stderr, I think it would be best not to pipe that into other processes but write to files (could even be tmpfs!) and process the files after the run?
Another option would be to change "log gsmtap" to not duplicate it to stdout/stderr and use it from mobile.
My attached test program (not using python 'subprocess' as I couldn't find a way to make it do non-blocking wait for the child to terminate) runs perfectly fine here
Alternative approach would be using asyncio module - one example can be found in osmo-python-tests/tests/test_py3.py
Just a suggestion though - have not tried either with thousands of processes.
On 10. Jan 2018, at 11:55, Harald Welte laforge@gnumonks.org wrote:
Hi Holger,
Sure, I understand. However, it is definitely a part that we're very much looking forward to have :)
me too... I dislike not having made progress here.
One might also think of a more structured format to return the data, but that could always added later. One could e.g. print a XML or JSON snippet that's easier to parse/consume by whoever processes it.
Good point. There is nothing that prevents us opening a file/socket/pipe from the lua code and writing out something. We would need to write out complete XML/JSON documents (and likely multiple of them) or define a framing ourselves..
What I also believe is very important is some kind of rate limiting / staggering when starting up. We know a single-BTS setup will for sure fail lots of LU if you stat 1k MS at the same time. So there should be some kind of provision to say something "start 1000 MS at a rate of 10 per second". I wouldn't go for more elaborate schemes, but simply a single linear rate/slope.
Makes sense.
Yes, I think it's ok to focus on the "tester" side and not on the IUT (implementation under test) side. So we assume that the user will somehow bring up the [virtual] cellular network before excuting the load test. One preferred way of doing this is - I agree - by reusing those parts from osmo-gsm-tester.
Good. I will leave this part.
Now of course '/bin/sleep' is a much simpler program to start, but the overhead of the python "orchestration" doesn't change with the resource footprint of the program started.
Thanks for looking at the process creation. I was concerned of Python's IO model to read from 2.5k FDs but apparently it has epoll. Computers are powerful and a single python process could be fine here...
okay. That reduces some degrees of the freedom. Where to put it? Into the OsmocomBB sources? Osmo GSM tester even if it might not share much code?
holger
Hi Holger,
On Wed, Jan 10, 2018 at 10:52:59PM +0000, Holger Freyther wrote:
okay. That reduces some degrees of the freedom. Where to put it? Into the OsmocomBB sources? Osmo GSM tester even if it might not share much code?
If you reuse osmo-gsm-tester code for templates or the like, then it should probably go there. If not, OsmocomBB seems like the more logical place. Would be great if in that case it is some kind of python library/module and a small 'main' wrapper around. This way the library could later be used by whatever is starting both the network and the 'mobile's.
On 10. Jan 2018, at 23:39, Harald Welte laforge@gnumonks.org wrote:
Hi!
If you reuse osmo-gsm-tester code for templates or the like, then it should probably go there. If not, OsmocomBB seems like the more logical place. Would be great if in that case it is some kind of python library/module and a small 'main' wrapper around. This way the library could later be used by whatever is starting both the network and the 'mobile's.
small progress update. I will use "luasocket" to open a af_unix dgram socket from the testcase, a small json library, the template, log and maybe the process code from osmo-gsm-tester. I will give Python's asyncio a try as well. It has unix socket and subprocess support.
Right now there a lot of loose ends but I should be able to make progress from here.
holger
On 21. Jan 2018, at 20:16, Holger Freyther holger@freyther.de wrote:
Hey!
some more progress. I had to learn a bit about asyncio and in the end will be forced to use a stream socket[1] and will end up using the IPA and add a reservation for JSON events. I will probably push my unfinished code to gerrit but it is too early to review.
holger
small progress update. I will use "luasocket" to open a af_unix dgram socket from the testcase, a small json library, the template, log and maybe the process code from osmo-gsm-tester. I will give Python's asyncio a try as well. It has unix socket and subprocess support.
Right now there a lot of loose ends but I should be able to make progress from here.
holger
[1] There is a asyncio.start_unix_server to create a AF_UNIX socket but only with SOCK_STREAM. The API has (thought not documented) a way to pass a file descriptor but when passing a socket they will check if it is a stream socket and if not fail. This is quite annoying in some ways.. First I want reliable delivery (excludes UDP) and have messages/datagrams (don't want to do an IPA style header..). Second thanks to the BSD socket API the accept will behave exactly the same for TCP, SCTP, UNIX, Bluetooth, etc and the python API allows to pass a socket factory.
Hi Holger,
thanks for your status update. Looking forward to the related code.
On 29. Jan 2018, at 10:50, Harald Welte laforge@gnumonks.org wrote:
Hi Holger,
thanks for your status update. Looking forward to the related code.
hehe. It is just a couple of hundred lines so far. It should bring us far enough. Will continue with it in a few minutes. For simplicity I am not segmenting mobile instances (one lua script for one mobile process).
holger
On 29. Jan 2018, at 21:52, Holger Freyther holger@freyther.de wrote:
Hey,
I wish I had better news. Instead of an end to end test I realize that picking "asyncio" was a grave mistake. Besides the lack AF_UNIX SOCK_DGRAM support, the process support is dangerous and doesn't scale. Today I stumbled into SIGCHLD handling of spawned processes. While normally neither virthphy/mobile would prematurely exit, a few exits could take the entire test down.
Speculating from the behavior of strace:
* SIGCHLD arrives * Something will be written into one end of a socketpair[1] * In the python code on wait(2) will be called on every registered process (https://github.com/python/cpython/blob/3.6/Lib/asyncio/unix_events.py#L819)
As more processes exit the main python code is still in the for loop and the buffer runs full. It also means that the main application will be busy executing Xk syscalls instead of launching processes or events.
It is unfortunate that I only notice now but probably still better than having to deal mysterious failures of processes not starting and all tests failing. As not many people here know Go. I think I will stay with python but use the lower level epoll API directly. So we end up with a single "select" system... I should finally have something during the week.
I hope all of you had a nice fosdem!
holger
[1] A common trick to notify one thread or part of the application of an event. There is not much that is allowed in a signal handler so using a buffer is a reasonable approach.. It should forward the PID of the process that exited though.
On 4. Feb 2018, at 22:35, Holger Freyther holger@freyther.de wrote:
Hi,
- SIGCHLD arrives
- Something will be written into one end of a socketpair[1]
- In the python code on wait(2) will be called on every registered process
(https://github.com/python/cpython/blob/3.6/Lib/asyncio/unix_events.py#L819)
there is a way to switch to a child reaper that is using wait(-1.. but even then we have two issues.
1.) If I use await asyncio.create_subprocess_exec... then we can't tick with 50ms. On a 60s schedule to start 10k processes there is a delay of +60s.
2.) If I delay the process creation then I can almost keep the schedule most of the time but we run into buffering exceptions and/or not all tasks are started within a reasonable amount of time.
Knowing what I know now I would have gone for Go. It is made for the kind of concurrency where python seems unusable. I hesitated to build my own event loop but will build something that looks like our C apps in python (single select, no async/await).
holger
On 7. Feb 2018, at 09:31, Holger Freyther holger@freyther.de wrote:
Hey,
Knowing what I know now I would have gone for Go. It is made for the kind of concurrency where python seems unusable. I hesitated to build my own event loop but will build something that looks like our C apps in python (single select, no async/await).
finally... still plenty of things to improve and do... The next thing is to improve "mobile" to not add the >30s timeout for first cell selection.
python3 -mosmo_ms_driver 22:59:56.255200 --- -: Going to store files in {tmp_dir='/tmp/tmp5wbisu2hosmo-ms-driver'} 22:59:56.256180 run lu_test: Starting testcase 22:59:56.256813 run lu_test: Pre-launching all virtphy's 22:59:56.257391 run osmo-ms/00000: Starting virtphy process [lu_test↪osmo-ms/00000] 22:59:56.258454 run osmo-ms/00000: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00000 [lu_test↪osmo-ms/00000] 22:59:56.262654 run osmo-ms/00001: Starting virtphy process [lu_test↪osmo-ms/00001] 22:59:56.262793 run osmo-ms/00001: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00001 [lu_test↪osmo-ms/00001] 22:59:56.269087 run osmo-ms/00002: Starting virtphy process [lu_test↪osmo-ms/00002] 22:59:56.269241 run osmo-ms/00002: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00002 [lu_test↪osmo-ms/00002] 22:59:56.274869 run osmo-ms/00003: Starting virtphy process [lu_test↪osmo-ms/00003] 22:59:56.275031 run osmo-ms/00003: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00003 [lu_test↪osmo-ms/00003] 22:59:56.280416 run osmo-ms/00004: Starting virtphy process [lu_test↪osmo-ms/00004] 22:59:56.280570 run osmo-ms/00004: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00004 [lu_test↪osmo-ms/00004] 22:59:56.285699 run osmo-ms/00005: Starting virtphy process [lu_test↪osmo-ms/00005] 22:59:56.285809 run osmo-ms/00005: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00005 [lu_test↪osmo-ms/00005] 22:59:56.291308 run osmo-ms/00006: Starting virtphy process [lu_test↪osmo-ms/00006] 22:59:56.291488 run osmo-ms/00006: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00006 [lu_test↪osmo-ms/00006] 22:59:56.300046 run osmo-ms/00007: Starting virtphy process [lu_test↪osmo-ms/00007] 22:59:56.300352 run osmo-ms/00007: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00007 [lu_test↪osmo-ms/00007] 22:59:56.306980 run osmo-ms/00008: Starting virtphy process [lu_test↪osmo-ms/00008] 22:59:56.307134 run osmo-ms/00008: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00008 [lu_test↪osmo-ms/00008] 22:59:56.312373 run osmo-ms/00009: Starting virtphy process [lu_test↪osmo-ms/00009] 22:59:56.312526 run osmo-ms/00009: virtphy --l1ctl-sock=/tmp/tmp5wbisu2hosmo-ms-driver/osmocom_l2_00009 [lu_test↪osmo-ms/00009] 22:59:56.319714 run lu_test: Checking if sockets are in the filesystem 22:59:56.539503 run osmo-ms/00000: Starting process [lu_test↪osmo-ms/00000] 22:59:56.539977 run osmo-ms/00000: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00000.cfg --vty-port=0 [lu_test↪osmo-ms/00000] 22:59:56.607714 run lu_test: MS start registered {at=282743.473796635, delay=0.06289598299190402, ms=00000} 23:00:09.926047 run osmo-ms/00001: Starting process [lu_test↪osmo-ms/00001] 23:00:09.926563 run osmo-ms/00001: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00001.cfg --vty-port=0 [lu_test↪osmo-ms/00001] 23:00:09.994826 run lu_test: MS start registered {at=282756.860905894, delay=0.06362233200343326, ms=00001} 23:00:15.424151 run osmo-ms/00002: Starting process [lu_test↪osmo-ms/00002] 23:00:15.425052 run osmo-ms/00002: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00002.cfg --vty-port=0 [lu_test↪osmo-ms/00002] 23:00:15.488956 run lu_test: MS start registered {at=282762.355019533, delay=0.05711088899988681, ms=00002} 23:00:19.724246 run osmo-ms/00003: Starting process [lu_test↪osmo-ms/00003] 23:00:19.724996 run osmo-ms/00003: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00003.cfg --vty-port=0 [lu_test↪osmo-ms/00003] 23:00:19.785684 run lu_test: MS start registered {at=282766.651760042, delay=0.05416889599291608, ms=00003} 23:00:23.325812 run osmo-ms/00004: Starting process [lu_test↪osmo-ms/00004] 23:00:23.326760 run osmo-ms/00004: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00004.cfg --vty-port=0 [lu_test↪osmo-ms/00004] 23:00:23.393305 run lu_test: MS start registered {at=282770.259381962, delay=0.06159330299124122, ms=00004} 23:00:26.427752 run osmo-ms/00005: Starting process [lu_test↪osmo-ms/00005] 23:00:26.428336 run osmo-ms/00005: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00005.cfg --vty-port=0 [lu_test↪osmo-ms/00005] 23:00:26.494263 run lu_test: MS start registered {at=282773.360336693, delay=0.059708705986849964, ms=00005} 23:00:29.627894 run osmo-ms/00006: Starting process [lu_test↪osmo-ms/00006] 23:00:29.628553 run osmo-ms/00006: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00006.cfg --vty-port=0 [lu_test↪osmo-ms/00006] 23:00:29.683842 run lu_test: MS start registered {at=282776.54991861, delay=0.04898471699561924, ms=00006} 23:00:33.226115 run osmo-ms/00007: Starting process [lu_test↪osmo-ms/00007] 23:00:33.226723 run osmo-ms/00007: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00007.cfg --vty-port=0 [lu_test↪osmo-ms/00007] 23:00:33.288238 run lu_test: MS start registered {at=282780.154311069, delay=0.055712109024170786, ms=00007} 23:00:34.052813 run lu_test: MS performed LU {at=282780.918918862, lu_delay=37.44512222701451, ms=00000} 23:00:37.525812 run osmo-ms/00008: Starting process [lu_test↪osmo-ms/00008] 23:00:37.526370 run osmo-ms/00008: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00008.cfg --vty-port=0 [lu_test↪osmo-ms/00008] 23:00:37.590594 run lu_test: MS start registered {at=282784.456672404, delay=0.05611523304833099, ms=00008} 23:00:43.028214 run osmo-ms/00009: Starting process [lu_test↪osmo-ms/00009] 23:00:43.028898 run osmo-ms/00009: mobile -c /tmp/tmp5wbisu2hosmo-ms-driver/mob_00009.cfg --vty-port=0 [lu_test↪osmo-ms/00009] 23:00:43.087549 run lu_test: MS start registered {at=282789.95362104, delay=0.05322436196729541, ms=00009} 23:00:43.087727 run lu_test: All started... {duration=46.56737156602321, too_slow=0} 23:00:47.231257 run lu_test: MS performed LU {at=282794.097388755, lu_delay=37.23648286098614, ms=00001} 23:00:52.878843 run lu_test: MS performed LU {at=282799.744935023, lu_delay=37.38991548999911, ms=00002} 23:00:56.661687 run lu_test: MS performed LU {at=282803.527769994, lu_delay=36.87600995198591, ms=00003} 23:01:00.411273 run lu_test: MS performed LU {at=282807.277505226, lu_delay=37.01812326401705, ms=00004} 23:01:04.194030 run lu_test: MS performed LU {at=282811.06022426, lu_delay=37.699887567025144, ms=00005} 23:01:06.060154 run lu_test: MS performed LU {at=282812.926287126, lu_delay=36.37636851600837, ms=00006} 23:01:09.847484 run lu_test: MS performed LU {at=282816.713489132, lu_delay=36.559178063005675, ms=00007} 23:01:15.475567 run lu_test: MS performed LU {at=282822.341696909, lu_delay=37.8850245049689, ms=00008} 23:01:21.124497 run lu_test: MS performed LU {at=282827.990630504, lu_delay=38.037009463994764, ms=00009} 23:01:56.525619 run lu_test: Tests done {all_completed=True, max=38.037009463994764, min=36.37636851600837}
-
Harald Welte -
Holger Freyther -
Max -
Pau Espin Pedrol