Dear all,
for some time, I've been wanting to have an easy way to run m3ua-testtool and sua-testtool as part of our regular regression testing setup.
After some very disappointing trials using docker (see my blog), I managed to make this work using the 'unshare' program (part of util-linux).
The key regquirement basically is: * create a network namespace * configure some netdevice / ip adresses in it * run the software under test (osmo-stp in this case) * run the test suite * tear down everything
Initially I tried to do this using 'ip netns', but the problem is that you normally require root permissions to configure the netdevice inside the new namespace. And being root is of course difficult if you think of a jenkins build slave.
However, 'unshare' offers the ability to map your real UID to root inside the namespace. At that point, everything can be done as non-root.
The resulting scripts are in libosmo-sccp/contrib/test: "run-in-ns.sh" is a simple wrapper around unshare, so you have to call something like ./run-in-ns.sh ./test-m3ua.sh
In order to use the m3ua-testtool, you will need to * install guile * clone + build + install https://github.com/nplab/guile-sctp * clone + build m3ua-testtool and sua-testtool from git.osmocom.org, no installation of this require.
I would appreciate feedback from others trying to reproduce the above setup on their machine[s]. All tests should be GREEN/PASS.
If anyone knowns how to integrate this with jenkins for libosmo-sccp,I think it would be great to have this as part of our testing of gerrit patches.
I'm not so sure if it makes sense to try to integrate this with "make check", as it is not a unit test (and it would only work on Linux).
I'm planning to use the saee "run-in-ns.sh" approach for my SCCP level tests implemented in TTCN-3/Titan.
One further idea is if it make sense to report the test reuslts in a more friendly way. Right now it's deailed osmo-stp logs interspersed with output from the tester. Mayb there's some more information we can feed back into jenkins? Again, I have very little understanding of jenkins.
Regards, Harald
Hi all,
I've tried to run the tests on my machine, but didn't succeed and need some help to do so.
The resulting scripts are in libosmo-sccp/contrib/test: "run-in-ns.sh" is a simple wrapper around unshare, so you have to call something like ./run-in-ns.sh ./test-m3ua.sh
I could not find mentioned script in current master branch (872c6b2a8e309ca6739ef295f1fc468c189e4ec9) [1].
clone + build m3ua-testtool and sua-testtool from git.osmocom.org, no installation of this require.
I could not find {m3ua,su}-testtool on git.osmocom.org [2][3], so I cloned them from https://github.com/nplab/. Is this correct?
One further idea is if it make sense to report the test reuslts in a more friendly way. Right now it's deailed osmo-stp logs interspersed with output from the tester. Mayb there's some more information we can feed back into jenkins? Again, I have very little understanding of jenkins.
Once I had to visual PowerShell unit tests in Jenkins and back then we converted the output/testlog to a JUnit XML report as these reports are beautiful visualised. Moreover those reports would introduce the third build status "UNSTABLE" in case of a failed test.
I will try to create such testlog_2_xml_report script to visualise osmocom testlog and will let you know about my try.
Regards, André
[1] http://git.osmocom.org/libosmo-sccp/tree/contrib?id=872c6b2a8e309ca6739ef295... [2] http://git.osmocom.org/?q=sua-testtool [3] http://git.osmocom.org/?q=m3ua-testtool
Hi Andre,
thanks for looking into this!
On Wed, May 17, 2017 at 11:47:55AM +0200, André Boddenberg wrote:
Hi all,
I've tried to run the tests on my machine, but didn't succeed and need some help to do so.
The resulting scripts are in libosmo-sccp/contrib/test: "run-in-ns.sh" is a simple wrapper around unshare, so you have to call something like ./run-in-ns.sh ./test-m3ua.sh
I could not find mentioned script in current master branch (872c6b2a8e309ca6739ef295f1fc468c189e4ec9) [1].
thanks for pointing this out. I forgot to submit https://gerrit.osmocom.org/#/c/2611/ - which has just been done.
no installation of this require.
I could not find {m3ua,su}-testtool on git.osmocom.org [2][3],
they are not yet listed on the index, but you should be able to clone them via git://git.osmocom.org/nplab/m3ua-testtool and git://git.osmocom.org/nplab/sua-testtool already. I just tried, works for me [tm].
so I cloned them from https://github.com/nplab/. Is this correct?
no, this is not correct, sorry.
I will try to create such testlog_2_xml_report script to visualise osmocom testlog and will let you know about my try.
ok. For the m3ua/sua testtool, the 'runm3uatest' program already contains some custom colorized output, you might simply be able to patch that to generate something useful to jenkins?
Hi Harald,
they are not yet listed on the index, but you should be able to clone them via git://git.osmocom.org/nplab/m3ua-testtool and git://git.osmocom.org/nplab/sua-testtool already. I just tried, works for me [tm].
Thanks, works for me too. Now I can see the test output, although tests are failing. Probably "culprit" is line 29 in test-m3ua.sh:
Line29 => $STP_DIR/osmo-stp -c $STP_CONFIG &
Error=> ./test-m3ua.sh: 29: ./test-m3ua.sh: ../..//stp/osmo-stp: not found
Can you please help me to fix this issue?
ok. For the m3ua/sua testtool, the 'runm3uatest' program already contains some custom colorized output, you might simply be able to patch that to generate something useful to jenkins?
That was my first approach, but I didn't manage to pipe the colorized output to a file. Any ideas?
So I (temporarily) changed following line in run-all-sgp-tests.sh:
$ (./runm3uatest -d . -t $timeout $testcase > /dev/tty) >& /dev/null
to
$ ./runm3uatest -d . -t $timeout $testcase
Then executing
$ ./run-in-ns.sh ./test-m3ua.sh 2>&1 | tee m3ua-test-report.plain
gives a very verbose output, which is saved to a "m3ua-test-report.plain" for XML generation after all tests have been run. At this point I got the feeling that a more verbose output could also be more helpful in a Jenkins job than prompting the summary, which is probably more suited for builds run by a developer.(?)
What is your general opinion about the verboseness of the Jenkins log and the XML report?
Here is how a XML report looks like when we would simply evaluate the "test output summary":
<testsuites> <testsuite name="" tests="32" failures="32" errors="32"/> <testcase name="m3ua-sgp-aspsm-v-002"/> <testcase name="m3ua-sgp-aspsm-v-001"/> </testsuite> </testsuites>
This XML snippet represent the current state after my hands on yesterday:
<testsuites> <testsuite name="" tests="32" failures="32" errors="32"/> <testcase name="m3ua-sgp-aspsm-v-001" time="51011250"/> <error type=" In procedure sctp-sendmsg">
Some deprecated features have been used. Set the environment variable GUILE_WARN_DEPRECATED to "detailed" and rerun the program to get more information. Set it to "no" to suppress this message. FAILED </error> <testcase name="m3ua-sgp-aspsm-v-003" time="58336593"/> ...... </testsuite> </testsuites>
I personally always prefer when Jenkins provides some information about the failure rather than only pointing out that something failed and referring to the console log.
Something else, can I may request a failing test or information how to change one to let it fail? :) Without such test case it's hard to test the script, which shall produce the XML report.
Regards, André
Hi Andre,
On Thu, May 18, 2017 at 11:46:52AM +0200, André Boddenberg wrote:
they are not yet listed on the index, but you should be able to clone them via git://git.osmocom.org/nplab/m3ua-testtool and git://git.osmocom.org/nplab/sua-testtool already. I just tried, works for me [tm].
Thanks, works for me too. Now I can see the test output, although tests are failing. Probably "culprit" is line 29 in test-m3ua.sh:
Line29 => $STP_DIR/osmo-stp -c $STP_CONFIG &
Error=> ./test-m3ua.sh: 29: ./test-m3ua.sh: ../..//stp/osmo-stp: not found
Can you please help me to fix this issue?
you are not starting the script from within contrib/test, or didn't build osmo-stp as part of libosmo-sccp?
What is your general opinion about the verboseness of the Jenkins log and the XML report?
I think it would be useful to have a summay at first sight (at top of the log?) and then the details at the end or after another click?
I personally always prefer when Jenkins provides some information about the failure rather than only pointing out that something failed and referring to the console log.
yes, the question is how can we get both the summary and the detailed log.
Something else, can I may request a failing test or information how to change one to let it fail? :)
Just change the configuration in the m3ua-param-testtool.scm i.e. to a different IP address or port number ('tester-addr', 'tester-port'; will make all tests fail) or to a different point code ('tester-pc'; will make some tests fail)
Hi Harald,
Thanks for your input on "how to let tests fail"!
Before speaking about the publishing of XML reports on Jenkins, I'd like to make a small infrastructure-detour after reading your blog post.
Long story short, here's a Dockerfile (attached) capable of running tests on top of a compiled (via ./contrib/jenkins.sh) libosmo-sctp repository. Only one container is used and IPs can be created on loopback interface in fact of using "--add-cap=NET_ADMIN", which imo is a mighty feature - able of pushing docker beyond its proposed use cases.
After building the Dockerfile execute following "line" inside compiled libosmo-sctp repo:
docker run -it --rm --cap-add=NET_ADMIN \ -v $(pwd):/workdir/libosmo-sctp \ <NAME_OF_IMAGE> ./test-m3ua.sh 2>&1 | tee result.plain
Note: result.plain will be saved in directory where docker run is executed. In other words the pipe wraps docker command - not command executed within docker.
Imho this is what we need in order to: "... execute the entire testsuite e.g. during jenkins test without having IP address or port number conflicts. It could even run multiple times in parallel on one buildhost, verifying different patches as part of the continuous integration setup." [1]. Moreover, we could also split up tests and execute them in separate containers inside the same job, allowing us to cut down the verification time of each patch set.
Does the container work for you, too?
What do you think about such approach in general?
Furthermore I wrote a protopy-ish script (attached) able to parse "result.plain" (generated by above stated docker command) and create a valid JUnit XML report, which then can be parsed and visualized by Jenkins.
A test job [2] has been created showing how such a "publishing" looks like. One simply clicks on "Latest Test Result" and sees a list of all failures. Each failure can be expanded to see the "Stacktrace:/Backtrace:". Furthermore, the first line of such a "Stacktrace" specifies whether it is a "pure" failure or a timeout.
The console log of a jenkins job using mentioned Dockerfile and script (create_xml_report.sh) would first hold the verbose log of the docker command. Then a summary would follow as an result of executing create_xml_report.sh. Thus giving full details as well as a summary inside the console log. (Although the summary would use multiple lines per test case -> less beautiful)
In addition I'd like to mention the possibility of saving duration of each test case inside XML report. This could enable performance statistics!?
What do you think about such approach of creating and publishing XML reports in general?
Please note that create_xml_report.sh still needs a clean up, but at the same time I thought it's already worth sharing my hands-on results. Furthermore, feel free to request more details on specific topics e.g. JUnit XML reports, Jenkins, Docker (tried to keep mail crisp).
Best Regards, André
[1] http://laforge.gnumonks.org/blog/20170503-docker-overhyped/ [2] https://jenkins.blobb.me/job/check_xml_report/
Hi Andre,
On Tue, May 23, 2017 at 07:20:08PM +0200, André Boddenberg wrote:
Long story short, here's a Dockerfile (attached) capable of running tests on top of a compiled (via ./contrib/jenkins.sh) libosmo-sctp repository. Only one container is used and IPs can be created on loopback interface in fact of using "--add-cap=NET_ADMIN", which imo is a mighty feature - able of pushing docker beyond its proposed use cases.
After building the Dockerfile execute following "line" inside compiled libosmo-sctp repo:
docker run -it --rm --cap-add=NET_ADMIN \ -v $(pwd):/workdir/libosmo-sctp \ <NAME_OF_IMAGE> ./test-m3ua.sh 2>&1 | tee result.plain
# docker run -it --rm --cap-add=NET_ADMIN -v $(pwd):/workdir/libosmo-sctp 16e576fa31f9 ./test-m3ua.sh 2>&1 | tee result.plain Test m3ua-sgp-aspsm-v-001 /workdir/libosmo-sctp/stp/.libs/osmo-stp: error while loading shared libraries: libosmovty.so.3: cannot open shared object file: No such file or directory
I guess there's a ldconfig missing somewhere? How are you actually building the code if there's no libosmovty installed from source in your Dockerfile?
BTW: should we add your Docker file to the collection of Dockerfiles in http://git.osmocom.org/docker-playground/ ?
Does the container work for you, too?
unfortunately not, see above :/
What do you think about such approach in general?
I'd like to play with it, but first have to get it to work.
In general, I don't really care too much *how* the tests are excecuted, just as long as it is based on technology that works well and is present on our buildhosts/jenkins.
What was wrong with my simple 'unshare' based script?
A test job [2] has been created showing how such a "publishing" looks like. One simply clicks on "Latest Test Result" and sees a list of all failures.
looks great!
Each failure can be expanded to see the "Stacktrace:/Backtrace:". Furthermore, the first line of such a "Stacktrace" specifies whether it is a "pure" failure or a timeout.
interesting
In addition I'd like to mention the possibility of saving duration of each test case inside XML report. This could enable performance statistics!?
yes, but I don't think it is of much use, as the build hosts might be doing all kinds of other things in parallel?
What do you think about such approach of creating and publishing XML reports in general?
Looks fine to me. Maybe there's even a simple/straightforward way to do that for the gnu autotest based unit tests we have in the various osmocom projects?
Please note that create_xml_report.sh still needs a clean up, but at the same time I thought it's already worth sharing my hands-on results. Furthermore, feel free to request more details on specific topics e.g. JUnit XML reports, Jenkins, Docker (tried to keep mail crisp).
wouldn't it make sense to simply replace the 'runm3uatest' executor and the shell script that repeatedly executes it with something that generates the required XML directly? It could be a python script directly, doing away with the C program that does nothing but executing a guile interpreter ;)
Hi,
Thanks, works for me too. Now I can see the test output, although tests are failing. Probably "culprit" is line 29 in test-m3ua.sh:
Line29 => $STP_DIR/osmo-stp -c $STP_CONFIG &
Error=> ./test-m3ua.sh: 29: ./test-m3ua.sh: ../..//stp/osmo-stp: not found
I've fixed this by properly building libosomo-sccp (using ./contrib/jenkins.sh). Please excuse impatience.
Regards, André
-
André Boddenberg -
Harald Welte