m3ua and sua testing as part of jenkins?

[André Boddenberg]

Hi Harald,

>> they are not yet listed on the index, but you should be able to clone
>> them via git://git.osmocom.org/nplab/m3ua-testtool and
>> git://git.osmocom.org/nplab/sua-testtool already.  I just tried, works
>> for me [tm].

Thanks, works for me too. Now I can see the test output, although
tests are failing. Probably "culprit" is line 29 in test-m3ua.sh:

Line29 =>  $STP_DIR/osmo-stp -c $STP_CONFIG &

Error=> ./test-m3ua.sh: 29: ./test-m3ua.sh: ../..//stp/osmo-stp: not found

Can you please help me to fix this issue?


>> ok. For the m3ua/sua testtool, the 'runm3uatest' program already
>> contains some custom colorized output, you might simply be able to patch
>> that to generate something useful to jenkins?

That was my first approach, but I didn't manage to pipe the colorized
output to a file. Any ideas?

So I (temporarily) changed following line in run-all-sgp-tests.sh:

$ (./runm3uatest -d . -t $timeout $testcase > /dev/tty) >& /dev/null

to

$ ./runm3uatest -d . -t $timeout $testcase


Then executing

$ ./run-in-ns.sh ./test-m3ua.sh 2>&1 | tee m3ua-test-report.plain

gives a very verbose output, which is saved to a
"m3ua-test-report.plain" for XML generation after all tests have been
run. At this point I got the feeling that a more verbose output could
also be more helpful in a Jenkins job than prompting the summary,
which is probably more suited for builds run by a developer.(?)

What is your general opinion about the verboseness of the Jenkins log
and the XML report?


Here is how a XML report looks like when we would simply evaluate the
"test output summary":

<testsuites>
<testsuite name="" tests="32" failures="32" errors="32"/>
<testcase name="m3ua-sgp-aspsm-v-002"/>
<testcase name="m3ua-sgp-aspsm-v-001"/>
</testsuite>
</testsuites>

This XML snippet represent the current state after my hands on yesterday:

<testsuites>
<testsuite name="" tests="32" failures="32" errors="32"/>
<testcase name="m3ua-sgp-aspsm-v-001" time="51011250"/>
<error type=" In procedure sctp-sendmsg">

Some deprecated features have been used. Set the environment
variable GUILE_WARN_DEPRECATED to "detailed" and rerun the
program to get more information. Set it to "no" to suppress
this message.
FAILED
</error>
<testcase name="m3ua-sgp-aspsm-v-003" time="58336593"/>
......
</testsuite>
</testsuites>

I personally always prefer when Jenkins provides some information
about the failure rather than only pointing out that something failed
and referring to the console log.


Something else, can I may request a failing test or information how to
change one to let it fail? :)
Without such test case it's hard to test the script, which shall
produce the XML report.


Regards,
  André