18 May
2017
18 May
'17
11:46 a.m.
Hi Harald,
> they are not yet listed on the index, but you
should be able to clone
> them via git://git.osmocom.org/nplab/m3ua-testtool and
> git://git.osmocom.org/nplab/sua-testtool already. I just tried, works
> for me [tm].
Thanks, works for me too. Now I can see the test output, although
tests are failing. Probably "culprit" is line 29 in test-m3ua.sh:
Line29 => $STP_DIR/osmo-stp -c $STP_CONFIG &
Error=> ./test-m3ua.sh: 29: ./test-m3ua.sh: ../..//stp/osmo-stp: not found
Can you please help me to fix this issue?
> ok. For the m3ua/sua testtool, the
'runm3uatest' program already
> contains some custom colorized output, you might simply be able to patch
> that to generate something useful to jenkins?
That was my first approach, but I didn't manage to pipe the colorized
output to a file. Any ideas?
So I (temporarily) changed following line in run-all-sgp-tests.sh:
$ (./runm3uatest -d . -t $timeout $testcase > /dev/tty) >& /dev/null
to
$ ./runm3uatest -d . -t $timeout $testcase
Then executing
$ ./run-in-ns.sh ./test-m3ua.sh 2>&1 | tee m3ua-test-report.plain
gives a very verbose output, which is saved to a
"m3ua-test-report.plain" for XML generation after all tests have been
run. At this point I got the feeling that a more verbose output could
also be more helpful in a Jenkins job than prompting the summary,
which is probably more suited for builds run by a developer.(?)
What is your general opinion about the verboseness of the Jenkins log
and the XML report?
Here is how a XML report looks like when we would simply evaluate the
"test output summary":
<testsuites>
<testsuite name="" tests="32" failures="32"
errors="32"/>
<testcase name="m3ua-sgp-aspsm-v-002"/>
<testcase name="m3ua-sgp-aspsm-v-001"/>
</testsuite>
</testsuites>
This XML snippet represent the current state after my hands on yesterday:
<testsuites>
<testsuite name="" tests="32" failures="32"
errors="32"/>
<testcase name="m3ua-sgp-aspsm-v-001" time="51011250"/>
<error type=" In procedure sctp-sendmsg">
Some deprecated features have been used. Set the environment
variable GUILE_WARN_DEPRECATED to "detailed" and rerun the
program to get more information. Set it to "no" to suppress
this message.
FAILED
</error>
<testcase name="m3ua-sgp-aspsm-v-003" time="58336593"/>
......
</testsuite>
</testsuites>
I personally always prefer when Jenkins provides some information
about the failure rather than only pointing out that something failed
and referring to the console log.
Something else, can I may request a failing test or information how to
change one to let it fail? :)
Without such test case it's hard to test the script, which shall
produce the XML report.
Regards,
André