Hi Holger,
On Sun, Nov 13, 2011 at 01:30:16AM +0100, Holger Hans Peter Freyther wrote:
Hi Pablo, all
I have pushed GNU autotest[1] integration of libosmocore into the zecke/gnu-autotest branch and invoking make check will execute the testsuite.
The output looks like this: ## ------------------------------------- ## ## libosmocore 0.4.0.10-c015 test suite. ## ## ------------------------------------- ##
Regression tests.
1: bits ok 2: msgfile ok 3: sms ok 4: smscb ok 5: timer FAILED (testsuite.at:38) 6: ussd FAILED (testsuite.at:44)
GNU autotest will execute an external application and then can check the exit code, compare the stdout/stderr to a file. In this case the timer test fails as the test itself is randomized and does not always provide the same output.
Interesting, never played with this autotest stuff so far.
Pablo if your time permits it would be nice if you could:
- Provide a cli option to make the test have less iterations (to make make check run faster)
Patch attached for this.
- Provide a cli option to produce a repeatable output (e.g. by omitting the expired output).
Can we tell the tool to compare stdout but to ignore stderr? If so, we can display the repeatable output in stdout and the non-repeatable output in stderr.
What do you think? In some ways I think that executing the timer test as part of our regression tests makes sense but maybe specially on a loaded machine the test might be flaky...
Yes, with lots of timers, the expiration may not be done in time on a loaded machine. If we can ignore the stderr output, we can put the information about timers not expiring in time to stderr.