Hi Pablo, all
I have pushed GNU autotest[1] integration of libosmocore into the zecke/gnu-autotest branch and invoking make check will execute the testsuite.
The output looks like this: ## ------------------------------------- ## ## libosmocore 0.4.0.10-c015 test suite. ## ## ------------------------------------- ##
Regression tests.
1: bits ok 2: msgfile ok 3: sms ok 4: smscb ok 5: timer FAILED (testsuite.at:38) 6: ussd FAILED (testsuite.at:44)
GNU autotest will execute an external application and then can check the exit code, compare the stdout/stderr to a file. In this case the timer test fails as the test itself is randomized and does not always provide the same output.
Pablo if your time permits it would be nice if you could: - Provide a cli option to make the test have less iterations (to make make check run faster) - Provide a cli option to produce a repeatable output (e.g. by omitting the expired output).
What do you think? In some ways I think that executing the timer test as part of our regression tests makes sense but maybe specially on a loaded machine the test might be flaky...
holger
[1] http://www.gnu.org/s/hello/manual/autoconf/Using-Autotest.html#Using-Autotes...
Hi Holger,
On Sun, Nov 13, 2011 at 01:30:16AM +0100, Holger Hans Peter Freyther wrote:
Hi Pablo, all
I have pushed GNU autotest[1] integration of libosmocore into the zecke/gnu-autotest branch and invoking make check will execute the testsuite.
The output looks like this: ## ------------------------------------- ## ## libosmocore 0.4.0.10-c015 test suite. ## ## ------------------------------------- ##
Regression tests.
1: bits ok 2: msgfile ok 3: sms ok 4: smscb ok 5: timer FAILED (testsuite.at:38) 6: ussd FAILED (testsuite.at:44)
GNU autotest will execute an external application and then can check the exit code, compare the stdout/stderr to a file. In this case the timer test fails as the test itself is randomized and does not always provide the same output.
Interesting, never played with this autotest stuff so far.
Pablo if your time permits it would be nice if you could:
- Provide a cli option to make the test have less iterations (to make make check run faster)
Patch attached for this.
- Provide a cli option to produce a repeatable output (e.g. by omitting the expired output).
Can we tell the tool to compare stdout but to ignore stderr? If so, we can display the repeatable output in stdout and the non-repeatable output in stderr.
What do you think? In some ways I think that executing the timer test as part of our regression tests makes sense but maybe specially on a loaded machine the test might be flaky...
Yes, with lots of timers, the expiration may not be done in time on a loaded machine. If we can ignore the stderr output, we can put the information about timers not expiring in time to stderr.
On 11/13/2011 02:25 AM, Pablo Neira Ayuso wrote:
Interesting, never played with this autotest stuff so far.
I saw it in GNU Smalltalk. Somehow the testsuite.at is too verbose and we would deserve an osmo.m4 macro but then we are in trouble in regard to installing this one. :}
Patch attached for this.
thanks, picked into the branch.
Can we tell the tool to compare stdout but to ignore stderr? If so, we can display the repeatable output in stdout and the non-repeatable output in stderr.
Good idea, I comitted a patch to do this, would be awesome if you play with sending things to stdout/stderr.
Yes, with lots of timers, the expiration may not be done in time on a loaded machine. If we can ignore the stderr output, we can put the information about timers not expiring in time to stderr.
Not expiring at all should probably be counted as test failure?
thanks for the fast response
On Sun, Nov 13, 2011 at 09:05:41AM +0100, Holger Hans Peter Freyther wrote:
On 11/13/2011 02:25 AM, Pablo Neira Ayuso wrote:
Interesting, never played with this autotest stuff so far.
I saw it in GNU Smalltalk. Somehow the testsuite.at is too verbose and we would deserve an osmo.m4 macro but then we are in trouble in regard to installing this one. :}
Not sure I understood the problem. We can store the custom xyz.m4 file under the m4/ directory and distribute it with the tarball, if needed.
Patch attached for this.
thanks, picked into the branch.
Can we tell the tool to compare stdout but to ignore stderr? If so, we can display the repeatable output in stdout and the non-repeatable output in stderr.
Good idea, I comitted a patch to do this, would be awesome if you play with sending things to stdout/stderr.
Thanks for applying the patch. I made one patch for this.
Yes, with lots of timers, the expiration may not be done in time on a loaded machine. If we can ignore the stderr output, we can put the information about timers not expiring in time to stderr.
Not expiring at all should probably be counted as test failure?
You mean not expiring in time? I think so. If we use very few timers for the test (like it is the case for 5 steps), it seems to me very unlikely that we'll fail to make it in time. Well, if you launch some very CPU intensive tasks I might be wrong, but I don't think this will be the case for people running make check. After all, if we start getting reports of people that run the test that bogusly fail, we can relax the checkings.
Mreover it's a good idea to set some maximum wait time for the test to finish via signals. Check the second patch attached to this email, apply it if you like it.
Btw, please, don't remove me from the Cc, I check the list often but my mail client has a filter to put emails directed to me (To or Cc) in one specific folder, so I can prioritize emails that require some reply from me.
On 11/13/2011 05:54 PM, Pablo Neira Ayuso wrote:
On Sun, Nov 13, 2011 at 09:05:41AM +0100, Holger Hans Peter Freyther wrote:
Not sure I understood the problem. We can store the custom xyz.m4 file under the m4/ directory and distribute it with the tarball, if needed.
a) we copy the same file into every repository... b) we install it by libosmocore... but I tend to install my software with --prefix=$HOME/install/openbsc... so the normal autoreconf --install --force would not find osmo.m4 (without specifying more paths)
will try the patch in a second.
holger
-
Holger Hans Peter Freyther -
Pablo Neira Ayuso