13 Nov
2011
13 Nov
'11
1:30 a.m.
Hi Pablo, all
I have pushed GNU autotest[1] integration of libosmocore into the
zecke/gnu-autotest branch and invoking make check will execute the testsuite.
The output looks like this:
## ------------------------------------- ##
## libosmocore 0.4.0.10-c015 test suite. ##
## ------------------------------------- ##
Regression tests.
1: bits ok
2: msgfile ok
3: sms ok
4: smscb ok
5: timer FAILED (testsuite.at:38)
6: ussd FAILED (testsuite.at:44)
GNU autotest will execute an external application and then can check the exit
code, compare the stdout/stderr to a file. In this case the timer test fails
as the test itself is randomized and does not always provide the same output.
Pablo if your time permits it would be nice if you could:
- Provide a cli option to make the test have less iterations (to make
make check run faster)
- Provide a cli option to produce a repeatable output (e.g. by
omitting the expired output).
What do you think? In some ways I think that executing the timer test as part
of our regression tests makes sense but maybe specially on a loaded machine
the test might be flaky...
holger
[1] http://www.gnu.org/s/hello/manual/autoconf/Using-Autotest.html#Using-Autote…
13 Nov
13 Nov
2:25 a.m.
Hi Holger,
On Sun, Nov 13, 2011 at 01:30:16AM +0100, Holger Hans Peter Freyther wrote:
Hi Pablo, all
I have pushed GNU autotest[1] integration of libosmocore into the
zecke/gnu-autotest branch and invoking make check will execute the testsuite.
The output looks like this:
## ------------------------------------- ##
## libosmocore 0.4.0.10-c015 test suite. ##
## ------------------------------------- ##
Regression tests.
1: bits ok
2: msgfile ok
3: sms ok
4: smscb ok
5: timer FAILED (testsuite.at:38)
6: ussd FAILED (testsuite.at:44)
GNU autotest will execute an external application and then can check the exit
code, compare the stdout/stderr to a file. In this case the timer test fails
as the test itself is randomized and does not always provide the same output.
Interesting, never played with this autotest stuff so far.
Pablo if your time permits it would be nice if you
could:
- Provide a cli option to make the test have less iterations (to make
make check run faster)
Patch attached for this.
- Provide a cli option to produce a repeatable output
(e.g. by
omitting the expired output).
Can we tell the tool to compare stdout but to ignore stderr? If so, we
can display the repeatable output in stdout and the non-repeatable output
in stderr.
What do you think? In some ways I think that executing
the timer test as part
of our regression tests makes sense but maybe specially on a loaded machine
the test might be flaky...
Yes, with lots of timers, the expiration may not be done in time on
a loaded machine. If we can ignore the stderr output, we can put the
information about timers not expiring in time to stderr.
9:05 a.m.
On 11/13/2011 02:25 AM, Pablo Neira Ayuso wrote:
Interesting, never played with this autotest stuff so
far.
I saw it in GNU Smalltalk. Somehow the testsuite.at is too verbose and we
would deserve an osmo.m4 macro but then we are in trouble in regard to
installing this one. :}
Patch attached for this.
thanks, picked into the branch.
Can we tell the tool to compare stdout but to ignore
stderr? If so, we
can display the repeatable output in stdout and the non-repeatable output
in stderr.
Good idea, I comitted a patch to do this, would be awesome if you play with
sending things to stdout/stderr.
Yes, with lots of timers, the expiration may not be
done in time on
a loaded machine. If we can ignore the stderr output, we can put the
information about timers not expiring in time to stderr.
Not expiring at all should probably be counted as test failure?
thanks for the fast response
5:54 p.m.
On Sun, Nov 13, 2011 at 09:05:41AM +0100, Holger Hans Peter Freyther wrote:
Thanks for applying the patch. I made one patch for this.
You mean not expiring in time? I think so. If we use very few timers
for the test (like it is the case for 5 steps), it seems to me very
unlikely that we'll fail to make it in time. Well, if you launch some
very CPU intensive tasks I might be wrong, but I don't think this will
be the case for people running make check. After all, if we start getting
reports of people that run the test that bogusly fail, we can relax the
checkings.
Mreover it's a good idea to set some maximum wait time for the test to
finish via signals. Check the second patch attached to this email, apply
it if you like it.
Btw, please, don't remove me from the Cc, I check the list often but
my mail client has a filter to put emails directed to me (To or Cc) in
one specific folder, so I can prioritize emails that require some
reply from me.
On 11/13/2011 02:25 AM, Pablo Neira Ayuso wrote:
Not sure I understood the problem. We can store the custom xyz.m4 file
under the m4/ directory and distribute it with the tarball, if needed.
Interesting, never played with this autotest
stuff so far.
I saw it in GNU Smalltalk. Somehow the testsuite.at is too verbose and we
would deserve an osmo.m4 macro but then we are in trouble in regard to
installing this one. :} Patch attached
for this.
thanks, picked into the branch.
Can we tell the tool to compare stdout but to
ignore stderr? If so, we
can display the repeatable output in stdout and the non-repeatable output
in stderr.
Good idea, I comitted a patch to do this, would be awesome if you play with
sending things to stdout/stderr. Yes, with lots
of timers, the expiration may not be done in time on
a loaded machine. If we can ignore the stderr output, we can put the
information about timers not expiring in time to stderr.
Not expiring at all should probably be counted as test failure?
6:45 p.m.
On 11/13/2011 05:54 PM, Pablo Neira Ayuso wrote:
On Sun, Nov 13, 2011 at 09:05:41AM +0100, Holger Hans
Peter Freyther wrote:
Not sure I understood the problem. We can store the custom xyz.m4 file
under the m4/ directory and distribute it with the tarball, if needed.
a) we copy the same file into every repository...
b) we install it by libosmocore... but I tend to install my software with
--prefix=$HOME/install/openbsc... so the normal autoreconf --install --force
would not find osmo.m4 (without specifying more paths)
will try the patch in a second.
holger
4558
days inactive
4558
days old
4 comments
2 participants
participants (2)
-
Holger Hans Peter Freyther -
Pablo Neira Ayuso