11 Sep
2017
11 Sep
'17
9:09 p.m.
Dear all,
for probably about a year (or longer) we have been putting up with VTY
tests which cause builds to break under unclear circumstances. I personally
believe the probability of a VTY test failing has recently increased again,
and this is barely tolerable anymore. Often, rebasing/cherry-picking the given
patch one or two times also doesn't work. Yet, the given patch-under-test
is not even touching anything related to VTY, like
In https://gerrit.osmocom.org/3899 which has failed in
https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2451/ and
https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2454/
I know Neels and others have spend already significant time in the past trying
to resolve this - unsuccessfully.
So I think the situation has reached a point where we should disable the vty
tests, or at least the specific part of the vty tests that is known to break
most frequently.
I definitely want us to have *more* testing, not less. However, when the test
itself is not stable yet - particularly after that much time - we cannot
have that buggy test delay our development.
I would vote for running those tests regularly (daily, every few hours, you name
it), but not as part of the mandatory build verification for gerrit V+1.
What do others think?
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
12 Sep
12 Sep
1:45 p.m.
On Mon, Sep 11, 2017 at 09:09:06PM +0200, Harald Welte wrote:
Dear all,
for probably about a year (or longer) we have been putting up with VTY
tests which cause builds to break under unclear circumstances. I personally
believe the probability of a VTY test failing has recently increased again,
and this is barely tolerable anymore. Often, rebasing/cherry-picking the given
patch one or two times also doesn't work. Yet, the given patch-under-test
is not even touching anything related to VTY, like
In https://gerrit.osmocom.org/3899 which has failed in
https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2451/ and
https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2454/
I know Neels and others have spend already significant time in the past trying
to resolve this - unsuccessfully.
So I think the situation has reached a point where we should disable the vty
tests, or at least the specific part of the vty tests that is known to break
most frequently.
I definitely want us to have *more* testing, not less. However, when the test
itself is not stable yet - particularly after that much time - we cannot
have that buggy test delay our development.
If there are no resources / noone with an assignment to actively
maintain this, then it's reasonable to disable it, or at least disable
the tests that are breaking things now.
I would vote for running those tests regularly (daily,
every few hours, you name
it), but not as part of the mandatory build verification for gerrit V+1.
I think this is fine, so we get fallout later on that we can address
via robots, and make things a bit more agile.
In the Linux kernel, we usually get all these reports from robots
afterwards, so I would say it's reasonable to follow the same
approach.
13 Sep
13 Sep
1:41 a.m.
On Mon, Sep 11, 2017 at 09:09:06PM +0200, Harald Welte wrote:
In https://gerrit.osmocom.org/3899 which has failed
in
https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2451/ and
https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2454/
This particular failure is due to a VTY change in libosmocore. I have fixed it
in osmo-bsc.git, and this needs to be applied to openbsc.git as well.
Change-Id: I77931d6a09c42c443c6936000592f22a7fd06cab
However, let's decide when we stop developing on openbsc.git. Every patch that
is merged to openbsc.git now needs additional work to be applied to osmo-*.git.
Vice versa, we only need to backport serious fixes to openbsc.git, this is one
of them.
Here is the backport to openbsc.git:
https://gerrit.osmocom.org/3921
I know Neels and others have spend already significant
time in the past trying
to resolve this - unsuccessfully.
That was the testBSCreload running into "Broken Pipe" errors.
If you see more of those, we may want to disable the testBSCreload:
https://gerrit.osmocom.org/3922
~N
10:08 a.m.
On 13. Sep 2017, at 07:41, Neels Hofmeyr
<nhofmeyr(a)sysmocom.de> wrote:
Hi!
On Mon, Sep 11, 2017 at 09:09:06PM +0200, Harald Welte
wrote:
This particular failure is due to a VTY change in libosmocore. I have fixed it
in osmo-bsc.git, and this needs to be applied to openbsc.git as well.
Change-Id: I77931d6a09c42c443c6936000592f22a7fd06cab
Great. So the VTY tests found a behavior change and did its job. I think
disabling tests is a slippery slope. Let's assume we would run it daily
and send emails. How likely would it be that n-failures in a row trigger
a question to disable the mail notifications?
We do run into some form of resource limitation and mitigated by reducing
the number of executors (but that is up again). In the past the VTY test
runner forgot to close sockets but we were still running into something.
So either a form of kernel limit (and I couldn't find a MIB counting it)
or something caused by "slow" (as recently pointed out) disk leading to a
slow start of the software under test?
I know Neels
and others have spend already significant time in the past trying
to resolve this - unsuccessfully.
That was the testBSCreload running into "Broken Pipe" errors.
If you see more of those, we may want to disable the testBSCreload:
1:02 p.m.
On Wed, Sep 13, 2017 at 04:08:51PM +0800, Holger Freyther wrote:
Also it has just uncovered that the osmo-nitb VTY's 'nitb' vty node lacked
the
default node commands (including 'exit' and 'end'), so it was never
really
necessary to leave the 'nitb' node. It always took the parent node's exit/end
command. Fix applied in https://gerrit.osmocom.org/3921, hope it gets +V now,
then we can rebase the other patches onto it.
This
particular failure is due to a VTY change in libosmocore. I have fixed it
in osmo-bsc.git, and this needs to be applied to openbsc.git as well.
Change-Id: I77931d6a09c42c443c6936000592f22a7fd06cab
Great. So the VTY tests found a behavior change and did its job. I think
disabling tests is a slippery slope.
About the particular testBSCreload test that apparently still fails sometimes,
and despite extensive digging we never came up with a way to solidify it, I
think that we might be ok with disabling that one. Only that one.
We do run into some form of resource limitation and
mitigated by reducing
the number of executors (but that is up again). In the past the VTY test
runner forgot to close sockets but we were still running into something.
So either a form of kernel limit (and I couldn't find a MIB counting it)
or something caused by "slow" (as recently pointed out) disk leading to a
slow start of the software under test?
[...]
broken pipe still sounds like either we kill the TCP
connection before
we want to or the remote process terminated. Could we dump core and check
if these exist at the end of the test run (and check that dumping core in
a container works).
The debug output we've added so far didn't suggest an obvious error. It would
be great to find a solution ... but I feel that other todos are more important
ATM :/
~N
7:34 p.m.
On Wed, Sep 13, 2017 at 01:02:57PM +0200, Neels Hofmeyr wrote:
Also it has just uncovered that the osmo-nitb
VTY's 'nitb' vty node lacked the
default node commands (including 'exit' and 'end'), so it was never
really
necessary to leave the 'nitb' node. It always took the parent node's exit/end
s/necessary/possible
command. Fix applied in
https://gerrit.osmocom.org/3921, hope it gets +V now,
then we can rebase the other patches onto it.
~N
2427
days inactive
2429
days old
5 comments
4 participants
participants (4)
-
Harald Welte -
Holger Freyther -
Neels Hofmeyr -
Pablo Neira Ayuso