VTY tests still causing build failures

[Harald Welte]

Dear all,

for probably about a year (or longer) we have been putting up with VTY
tests which cause builds to break under unclear circumstances.  I personally
believe the probability of a VTY test failing has recently increased again,
and this is barely tolerable anymore.  Often, rebasing/cherry-picking the given
patch one or two times also doesn't work.  Yet, the given patch-under-test
is not even touching anything related to VTY, like
In https://gerrit.osmocom.org/3899 which has failed in
https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2451/ and
https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2454/

I know Neels and others have spend already significant time in the past trying
to resolve this - unsuccessfully.

So I think the situation has reached a point where we should disable the vty
tests, or at least the specific part of the vty tests that is known to break
most frequently.

I definitely want us to have *more* testing, not less.  However, when the test
itself is not stable yet - particularly after that much time - we cannot
have that buggy test delay our development.

I would vote for running those tests regularly (daily, every few hours, you name
it), but not as part of the mandatory build verification for gerrit V+1.

What do others think?

-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)