VTY tests still causing build failures

[Pablo Neira Ayuso]

On Mon, Sep 11, 2017 at 09:09:06PM +0200, Harald Welte wrote:
> Dear all,
> 
> for probably about a year (or longer) we have been putting up with VTY
> tests which cause builds to break under unclear circumstances.  I personally
> believe the probability of a VTY test failing has recently increased again,
> and this is barely tolerable anymore.  Often, rebasing/cherry-picking the given
> patch one or two times also doesn't work.  Yet, the given patch-under-test
> is not even touching anything related to VTY, like
> In https://gerrit.osmocom.org/3899 which has failed in
> https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2451/ and
> https://jenkins.osmocom.org/jenkins/job/OpenBSC-gerrit/2454/
> 
> I know Neels and others have spend already significant time in the past trying
> to resolve this - unsuccessfully.
> 
> So I think the situation has reached a point where we should disable the vty
> tests, or at least the specific part of the vty tests that is known to break
> most frequently.
> 
> I definitely want us to have *more* testing, not less.  However, when the test
> itself is not stable yet - particularly after that much time - we cannot
> have that buggy test delay our development.

If there are no resources / noone with an assignment to actively
maintain this, then it's reasonable to disable it, or at least disable
the tests that are breaking things now.

> I would vote for running those tests regularly (daily, every few hours, you name
> it), but not as part of the mandatory build verification for gerrit V+1.

I think this is fine, so we get fallout later on that we can address
via robots, and make things a bit more agile.

In the Linux kernel, we usually get all these reports from robots
afterwards, so I would say it's reasonable to follow the same
approach.