On Wed, Dec 20, 2017 at 03:13:38PM +0100, Harald Welte wrote:
- how can a patch that breaks 'make check' of osmo-bts/openbsc/osmo-pcu, etc. be merged to libosmcoore? Isn't that what we have inverse dependencies for in the gerrit verification jobs?
In libosmocore gerrit, we don't test depending projects. We'd find the master jobs failing if we merge a breaking change.
- how can it be that even after the patch is merged, we find the osmo-sgsn issue only by manual testing and not by automatic testing?
I also wondered about this, and AFAICT the reason is that osmo-gsm-tester doesn't do more than one GPRS subscriber. SGSN worked fine with one subscriber, so we need a test that adds a second GPRS user to osmo-gsm-tester...
https://osmocom.org/issues/2820 Maybe a titan test would be more fitting to add a number of subscribers?
b) introduce an intermediate stage, something like a "staging" branch. So the existing code review and gerrit-jenkins-build-testing would make patches end up in 'staging' first, where then the slower (hourly/daily/...) tests like osmo-gsm-tester are executed. And only once that's green, we go towards master.
We could cause each and every gerrit patch to trigger osmo-gsm-tester and/or titan test runs; i.e. perfect coverage means "infinite" jenkins job effort and time to wait for V+1.
Aiming for perfect coverage before merging patches seems to me to be fighting windmills; we can get better with every failure, but I guess we'll always need reassurance from real-world tests.
So far the tradeoff was to rely on master job failures and osmo-gsm-tester failures to be reported. Which works only as long as we watch those and as long as they are able to catch the particular issues.
Maybe an IRC bot to report build failures to #osmocom-dev could be nice besides mail notifications... just dreaming.
I don't think that any given single "event network" should dictate the patch merging.
Was of course a selfish, not entirely serious request, with a wink.
~N