Dear fellow Osmocom developers,
I just would like to use this as a gentle reminder that we shouldn't be merging new features without having automatic tests for them in place.
What comes to my mind immediately right now (but I'm sure there are other examples) is the "subscriber create on demand" feature. I couldn't find any tests in HLR_Tests.ttcn about this feature.
What this means is, that it will *eventually* break unnoticed, particularly until somebody is using that feature and updating frequently to master, which is unlikely to happen. Either you run a production system and then you don't follow master, or you're just playing around and then that feature is probably not something you'd be using in many cases.
What actually bugs me most about it, is that the tests should have been written first. For most of our development [1], the existing infrastructure in terms of TTCN3 Modules is very strong, and adding related tests is rather quick. This means it's *very* feasible to write the tests first and then the actual code. In fact, my own experience shows that development is much faster this way, as maual testing of the entire stack with phones is sloooow.
This is not to complain to Oliver or any single person, but just a general reminder to all of us. That includes first and foremost myself as I'm merging most of the development, but is addressed to all of the developers and reviewers: IF something doesn't have a test, but could reasonably be tested without spending a multitude of the development time on tests, we should mandate that tests are available at the time of merge.
We of course cannot mandate or enforce that developers write the tests first and follow test-driven development methodology. But I would at least strongly encourage everyone to try that. If you first spend tons of time with manual testing and then write a test case, it's much less efficient as you spend time for both. OTOTH, if you first invest the time into writing the test, then the development can make very quick progress and you save a lot of time that would otherwise be wasted with manual testing.
Regards, Harald
[1] I'm referring to rather self-cntained, small features. For sure, e.g. testing inter-MSC-HO requires lots of new tsting infrastructure to be developed, as does testing of the PCU. So yes, there are exception.
Dear Harald and ML,
thank you for the feedback. I fully agree with your mail, and will create the missing tests for "subscriber create on demand" (and the related check IMEI code) soon. And for future features, try to write them first.
Regards, Oliver
On 5/24/19 11:36 AM, Harald Welte wrote:
Dear fellow Osmocom developers,
I just would like to use this as a gentle reminder that we shouldn't be merging new features without having automatic tests for them in place.
What comes to my mind immediately right now (but I'm sure there are other examples) is the "subscriber create on demand" feature. I couldn't find any tests in HLR_Tests.ttcn about this feature.
What this means is, that it will *eventually* break unnoticed, particularly until somebody is using that feature and updating frequently to master, which is unlikely to happen. Either you run a production system and then you don't follow master, or you're just playing around and then that feature is probably not something you'd be using in many cases.
What actually bugs me most about it, is that the tests should have been written first. For most of our development [1], the existing infrastructure in terms of TTCN3 Modules is very strong, and adding related tests is rather quick. This means it's *very* feasible to write the tests first and then the actual code. In fact, my own experience shows that development is much faster this way, as maual testing of the entire stack with phones is sloooow.
This is not to complain to Oliver or any single person, but just a general reminder to all of us. That includes first and foremost myself as I'm merging most of the development, but is addressed to all of the developers and reviewers: IF something doesn't have a test, but could reasonably be tested without spending a multitude of the development time on tests, we should mandate that tests are available at the time of merge.
We of course cannot mandate or enforce that developers write the tests first and follow test-driven development methodology. But I would at least strongly encourage everyone to try that. If you first spend tons of time with manual testing and then write a test case, it's much less efficient as you spend time for both. OTOTH, if you first invest the time into writing the test, then the development can make very quick progress and you save a lot of time that would otherwise be wasted with manual testing.
Regards, Harald
[1] I'm referring to rather self-cntained, small features. For sure, e.g. testing inter-MSC-HO requires lots of new tsting infrastructure to be developed, as does testing of the PCU. So yes, there are exception.
On 24/05/2019 11:36, Harald Welte wrote:
Dear fellow Osmocom developers, [...]
Hi Harald..
I certainly have in mind that in order to continue developing, especially if it involves anything approaching a "new feature", I need to learn how to write tests, and then of course to test the tests before using the tests to test! Given that ttcn3 is still a new language to me, this represents something of a learning curve, of course.
I know that there is a docker setup (I haven't played with it yet) - I'm just wondering if there is a easier (better) way than each individual developer running there own entire ttcn3/osmo stack test environment?
Maybe I'm over estimating the complexity of that?
Thanks.
k
Hi Keith,
On Fri, May 24, 2019 at 02:41:39PM +0200, Keith wrote:
Given that ttcn3 is still a new language to me, this represents something of a learning curve, of course.
I underestand and appreciate that. IT was new to all of us some ~2.5 years ago.
The advantage *now* is that we pretty much have the infrastructure in place for most of the things we'd typically encounter in testing (with exception of the PCU), and plenty of examples (the existing test cases). This should hopefully make writing new tests a lesser challenge. From a syntactical point of view, it's luckily not so far from C. And for sure one can simply hack at it without understanding the full language and/or all of its concepts.
Eric is currently going thorugh this, and just got his first new test (about initial TA in channel activation) merged.
I know that there is a docker setup (I haven't played with it yet) - I'm just wondering if there is a easier (better) way than each individual developer running there own entire ttcn3/osmo stack test environment?
For executing the test suite, the dockerized setup is clearly the lowest entrance barrier. It requires virtually zero configuration (aside from docker being around) - but building the images, etc. of course takes time - and we don't use ccache in that setup, so build times are long.
I wuold assume that in order to have quick development cycles and to control various things interactively one would normally want to run the testsuite natively while developing tests. There's a bit of hassle involved in that, in order to get the adresses/ports/etc. all configured in a way to work. But that should be one-time exercise...
We can have jenkins jobs to execute newly developed tests from personal branches on build hosts, if you think that would make it simpler. But honestly, I think one wants to see the test running locally, quickly edit a line, restart, ...
Maybe I'm over estimating the complexity of that?
executing the tests shouldn't be the complex part - at least not more complex than setting up a cellular network with elements of different suppliers, where all kinds of addresses, parameters, etc. have to be set up (i.e. not having all the 'atomagic' Osmocom stuff we introduced to simplify nitb-like setups, with automatic routing key management, ...)
Regards, Harald
-
Harald Welte -
Keith -
Oliver Smith