Hi all,
we have recently merged a profound change to the inner workings of the VTY configuration. We've hit some fallout due to that, hence I would like to let you know that you might hit the same.
The VTY parsing of config files is now strict about indenting. A child node *must* be indented below the parent node, and indenting must be consistent.
For more details on the reason why and a definition of 'consistent', see the commit log of https://git.osmocom.org/libosmocore/commit/?id=4a31ffa2f0097d96201f80305a049...
So, if you are in the near future faced with config files being rejected that worked perfectly before, it is most probably because your indenting was wrong all the time, and only now did we start checking it.
If the cause is hard to figure out, a good aid is the telnet VTY, where you may either 'show running-config' or traverse the nodes manually to query which command exists on which level. If your current config is broken, you may be able to start the program with an empty or stripped down config file to make its telnet available.
The above is about reading config files, but we have also dropped the implicit 'exit' to parent level on the telnet VTY console. This caused fallout where nodes by plain omission lack the 'exit' command: one is unable to leave such a node once it is entered. That is a fault in the program's VTY setup that was not found before, because the VTY would often just find the parent node's exit command instead. I have a patch to avoid all of these everywhere, by installing the exit and end commands automatically for every VTY node that gets created; it is not merged yet: https://gerrit.osmocom.org/3998
------ Config Fallout Example ------
For example, we hit a breakage with this osmo-bts-trx.cfg:
phy 0 instance 0 osmotrx rx-gain 25 osmotrx tx-attenuation oml osmotrx ip local 10.23.42.1 osmotrx ip remote 10.23.42.2
Before, this worked perfectly. Now it says the command 'osmotrx rx-gain' does not exist. The reason is that 'osmotrx rx-gain' is a child node of 'instance 0'. The first fix looked like this:
phy 0 instance 0 osmotrx rx-gain 25 osmotrx tx-attenuation oml osmotrx ip local 10.23.42.1 osmotrx ip remote 10.23.42.2
But alas, this time it said the command 'osmotrx ip local' does not exist. I suspected bugs in the new parsing, but indeed, 'osmotrx ip' is actually a direct child of the 'phy 0' level. Before, the VTY would implicitly step out of a child level if it found a matching command one parent above. That is confusing in other situations (see above commit log), hence we now require indenting to clarify the structure. This is the correct one:
phy 0 instance 0 osmotrx rx-gain 25 osmotrx tx-attenuation oml osmotrx ip local 10.23.42.1 osmotrx ip remote 10.23.42.2
Or rephrased:
phy 0 osmotrx ip local 10.23.42.1 osmotrx ip remote 10.23.42.2 instance 0 osmotrx rx-gain 25 osmotrx tx-attenuation oml
To query the structure via telnet, I did:
$ ./osmo-bts/src/osmo-bts-trx/osmo-bts-trx -c osmo-bts/doc/examples/trx/osmo-bts.cfg
$ telnet localhost 4241 OsmoBTS> enable OsmoBTS# configure terminal OsmoBTS(config)# phy 0 OsmoBTS(phy)# list [...] osmotrx ip HOST osmotrx ip (local|remote) A.B.C.D [...] OsmoBTS(phy)# inst OsmoBTS(phy)# instance 0 OsmoBTS(phy-inst)# list [...] osmotrx rx-gain <0-50> osmotrx tx-attenuation <0-50> osmotrx tx-attenuation oml [...]
------ No Exit Example ------
The same osmo-trx VTY nodes also show the exit failure:
$ telnet localhost 4241 OsmoBTS> enable OsmoBTS# configure terminal OsmoBTS(config)# phy 0 OsmoBTS(phy)# exit % Unknown command. OsmoBTS(phy)# instance 0 OsmoBTS(phy-inst)# exit % Unknown command.
On the phy level, we would previously find the parent's 'exit' command, but in the 'instance' level, 'exit' has never been available. Above patch should fix all of these.
~N
Hi Neels,
On Wed, Sep 20, 2017 at 04:06:16PM +0200, Neels Hofmeyr wrote:
Hi all,
we have recently merged a profound change to the inner workings of the VTY configuration. We've hit some fallout due to that, hence I would like to let you know that you might hit the same.
The VTY parsing of config files is now strict about indenting. A child node *must* be indented below the parent node, and indenting must be consistent.
For more details on the reason why and a definition of 'consistent', see the commit log of https://git.osmocom.org/libosmocore/commit/?id=4a31ffa2f0097d96201f80305a049...
I understand there are a good reasons for this and that a lot of experimentation is going on to consolidate the project.
But it would be good to keep in the horizon to have a more stricter policy on breaking backward compatibility, even if there are good reason to fix up inconsistent things, or fix in new software, eg. osmo-bsc while keeping the current behaviour around in the legacy openbsc.git tree.
I don't want to create any long debate on this, just an observation.
Thanks!
Hi Pablo,
On Thu, Sep 21, 2017 at 01:44:02PM +0200, Pablo Neira Ayuso wrote:
I understand there are a good reasons for this and that a lot of experimentation is going on to consolidate the project.
But it would be good to keep in the horizon to have a more stricter policy on breaking backward compatibility, even if there are good reason to fix up inconsistent things, or fix in new software, eg. osmo-bsc while keeping the current behaviour around in the legacy openbsc.git tree.
That's of course hard if the given code is part of libosmocore, which is used by all osmocom programs. However, you can (and will be able to) build osmo-nitb still against older versions of libosmocore to retain the old behavior.
One could of course introduce a runtime swithc or (beware) a compile-time switch. But then, you also have to maintain two sets of documentation, test both behaviours from the VTY tests, etc. - and the effort quickly explodes.
I don't want to create any long debate on this, just an observation.
Your thoughts are much appreciated, and I know that this is a contentious issue.
The problem is that with the constant scarcity of resources and the limited funding (and even more so, code contributions) to cover an extremely large problem space - basically the entire 2G/3G cellular network architecture, covering all network elements and all layers of each of the protocol stacks on its interfaces. Supporting too many different 'ways of doing things' is going make the matrix explode beyond what a small team can manage to develop and maintain. We have to invest our time wisely.
So, sometimes in earlier days there were now seemingly stupid choices that we need to change later on. Or we inherited code with properties that were not fully understood (like the VTY code from zebra/quagga, with its "implicitly try commands of the parent node" feature that caused fall-out now and had to be fixed).
Those kind of changes are not introduced lightly, and only when we see no real alternative. What we could and probably should do better is to properly document this, e.g. in form of a manually written changelog that gets published at the time we tag a new version of a given project, That changlog update should be part of the patch that introduces some compatibility related code change. But if ti is missed, that changelog update could also come from another developer or user who finds out and who wants to ensure nobody else falls into the same trap.
The other thing we can possibly do is some scripts that can aid in the conversion, like a "fix the indenting of my config file" script that you can run once during a related upgrade.
At some point we intend to have formal "Osmocom releases", which would then receive some amount of maintenance and bugfix backporting. However, once again this consumes considerable resources. Putting all this on sysmocom shoulders is unlikely to work. We will need some people with significant interest in that to step up and say "yes, I'll maintain Release X" similar to e.g. a given -stable maintainer of the kernel.
We've tried to bundle a lot of the large changes with the NITB-split to make sure there's going to be one time when people do that migration where all even technically unrelated changes happen at once. That's for example the OpenGGSN->OsmoGGSN transition, whih has nothing to do with the NITB, but we thought it's a good idea to do that at the same time so people migrating from NITB to BSC+MSC+HLR will not have to do one change per month but all of them together.
Regards, Harald
On Thu, Sep 21, 2017 at 01:44:02PM +0200, Pablo Neira Ayuso wrote:
But it would be good to keep in the horizon to have a more stricter policy on breaking backward compatibility, even if there are good reason to fix up inconsistent things, or fix in new software, eg. osmo-bsc while keeping the current behaviour around in the legacy openbsc.git tree.
Yes, I mentioned this idea in the patch submission: to be able to pass a flag to the VTY to enable or disable the new indentation bound behavior. The patch was accepted without it, but I would still favor such an implementation. It would be logical if I did this, but I have a multitude of tasks for the current change to the new repositories. Not sure if sysmocom would fund that, given that the focus is on the new repositories.
As a workaround, it is always possible to use a libosmocore without the VTY patches:
git clone git://git.osmocom.org/libosmocore cd libosmocore git revert 430636328c2fbd9fffc0eac5114462c200b7f2cb 4a31ffa2f0097d96201f80305a0495c57552f0ad
Another workaround would be to not update the OsmoNITB installations on running systems. After all, our intention is to not spend time on new implementations for the old openbsc repository.
The reason why we are doing these things now is that it is a good thing to change everything at once, minimizing the effort for users to change over.
Nevertheless, there are various other features that the old OsmoNITB offered, which we either need to re-implement in the new split way, or which users need to farewell: - accept-all policy, entering phone numbers in the HLR automatically for arbitrary SIM cards. - keeping an IMEI[SV] database beyond volatile in-ram subscriber storage. - introspection of BTS data on the MSC level, i.e. correlate subscribers with actual lchans on a specific BTS. and probably some more.
Do not hesitate to ask about these, be they long discussions or not. The most important aspect after all is to keep the community thriving and affirmative towards Osmocom developments.
~N
osmocom-net-gprs@lists.osmocom.org