Hi All,
Using zebra/quagga VTY code was a great idea back then and has served us nicely in all those years. I like the general style of the command-line based interface with its various nodes, the strict syntax checking, the tab completion and the interactive context-sensivive help.
However, it feels a bit 20ieth-century-ish to have to manually write code to parse and to save the respective values. This is not a productive way to spend our development resources, and it is error prone. The "save" can be forgotten, resulting in non-saveable config parameters. The save can store values that the "parse" function will not be able to read again, etc.
Furthermore, the VTY is inherently a human user interface, not intended for programmatic consumption. For programmatic access, we have developed the control interface. In reality though, only 1% of the parameters available in the VTY are exported via the control interface. And rather than adding all the missing bits and pieces with hand-crafted code to the control interface, we should have a generic way to define a new configuration value, and that value should then be automatically parse- and storable via VTY as well as a programmatic interface.
The next "problem' is that the current telnet connections live within the process of the application. This means that a user can effectively "stall" the main application by performing I/O intensive operation on the VTY, or even on as many VTY/telnet sessions as he wants. There shouldn't be any need for this, at least not for something as mundane as performing configuration changes. Of course the VTY has different use cases such as runtime introspection of system state as well as logging. But still.
Yet another concern is "VTY and telnet port proliferation". Particularly with the conversion from NITB to BSC+MSC+HLR, we are yet again getting more network elements with their own telnet ports. Remembering the port numbers is clumsy. It would be more convenient if a single command line interface could provide access to the configuration and the state of multiple different processes / network elements.
Last, but not least, the current implementation is fixed to telnet, without any form of authentication, and without a path to migrate e.g. to something like SSL or SSH. I don't think it is a major concern (you can always SSH to the system and then telnet locally).
So what I had in mind for quite some time (actually since netconf 1.2 about one year ago), is to have some kind of an external "VTY/MIB daemon" (or even separate daemons for each) which maintains a hierarchical database of configuration values. The MIB deamon simply offers an API (via client library) to GET or SET the individual values, or to NOTIFY an application about a changed value. This API is both used by the actual Osmocom programs to obtain their configuration (and obtain updates to it during runtime), as well as by the "VTY daemon" providing interactive shell access to it. Finally, other external applications could use the same interface/client library to do the same. A proxy to SNMP or REST interfaces can be imagined, for even more interfacing with the outside world.
What I don't like about many existing MIBs I've used (as a sysadmin/user, not a developer) is their simple type system. You can often specify any random value to such a MIB, even one that is completely useless. A simple INT or STRING type is not sufficient, we really want the ability to have ENUM types, to have integers with ranges, etc. - just like we have in the VTY.
Also, the MIBs I've used typically are nothing more than a hierarchical key-value store. They do not contain the information required for interactive, context-sensitive help needed for the "VTY" feel :( This is btw also the problem I have with OpenWRT's uci: It just has keys and values, with no way to constrain those values to something that's reasonable to the given program/context that is being configured, and there's no help or other information associated with those keys and values.
So what I would like to see in this context, is some way to have a machine-parseable description of a "MIB/config item", together with syntax, ranges, enum values, help texts, etc (ideally in the C source code, possibly in comments?) which is used by some kind of MIB compiler to build up the hierarchical structure of the MIB and all of its keys as well as their permitted values and syntax reference. This information is then available to the VTY/telnet connections, irrespective of whether a given application [using those values] is running at all.
Once an application starts, it can query for the configuration values it is interested in and populate its internal data structures from it. Ideally, one would even be able to generate a 'struct' from the MIB compiler, so that there is no need to explicitly call a "get" function on each single configuration value, but one simply gets a C-language 'struct' with all the values filled in by the client library.
As stated above, there should be some way how he MIB can notify applications about changes to certain nodes in the MIB tree, so the application[s] can react to that. I don't have a clear picture yet how transaction logic (like changing multiple values and then committing them at once) would work, or whether applications should even be able to reject/revert a modification?
In either case, I just wanted to share my thoughts on this. I know it sounds rather complex, but I think the investment in some kind of unified configuration database system would save us of a lot of boilerplate code and subtle bugs and inconsistencies.
I've created https://osmocom.org/issues/1975 with above text to keep track of it, but I think at this point it's more of a mailing list discussion rather than something we can put into an issue. I guess the progress would rather be * first discuss it here and/or at OsmoDevCon * create something like a specification in the wiki * then follow-up with actual tickets on the work items
All of this is brainstorming and vaporware, but I can't help to think of better ways of doing this. If somebody has experience with any existing systems and wants to share if and how they might fit, or what other ideas are useful to borrow: Please share!
Regards, Harald
On Thu, Mar 09, 2017 at 02:08:23AM +0100, Harald Welte wrote:
And rather than adding all the missing bits and pieces with hand-crafted code to the control interface, we should have a generic way to define a new configuration value, and that value should then be automatically parse- and storable via VTY as well as a programmatic interface.
+1
"stall" the main application by performing I/O intensive operation on the VTY
I once used a script that connects to nitb's vty, prints the lchan summary and exits. I ran that once a second. It often managed to crash the NITB! ... I never actually investigated it, but would have been good to do so.
Remembering the port numbers is clumsy. It would be more convenient if a single command line interface could provide access to the configuration and the state of multiple different processes / network elements.
[...]
This API is both used by the actual Osmocom programs to obtain their configuration (and obtain updates to it during runtime), as well as by the "VTY daemon" providing interactive shell access to it.
So would the central interface simply remember the port numbers for us? No, you're talking about a "config server" that the individual applications connect to. On the one hand nice, on the other yet another osmo-entity. Still, +1.
Would osmo-programs require the config server to run? Would they be able to launch the config daemon automatically -- portably?
GET SET NOTIFY ENUM, integers with ranges, etc. machine-parseable description of a "MIB/config item", together with syntax, ranges, enum values, help texts, etc (ideally in the C source code, possibly in comments?)
Add to the wishlist: an indicator whether changing the value immediately changes the application's behavior. A boolean unfortunately doesn't cut it, e.g. some settings take effect when a specific BTS' OML link is restarted. It could have the values "yes, immediate", "no, needs restart" and "please see comment" with a separate comment provided for humans.
With global structs like gsm_network, I could imagine a completely automagic code generation from API comments. The description could contain the global gsm_network instance's symbol name, and the machine built code could provide the NITB's side as well as the config server's side, in the NITB storing those values directly in the global gsm_network instance. Or maybe let the automagic code create a replacement for the config part of the gsm_network struct. That's all easy for singleton items.
The default NOTIFY could invoke a simple GET (sometimes storing a value in a struct is enough to change behavior immediately), but a manual on_notify() can be specified (presence of on_notify() doesn't guarantee immediate effect).
More complex would be things like fetching a subscriber with a given IMSI first and applying changes there. Arbitrary complexity of actions to update a value are thinkable, so we probably won't get around writing it manually? Semi?
But hold on, there are different cases: we don't want to store individual IMSIs in the config server (that's what the HLR is for), while we do want to store individual BTS configurations centrally.
For GET/SET/NOTIFY of a BTS element, we would need the API to have an array index argument, analogous to 'bts 0', 'bts 1'. For 'trx 0' etc we even need nested array indexes. With a path scheme? GET('bts[2].trx[0].arfcn').
But for operations on a subscriber, we would instead merely send instructions to the HLR to do/store/say things -- if we want to allow managing subscribers from the central "vty" application, and if we want to allow querying the current state of entities (like 'show lchan summary'), we need another class of items that don't GET/SET things from/to the central config server, but tell an application to do or say something. Call them COMMAND, and they would feature arbitrary arguments. In the VTY we just implemented whatever we want to happen, in this scheme we should explicitly differentiate config items from commands.
When we have separated BSC from NITB for good, centrally issued COMMANDs could also make up for the removed information match-up: in the NITB we always knew which BTS a subscriber was calling from with all of the detailed BTS state. With a separate BSC we don't anymore -- but the central "vty" could run queries with MSC and BSC to match up that information conveniently again.
All of this is brainstorming and vaporware
It would be nice to dabble up some hacky code to discuss about, to sharpen our teeth on; without spending too much time on it and having in mind to discard it for something better anytime?
Sounds a bit like re-inventing dbus. (I don't know much about dbus though, would it make sense to use dbus?? Is it a giant kraken of dependencies?)
Seems to me that with all the wishlist features we have collected now, the implementation of such a system would actually be quite complex and not a small effort.
I think the most important feature is that all our config items become a machine and human API at the same time -- and that is a killer feature for all those folks asking for web interfaces.
~N
On 9 March 2017 14:41:37 CET, Neels Hofmeyr nhofmeyr@sysmocom.de wrote:
I once used a script that connects to nitb's vty, prints the lchan summary and exits. I ran that once a second. It often managed to crash the NITB!
I do that often, once even forgot and left it running in a screen a few days. It didn't crash. I cat show lchan to nc and have to make it wait for 1 second otherwise it sometimes misses to vty output, (minimum wait is 1 second with nc) Sometimes i would like to poll lchan faster. Anyway, not an ideal solution for monitoring the lchans, as you know. Fairwaves have something using the control port, but maybe expanding the meas feed is a better way to do this kind of thing?
On 9 March 2017 14:41:37 CET, Neels Hofmeyr nhofmeyr@sysmocom.de wrote:
I once used a script that connects to nitb's vty, prints the lchan summary and exits. I ran that once a second. It often managed to crash the NITB!
I do that often, once even forgot and left it running in a screen a few days. It didn't crash. I cat show lchan to nc and have to make it wait for 1 second otherwise it sometimes misses to vty output, (minimum wait is 1 second with nc) Sometimes i would like to poll lchan faster. Anyway, not an ideal solution for monitoring the lchans, as you know. Fairwaves have something using the control port, but maybe expanding the meas feed is a better way to do this kind of thing?
Hi Neels,
On Thu, Mar 09, 2017 at 02:41:37PM +0100, Neels Hofmeyr wrote:
I once used a script that connects to nitb's vty, prints the lchan summary and exits. I ran that once a second. It often managed to crash the NITB! ... I never actually investigated it, but would have been good to do so.
this should not happen, and I don't see a reason why it would. There's no concurrency and no race conditions, everything runs synchronously in one thread. I've not observed such behavior so far (luckily). but it should be a relatively trivial scriptable test case: "vty load generation" ;)
So would the central interface simply remember the port numbers for us? No, you're talking about a "config server" that the individual applications connect to. On the one hand nice, on the other yet another osmo-entity. Still, +1.
I don't really see where the problem lies with 'yet another process'. People have been using zebra+quagga for some 20 years by now, and they always came as a suite of processes. And that was before systemd took care of re-spawning them ;)
I mean seriously, on my laptop I have 195 kernel threads cluttering my 'ps' output already, why would I care about a hand ful of osmocom processes?
Would osmo-programs require the config server to run?
yes.
Would they be able to launch the config daemon automatically -- portably?
I would probably not bother but simply print a meaningful error message and assume that people get their systemd rules right in terms of the dependencies?
GET SET NOTIFY ENUM, integers with ranges, etc. machine-parseable description of a "MIB/config item", together with syntax, ranges, enum values, help texts, etc (ideally in the C source code, possibly in comments?)
Add to the wishlist: an indicator whether changing the value immediately changes the application's behavior.
We could, but I would rather use such a massive change to do it properly, i.e. implement the 'immediate reaction' to value changes in the respective programs.
A boolean unfortunately doesn't cut it, e.g. some settings take effect when a specific BTS' OML link is restarted.
Yes, those are bugs / shortcomings of the code and should be adressed by people caring sufficiently about it? We can re-generate the BCCH/SACCH filling at any time and update them in the BTS. We can also re-configure the OML Managed Objects at runtime. The protoocl permits all of that.
With global structs like gsm_network, I could imagine a completely automagic code generation from API comments. The description could contain the global gsm_network instance's symbol name, and the machine built code could provide the NITB's side as well as the config server's side, in the NITB storing those values directly in the global gsm_network instance. Or maybe let the automagic code create a replacement for the config part of the gsm_network struct. That's all easy for singleton items.
Yes, I would go for separate config structs, and hand a newly-allocated one to the application, so the application can check for differences between old and new config, if neded, before applying the new config.
More complex would be things like fetching a subscriber with a given IMSI first and applying changes there. Arbitrary complexity of actions to update a value are thinkable, so we probably won't get around writing it manually? Semi?
That's not configuration data, is it? I would not try to do subscriber management within the MIB.
But hold on, there are different cases: we don't want to store individual IMSIs in the config server (that's what the HLR is for), while we do want to store individual BTS configurations centrally.
exactly.
For GET/SET/NOTIFY of a BTS element, we would need the API to have an array index argument, analogous to 'bts 0', 'bts 1'. For 'trx 0' etc we even need nested array indexes. With a path scheme? GET('bts[2].trx[0].arfcn').
yes. That's one part that OpenWRT's uci does ver nicely, but it has weak types (or none at all?) and is unsuitable in many other ways. Also, I think uci has limited path depth, which isn't nice either.
But for operations on a subscriber, we would instead merely send instructions to the HLR to do/store/say things -- if we want to allow managing subscribers from the central "vty" application, and if we want to allow querying the current state of entities (like 'show lchan summary'), we need another class of items that don't GET/SET things from/to the central config server, but tell an application to do or say something. Call them COMMAND, and they would feature arbitrary arguments. In the VTY we just implemented whatever we want to happen, in this scheme we should explicitly differentiate config items from commands.
yes. There are many details to be resolved, though. We could simply require each application to provide pretty-print functions for all of its 'introspectable structs's and then build a library/plugin that the "VTY application" can load to print even the "raw" binary structs. We'd just have to prefix them with some kind of type annotation (in the protocol, not at runtime in all processes). This way even the printf() effort is kept out of the respective applications, further reducing load that adminstrative interfaces can put on them.
When we have separated BSC from NITB for good, centrally issued COMMANDs could also make up for the removed information match-up: in the NITB we always knew which BTS a subscriber was calling from with all of the detailed BTS state. With a separate BSC we don't anymore -- but the central "vty" could run queries with MSC and BSC to match up that information conveniently again.
possibly, but that sounds rather more comples.
All of this is brainstorming and vaporware
It would be nice to dabble up some hacky code to discuss about, to sharpen our teeth on; without spending too much time on it and having in mind to discard it for something better anytime?
Yes, but not now [TM]. I at least think I need to spend lots of more time thinking about this (in the bathtub or elsewhere).
Sounds a bit like re-inventing dbus. (I don't know much about dbus though, would it make sense to use dbus?? Is it a giant kraken of dependencies?)
I don't think dbus is a configuration management system, but an IPC system. And yes, we'd need some kind of IPC, but I would ignore/defer the exact IPC method/transport for now, as it is not the key "idea" under discuission.
Seems to me that with all the wishlist features we have collected now, the implementation of such a system would actually be quite complex and not a small effort.
yes. The question is if we can find something that is possible to do in stages/milestones, rather than changing everything at once. Or some elegant hack that provide at least some of the features we need at limited effort.
I think the most important feature is that all our config items become a machine and human API at the same time -- and that is a killer feature for all those folks asking for web interfaces.
exactly. Whether we do this still inside the processes or in an external central process is a minor detail.
Also, what I'd really like is to see the generalized read/parse and save code, avoiding all the possible manual mistakes we have at places. Finally, being able to do away with the "vty-additions.xml" files but having more documentation at the same place where the config items are defined/declared is better and will make it feasible to have more documentation in the vty reference manuals.
What I find very surprising is that nobody seems to have come up with something like this already. But what can be found is miles far away from what we're thinking of. There's libconfuse, uci, libcli, clish, ... - all of them seem to have some interesting ideas, but fall quite short of a proper system. And then there's things like SNMP and net-snmp which are ugly and complex.
The IETF has come up with OpenConfig (and NETCONF and RESTCONF protocols) which goes very much into the direction I've been thinking of. But it seems they've mostly been working on architecture as well as OpenConfig models for regular IP-type network element configuration, (writtein in YANG, a language they invented for that purpose) and protocols, but I haven't seen any FOSS implemenataions yet? http://jedelman.com/home/openconfig-data-models-and-apis/ is a good intro into this part of the networking universe. Fully buzzword compatible ;) And again very heavy and complex. If there were many implementations out there, it might be worth investigating - but there aren't.
Regards, Harald
Hi Harald,
A very interesting topic indeed.
On Mar 9, 2017 4:15 AM, "Harald Welte" laforge@gnumonks.org wrote:
So what I had in mind for quite some time (actually since netconf 1.2 about one year ago), is to have some kind of an external "VTY/MIB daemon" (or even separate daemons for each) which maintains a hierarchical database of configuration values. The MIB deamon simply offers an API (via client library) to GET or SET the individual values, or to NOTIFY an application about a changed value. This API is both used by the actual Osmocom programs to obtain their configuration (and obtain updates to it during runtime), as well as by the "VTY daemon" providing interactive shell access to it. Finally, other external applications could use the same interface/client library to do the same. A proxy to SNMP or REST interfaces can be imagined, for even more interfacing with the outside world.
Right now VTY and control interface serve three functions: 1. Configuration ("configure terminal") 2. Object manipulation (eg "subscriber IMSI .. active ...") 3. Information query with polling and traps (eg "show ..." and control interface)
A centralized configuration DB massively works with use case (1), but doesn't really fit use cases (2) and (3) IMHO. You can do it, but then it'll serve just as a proxy to the actual application and may become a quite bloated and complicated piece of code.
Running another instance of yet another daemon is also something I personally would really like to avoid looking from a user/admin perspective. Number of processes we're already running on a NITB BTS is already getting out of hand :(
So stealing an idea from FreeSWITCH and building on top of it:
Why don't we keep the configuration DB inside each app, but expose a machine readable API only? Then VTY becomes just one of many applications talking to this API on par with web-ui, scripts and whatnot.
This API could be something like JSON or something more binary (Erlang terms binary format, BERT, etc). We can even make it more REST-like with only GET/SET/NOTIFY or more RPC like with a richer set of primitives. There are pros and cons in all these approaches, but the key point is that: 1. This works with all 3 use cases above 2. It's less intrusive into the current code base (I think) 3. Allows to have a single CLI/VTY app talking to any Osmocom app. Btw, the app can know default ports of standard apps and allow connection by neg let name like: $ osmo-cli -n osmo-bts 4. The app can still do TAB completion and help in a VTY way - FreeSWITCH handles this perfectly with its fs_cli.
What do you think? It's a completely different approach than a config DB, so I wanted to through this idea in without too much details to see if it sticks.
Please excuse typos. Written with a touchscreen keyboard.
-- Regards, Alexander Chemeris CTO/Founder Fairwaves, Inc. https://fairwaves.co
On Thu, Mar 09, 2017 at 10:20:11PM +0300, Alexander Chemeris wrote:
Right now VTY and control interface serve three functions:
- Configuration ("configure terminal")
- Object manipulation (eg "subscriber IMSI .. active ...")
- Information query with polling and traps (eg "show ..." and control
interface)
A centralized configuration DB massively works with use case (1), but doesn't really fit use cases (2) and (3) IMHO. You can do it, but then it'll serve just as a proxy to the actual application and may become a quite bloated and complicated piece of code.
I have some vague ideas here but need to think further about it before I think it's ready for an actual proposal.
Running another instance of yet another daemon is also something I personally would really like to avoid looking from a user/admin perspective. Number of processes we're already running on a NITB BTS is already getting out of hand :(
I don't see the problem. On modern systems it is simply a matter of a systemd job configuring that daemon as a dependency for the other daemons. And if you prefer some other system like upstart, it is basically the same. I don't think any user cares about the number of processes, at least not until you're talking about hundreds of them.
Why don't we keep the configuration DB inside each app, but expose a machine readable API only? Then VTY becomes just one of many applications talking to this API on par with web-ui, scripts and whatnot.
It's another idea, indeed. I guess to a large extent the same work is required for both my or your proposal: You need some data structures that describe the syntax of the configuration values, their permitted ranges, types, help texts, etc. The difference is then just where this configuration is stored / "owned". I like the idea to have it external, as some paramters can then easily be shared between processes. Let's say the 'gsm_network' configuration, i.e. things like MCC, MNC. You curently have to enter that on both the SGSN and the MSC, but this could be stored just as one 'network' config object that multiple processes refer to.
What do you think? It's a completely different approach than a config DB, so I wanted to through this idea in without too much details to see if it sticks.
Well, see above, I think it's more or less the same code, but running inside the same process vs. running it externaly?
Hi.
I won't claim in-depth experience with MIB frameworks, just want to contribute few buzzwords into brainstorming: NETCONF - rfc 6241 https://tools.ietf.org/html/rfc6241 YANG - rfc rfc 6020 https://tools.ietf.org/html/rfc6020
The former is IETF protocol for configuring network devices [1]. The latter is data modeling language describing device's configuration and state, manipulated by netconf [2].
From brief overview those seems to match well with Harald's outline. There might be few pieces still missing but it seems doable. I see following advantages in adopting netconf & yang: - it's ietf standard - there are open source (including GPL) implementations we can play with [3, 4] - it's already widely supported by telecom vendors so chances are in operators using Osmocom stack, will be some people familiar with it - it's SDN/NFV compatible which makes it future-proof - there's proposed extension RESTCONF which make integration with creepy web things like SOAP much easier [5] - and most important for me personally - there's Emacs mode for it [6] :-)
We're not the first to feel the need for proper MIB/configuration subsystem, and the task indeed looks huge. So the more we can rely on work already done in this field, the better. Even if we have to write some things, by relying on open and widely adopted standards we increase our chances of getting external contributions which is always nice.
[1] https://trac.ietf.org/trac/netconf [2] http://www.yang-central.org/twiki/bin/view/Main/WebHome [3] http://ensuite.sourceforge.net/ [4] https://github.com/CESNET/libnetconf [5] https://tools.ietf.org/html/draft-bierman-netconf-restconf-04 [6] https://www.emacswiki.org/emacs/yang-mode.el
Hi.
A little follow-up on this topic after OsmoDevCon. I've stumbled upon following project http://www.clicon.org/
From a quick glance over their tutorial at http://www.clicon.org/clicon_tutorial.pdf
it seems to fulfill almost all the properties discussed at OsmoDevCon: * autogeneration of cli and conf from single source * transactions support * both human-oriented cli and machine-oriented protocol from a single source
Added bonus is that it's available under GPL (dual-license actually): https://github.com/clicon/clicon
The only downside I see so far (besides the sheer amount of work for migration) is that this project is not widely used (yet?). Another thing I have not understood without digging further is how to apply some configs/commands in runtime without restarting the backend (OsmoBSC, OsmoMSC etc).
All-in-all it seems like smth definitely worth looking into.
-
Alexander Chemeris -
Harald Welte -
k＠rhizomatica.org -
Keith Whyte -
Max -
Neels Hofmeyr