I welcome a solution like this very much. However, it wouldn't be a real solution to the human vs machine interaction issue. There are a lot of cans of worms hidden in this problem space and our status quo.

The notion that we should only use the CTRL interface for automated interaction sounds like a good idea, but is quite impractical IMO. For TTCN3 tests or at congress, for example, it is so simple to just interact with the VTY instead of first implementing CTRL commands that do the same thing (and merging patches and so on), and the CTRL also has some problems in its protocol design... So in my own opinion / daily practice, I have kind of let go of this notion that the VTY is for humans only.

What you are proposing is an interesting solution for structured input into a program, but there is no easy way for *querying* structured information from a program's VTY existing implementation in that way. We can't only expose those VTY items that don't return any output (we have no way to distinguish those -- so we also can't easily translate GET and SET to VTY commands), so this will probably just return the same vty_out() as a VTY command would print, only on CTRL. That kind of obsoletes the CTRL interface, might as well just use the VTY, if it returns the same output anyway. Related to that is the lack of structure in the VTY: we don't formally compose a response and send it, instead we vty_out() maybe N times, maybe not... We'd have to redirect the vty_out() to CTRL somehow.

There's also the question of receiving larger responses, think a list of subscribers -- so far CTRL clients expect the entire response in one read(). I'm not entirely sure whether it is a fundamental flaw in the CTRL implementation, but AFAIR I have often looked at CTRL clients and found a lack of structuring there, on a protocol level. It would certainly be a problem if a multi-packet response by accident had a "GET_REPLY" at the start of a subsequent packet. We don't properly escape response content...

Doing both input and querying properly would IMO mean a lot of re-implementing. Every now and then I thought a bit about maybe implementing an object model where a command returns certain named values / lists thereof, and the VTY is then a renderer to put those into human readable formats, while the CTRL would return those values raw in a nicely parseable way (a bit like SQL responses). But that would also mean quite a lot of more effort to implement a VTY command compared to the simple C-code and vty_out() we use now. (let alone re-implementing every VTY command we have so far.)

In the end I so far mostly conclude that it is too much effort to rewrite the entire VTY, and that it is simplest / most practical to actually ignore the CTRL interface completely, also for machine interaction :/

~N

On Tue, Feb 16, 2021 at 04:23:12PM +0100, Harald Welte wrote:

Dear all,

we always had said the VTY is a human CLI, while the CTRL interface is for interfacing with other software. This was primarily related to asking users to not parse the VTY output of "show" commands or the like. For configuration via VTY, we have to provide a stable interface anyway, as otherwise loading config files would not be possible across [non-major] software upgrades.

While that is right in principle, the fact that we don't have a generic configuration store / API / MIB means that one would explicitly have to add CTRL command for all relevant settings, which may have been realistic in 2010, but is unrealistic with the hundreds of VTY configuration parameters today.

I was thinking wither it might make sense to add a generic CTRL interface GET/SET for "configuration" mode VTY settings. This would mean we write code once, and immediately expose all our VTY nodes to CTRL.

Thinking of the following example done via VTY:

---

OsmoBSC> enable OsmoBSC# configure terminal OsmoBSC(config)# network OsmoBSC(config-net)# bts 0 OsmoBSC(config-net-bts)# trx 1 OsmoBSC(config-net-bts-trx)# timeslot 0 OsmoBSC(config-net-bts-trx-ts)# training_sequence_code 3

---

We could have something like the following CTRL command:

SET network.bts.0.trx.1.timeslot.0.training_sequence_code 3

The devil is a bit in the details of the syntax

• the normal '.' as separator looks generally OK, I think we have [almost?] no VTY commands that include a ','

• we somehow need some kind of separator to tell where individual VTY commands are separated, like

  SET network."bts 0"."trx 1"."timeslot 0".training_sequence_code 3

or SET network|bts.0|trx.1|timeslot.0|training_sequence_code 3 or SET network#bts.0#trx.1#timeslot.0#training_sequence_code 3 or SET network.bts#0.trx#1.timeslot#0.training_sequence_code 3 SET network.bts@0.trx@1.timeslot@0.training_sequence_code 3

I think the last examples would be more "natural" for existing CTRL interface code, as the dot separat separates tokens, and we simply expand the '#' or '@' to spaces in the context of VTY commands

The generic vty-cfg-via-CTRL code then would basically emulate a user entering the sequence of commands - including the "exclusion" of not allowing multiple concurrent VTY or vty-cfg-via-CTRL users at the same time

What do you guys think?

Regards, Harald

--

• Harald Welte laforge@osmocom.org http://laforge.gnumonks.org/

============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)

On 16/02/2021 09:23, Harald Welte wrote:

Dear all,

we always had said the VTY is a human CLI, while the CTRL interface is for interfacing with other software. This was primarily related to asking users to not parse the VTY output of "show" commands or the like.

You mean like:

expect -c 'spawn -noecho telnet 0 4260; expect >; send enable\r; expect #; send "show pdp-context ggsn ggsn0\r"; expect #' | egrep 'IMSI|Control|End-User' | gawk 'BEGIN { RS="IMSI" } /./ { sub(/:[^$]*/, "", $10); sub(/\r/, "", $14); print "\033[32;1m SGSN: \  "$10"\033[30D\033[20C \033[33;1mIP: "$13" "$14"\033[36;1m \033[100D \033[50C IMSI: "$2"\033[1D \033[0m" }' | sort -t. -n -k3,3n -k4,4n -k6,6n -k7,7n

:))

Yes, I'm guilty of a lot of this...

But this proposal doesn't actually seem to be about that, but rather about an automagic way to expose anything added to the config nodes with SET...

SET network.bts#0.trx#1.timeslot#0.training_sequence_code 3

which sounds good to me, and I like the dot separator. and replacing space with #.

I don't think I have a lot of everyday use for modifying runtime configuration, although I have thought recently that maybe the best way to recover quickly from a power failure induced msc restart and loss of volatile VLR is to assign two LAC to each site and alternate them on restarts, forcing all the MS to LU as soon as the network is back up.

I would probably do this by modifying a config file (with sed or such) before starting the daemons, but maybe a CTRL command is better, which would have to be followed by a BTS SI update, right?  but I don't mean to hijack the thread with that.. :)

Anyway, Matt at UW recently added osmopy/osmo_ipa to the rhizomatica code, so maybe we'd make more use of CTRL in future.

k.

Hi,

SET network|bts.0|trx.1|timeslot.0|training_sequence_code 3 or SET network#bts.0#trx.1#timeslot.0#training_sequence_code 3

I'd probably go for this one from all the ones you proposed. This way, you keep the usual CTRL structure ("." separating/slitting parameters), while you simply add a new character to specify "levels" or "nodes" (as per VTY terminology).

Hence, each node is parsed the same way as the final command, and you can "easily" feed that into navigating the existing VTY tree. Example in pythonic way:

str = "network|bts.0|trx.1|timeslot.0|training_sequence_code 3" nodes = str.split('|') for node in nodes: argv = node.split('.') if attempt_to_match_vty_node_or_command(argv) < 0: return cmd_reply_error; return

The question here is how to define information in code to apply getter/setters for CTRL commands...

I also find that the CTRL format in general is a bit restraining, both in specifying parameters, etc. as well as in reply content. Somebody may not like the idea, (and may be not directly related to the topic at hand), but perhaps looking at something similar to a REST interface which provides json output and which can easily be interacted using python or even curl directly. REST interface provides already way to provide levels/nodes (URL directories)as well as parameters "?foo=bar&hello=bye"). So using HTTP may not be needed, but it may make sense to reuse some REST related ideas, like URLs and json output (with C implementation in 1 file like cjson).

Regards, Pau

On Tue, Feb 16, 2021 at 04:23:12PM +0100, Harald Welte wrote:

SET network.bts.0.trx.1.timeslot.0.training_sequence_code 3

The devil is a bit in the details of the syntax

• the normal '.' as separator looks generally OK, I think we have [almost?] no VTY commands that include a ','

• we somehow need some kind of separator to tell where individual VTY commands are separated, like

  SET network."bts 0"."trx 1"."timeslot 0".training_sequence_code 3

or SET network|bts.0|trx.1|timeslot.0|training_sequence_code 3 or SET network#bts.0#trx.1#timeslot.0#training_sequence_code 3 or SET network.bts#0.trx#1.timeslot#0.training_sequence_code 3 SET network.bts@0.trx@1.timeslot@0.training_sequence_code 3

We also have numerous commands where there are spaces that would look weird when written as e.g. # or @:

network handover1 power budget hysteresis (<0-999>|default)

We have commands where the values are interleaved with keyword tokens, i.e. where there is no clean name -> value relation:

neighbor lac <0-65535> arfcn <0-1023> bsic (<0-63>|any)

We have commands that are additive:

si2quater neighbor-list add earfcn <0-65535> thresh-hi <0-31> thresh-lo <0-32> prio <0-8> qrxlv <0-32> meas <0-8> si2quater neighbor-list del earfcn <0-65535>

Commands without a value part:

allow-attach no periodic location update

Trying to translate all of these cases into CTRL-like name->value feels very hacky to me, with many quirks and pitfalls waiting on us around numerous corners. We'll run into ambiguities, escaping issues, and unreadable commands.

A cleaner approach seems to me to have a CTRL command that invokes VTY lines, with a defined line separator, with VTY commands as-is in the "value" part of the CTRL command. E.g.:

SET config network;bts 0;trx 1;timeslot 0;training_sequence_code 3

Then we don't need to worry about translating VTY commands to valid CTRL identifiers, and except for concatenating multiple lines the VTY commands can be run 1:1 without translation mangling in-between. Many problems gone.

As line separator, ';' comes to mind from common programming languages, or '%' or '#' since those two are otherwise used for comments in the VTY and will not likely cause ambiguities. We could even send LF as line separator like:

SET config network bts 0 trx 1 timeslot 0 training_sequence_code 3

so that the "value" part of the CTRL command is *exactly* the same as we would feed to a telnet client.

At this point I'm asking myself again, what is the aim here -- an external program that can connect to CTRL might as well also connect to VTY and can already issue these config commands there. Is the aim to save one connection?

I still have the impression that this idea is well meant but in reality misses practicality and usefulness. It's just a one-shot telnet client piped through CTRL, in the worst case with lots of mangling involved. where is the benefit?

Would be nice if it made sense and things would fall into place, but AFAICT it unfortunately simply doesn't work out. VTY as we implemented it is too free-format and unstructured, we won't be able to tame it well.

~N