I've been investigating the availability of KPIs on the control interfaces. Is there a way to enumerate which metrics and counters exist? What would be the best way to do so? Thinking it would be a useful to add a list function to add to the VTY and/or the CTRL interface itself.
On Tue, Jan 22, 2019 at 04:19:50PM +0000, Omar Ramadan wrote:
I've been investigating the availability of KPIs on the control interfaces. Is there a way to enumerate which metrics and counters exist? What would be the best way to do so? Thinking it would be a useful to add a list function to add to the VTY and/or the CTRL interface itself.
Hi Omar,
though I have no answer ready, I briefly skimmed the code, and it looks to me like we have CTRL interface commands ready to query rate counters etc, but I couldn't find commands to actually list the available ones.
Also the VTY contains tailored counter listing commands in various programs but lacks a generic list of available names, AFAICT. I'm not 100% sure, maybe I missed it.
What comes to mind though is that at the CCC congress in december, we used the stats export which feeds all those counters to an external statistics interface, and there we could browse the available KPIs with a web interface, being able to compose dashboards in a javascript clickety way, combining KPIs with formulas and getting all sorts of graphs, fancy stuff like that. I forget the name, was it a graphite server? I'm not familiar with the details, but that could be interesting to explore.
In almost all the core network .cfg you can place a config like
stats reporter statsd disable remote-ip ${STATSD_IP} remote-port 9125 level global no prefix enable stats interval 5
and IIUC run a statsd instance to forward to a graphite ... <wave hands> ... stopping here because I didn't set it up and don't know the details, I hope someone else can take over here and provide pointers.
~N
Hi Neels,
[this message is not adressed specifically to you, but to the wider community]
On Wed, Jan 23, 2019 at 02:49:08AM +0100, Neels Hofmeyr wrote:
though I have no answer ready, I briefly skimmed the code, and it looks to me like we have CTRL interface commands ready to query rate counters etc, but I couldn't find commands to actually list the available ones.
See my other response.
In almost all the core network .cfg you can place a config like
stats reporter statsd disable remote-ip ${STATSD_IP} remote-port 9125 level global no prefix enable stats interval 5
The problem is that this appears to have zero documentation beyond the API reference of libosmocore, which I pointed out in my other response on this thread. What we need is a section in the "common" part of the manuals explainig this mechanism in detail. I created https://osmocom.org/issues/3768 for this.
This also brings me to http://osmocom.org/issues/3670 "TTCN-3 test for statsd exporter" which are missing so far.
Finally, https://media.ccc.de/v/SE8HRK is a talk by daniel on the related features which may help as background information.
Any volunteers? Seems like a very important topic operationally, but nothing anyone has been contributing any effort towards so far :/
Regards, Harald
Dear Omar,
On Tue, Jan 22, 2019 at 04:19:50PM +0000, Omar Ramadan wrote:
I've been investigating the availability of KPIs on the control interfaces. Is there a way to enumerate which metrics and counters exist? What would be the best way to do so? Thinking it would be a useful to add a list function to add to the VTY and/or the CTRL interface itself.
I'm sorry that nobody has so far responded to this question. I thought this had long been taken care of :/
The list of counters of every program can be displayed/exported if you enter "show asciidoc counters", which is what we use to generate the tables you find in the manuals of the respective program.
To get some more background about the individual counter types:
* osmo_stat_item http://ftp.osmocom.org/api/latest/libosmocore/core/html/group__osmo__stat__i...
* rate_counter http://ftp.osmocom.org/api/latest/libosmocore/core/html/group__rate__ctr.htm...
* osmo_counter (old, more or less deprecated) http://ftp.osmocom.org/api/latest/libosmocore/core/html/structosmo__counter....
* osmo_stats_reporter http://ftp.osmocom.org/api/latest/libosmocore/core/html/group__stats.html#de...
We have two of those in libosmcoore: One that reports to the logging sub-system, and another one that reports to statsd / behaves like a statsd exporter
For rate counters, there's an example script on how to export them at http://git.osmocom.org/python/osmo-python-tests/tree/scripts/osmo_rate_ctr2c...
If you have any more specific questions, please follow-up here. Thanks!
Regards, Harald
-
Harald Welte -
Neels Hofmeyr -
Omar Ramadan