9 Mar
2017
9 Mar
'17
2:08 a.m.
Hi All,
Using zebra/quagga VTY code was a great idea back then and has served us
nicely in all those years. I like the general style of the command-line
based interface with its various nodes, the strict syntax checking, the
tab completion and the interactive context-sensivive help.
However, it feels a bit 20ieth-century-ish to have to manually write
code to parse and to save the respective values. This is not a
productive way to spend our development resources, and it is error
prone. The "save" can be forgotten, resulting in non-saveable config
parameters. The save can store values that the "parse" function will
not be able to read again, etc.
Furthermore, the VTY is inherently a human user interface, not intended
for programmatic consumption. For programmatic access, we have
developed the control interface. In reality though, only 1% of the
parameters available in the VTY are exported via the control interface.
And rather than adding all the missing bits and pieces with hand-crafted
code to the control interface, we should have a generic way to define a
new configuration value, and that value should then be automatically
parse- and storable via VTY as well as a programmatic interface.
The next "problem' is that the current telnet connections live within
the process of the application. This means that a user can effectively
"stall" the main application by performing I/O intensive operation on
the VTY, or even on as many VTY/telnet sessions as he wants. There
shouldn't be any need for this, at least not for something as mundane as
performing configuration changes. Of course the VTY has different use
cases such as runtime introspection of system state as well as logging.
But still.
Yet another concern is "VTY and telnet port proliferation".
Particularly with the conversion from NITB to BSC+MSC+HLR, we are yet
again getting more network elements with their own telnet ports.
Remembering the port numbers is clumsy. It would be more convenient if
a single command line interface could provide access to the
configuration and the state of multiple different processes / network
elements.
Last, but not least, the current implementation is fixed to telnet,
without any form of authentication, and without a path to migrate e.g.
to something like SSL or SSH. I don't think it is a major concern (you
can always SSH to the system and then telnet locally).
So what I had in mind for quite some time (actually since netconf 1.2
about one year ago), is to have some kind of an external "VTY/MIB
daemon" (or even separate daemons for each) which maintains a
hierarchical database of configuration values. The MIB deamon simply
offers an API (via client library) to GET or SET the individual values,
or to NOTIFY an application about a changed value. This API is both
used by the actual Osmocom programs to obtain their configuration (and
obtain updates to it during runtime), as well as by the "VTY daemon"
providing interactive shell access to it. Finally, other external
applications could use the same interface/client library to do the same.
A proxy to SNMP or REST interfaces can be imagined, for even more
interfacing with the outside world.
What I don't like about many existing MIBs I've used (as a
sysadmin/user, not a developer) is their simple type system. You can
often specify any random value to such a MIB, even one that is
completely useless. A simple INT or STRING type is not sufficient, we
really want the ability to have ENUM types, to have integers with
ranges, etc. - just like we have in the VTY.
Also, the MIBs I've used typically are nothing more than a hierarchical
key-value store. They do not contain the information required for
interactive, context-sensitive help needed for the "VTY" feel :( This
is btw also the problem I have with OpenWRT's uci: It just has keys and
values, with no way to constrain those values to something that's
reasonable to the given program/context that is being configured, and
there's no help or other information associated with those keys and
values.
So what I would like to see in this context, is some way to have a
machine-parseable description of a "MIB/config item", together with
syntax, ranges, enum values, help texts, etc (ideally in the C source
code, possibly in comments?) which is used by some kind of MIB compiler
to build up the hierarchical structure of the MIB and all of its keys as
well as their permitted values and syntax reference. This information
is then available to the VTY/telnet connections, irrespective of whether
a given application [using those values] is running at all.
Once an application starts, it can query for the configuration values it
is interested in and populate its internal data structures from it.
Ideally, one would even be able to generate a 'struct' from the MIB
compiler, so that there is no need to explicitly call a "get" function
on each single configuration value, but one simply gets a C-language
'struct' with all the values filled in by the client library.
As stated above, there should be some way how he MIB can notify
applications about changes to certain nodes in the MIB tree, so the
application[s] can react to that. I don't have a clear picture yet how
transaction logic (like changing multiple values and then committing
them at once) would work, or whether applications should even be able to
reject/revert a modification?
In either case, I just wanted to share my thoughts on this. I know it
sounds rather complex, but I think the investment in some kind of
unified configuration database system would save us of a lot of
boilerplate code and subtle bugs and inconsistencies.
I've created https://osmocom.org/issues/1975 with above text to keep
track of it, but I think at this point it's more of a mailing list
discussion rather than something we can put into an issue. I guess the
progress would rather be
* first discuss it here and/or at OsmoDevCon
* create something like a specification in the wiki
* then follow-up with actual tickets on the work items
All of this is brainstorming and vaporware, but I can't help to think of
better ways of doing this. If somebody has experience with any existing
systems and wants to share if and how they might fit, or what other
ideas are useful to borrow: Please share!
Regards,
Harald
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
9 Mar
9 Mar
2:41 p.m.
On Thu, Mar 09, 2017 at 02:08:23AM +0100, Harald Welte wrote:
And rather than adding all the missing bits and pieces
with hand-crafted
code to the control interface, we should have a generic way to define a
new configuration value, and that value should then be automatically
parse- and storable via VTY as well as a programmatic interface.
+1
"stall" the main application by performing
I/O intensive operation on
the VTY
I once used a script that connects to nitb's vty, prints the lchan summary and
exits. I ran that once a second. It often managed to crash the NITB! ... I
never actually investigated it, but would have been good to do so.
Remembering the port numbers is clumsy. It would be
more convenient if
a single command line interface could provide access to the
configuration and the state of multiple different processes / network
elements.
[...]
This API is both
used by the actual Osmocom programs to obtain their configuration (and
obtain updates to it during runtime), as well as by the "VTY daemon"
providing interactive shell access to it.
So would the central interface simply remember the port numbers for us? No,
you're talking about a "config server" that the individual applications
connect
to. On the one hand nice, on the other yet another osmo-entity. Still, +1.
Would osmo-programs require the config server to run?
Would they be able to launch the config daemon automatically -- portably?
GET SET NOTIFY
ENUM, integers with ranges, etc.
machine-parseable description of a "MIB/config item", together with
syntax, ranges, enum values, help texts, etc (ideally in the C source
code, possibly in comments?)
Add to the wishlist: an indicator whether changing the value immediately
changes the application's behavior. A boolean unfortunately doesn't cut it,
e.g. some settings take effect when a specific BTS' OML link is restarted.
It could have the values "yes, immediate", "no, needs restart" and
"please see
comment" with a separate comment provided for humans.
With global structs like gsm_network, I could imagine a completely automagic
code generation from API comments. The description could contain the global
gsm_network instance's symbol name, and the machine built code could provide
the NITB's side as well as the config server's side, in the NITB storing those
values directly in the global gsm_network instance. Or maybe let the automagic
code create a replacement for the config part of the gsm_network struct. That's
all easy for singleton items.
The default NOTIFY could invoke a simple GET (sometimes storing a value in a
struct is enough to change behavior immediately), but a manual on_notify() can
be specified (presence of on_notify() doesn't guarantee immediate effect).
More complex would be things like fetching a subscriber with a given IMSI first
and applying changes there. Arbitrary complexity of actions to update a value
are thinkable, so we probably won't get around writing it manually? Semi?
But hold on, there are different cases: we don't want to store individual IMSIs
in the config server (that's what the HLR is for), while we do want to store
individual BTS configurations centrally.
For GET/SET/NOTIFY of a BTS element, we would need the API to have an array
index argument, analogous to 'bts 0', 'bts 1'. For 'trx 0' etc we
even need
nested array indexes. With a path scheme? GET('bts[2].trx[0].arfcn').
But for operations on a subscriber, we would instead merely send instructions
to the HLR to do/store/say things -- if we want to allow managing subscribers
from the central "vty" application, and if we want to allow querying the
current state of entities (like 'show lchan summary'), we need another class of
items that don't GET/SET things from/to the central config server, but tell an
application to do or say something. Call them COMMAND, and they would feature
arbitrary arguments. In the VTY we just implemented whatever we want to
happen, in this scheme we should explicitly differentiate config items from
commands.
When we have separated BSC from NITB for good, centrally issued COMMANDs could
also make up for the removed information match-up: in the NITB we always knew
which BTS a subscriber was calling from with all of the detailed BTS state.
With a separate BSC we don't anymore -- but the central "vty" could run
queries
with MSC and BSC to match up that information conveniently again.
All of this is brainstorming and vaporware
It would be nice to dabble up some hacky code to discuss about, to sharpen our
teeth on; without spending too much time on it and having in mind to discard it
for something better anytime?
Sounds a bit like re-inventing dbus. (I don't know much about dbus though,
would it make sense to use dbus?? Is it a giant kraken of dependencies?)
Seems to me that with all the wishlist features we have collected now, the
implementation of such a system would actually be quite complex and not a small
effort.
I think the most important feature is that all our config items become a
machine and human API at the same time -- and that is a killer feature for all
those folks asking for web interfaces.
~N
4:12 p.m.
On 9 March 2017 14:41:37 CET, Neels Hofmeyr <nhofmeyr(a)sysmocom.de> wrote:
I once used a script that connects to nitb's vty, prints the lchan
summary and
exits. I ran that once a second. It often managed to crash the NITB!
I do that often, once even forgot and left it running in a screen a few days.
It didn't crash. I cat show lchan to nc and have to make it wait for 1 second
otherwise it sometimes misses to vty output, (minimum wait is 1 second with nc) Sometimes
i would like to poll lchan faster.
Anyway, not an ideal solution for monitoring the lchans, as you know.
Fairwaves have something using the control port, but maybe expanding the meas feed is a
better way to do this kind of thing?
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
4:14 p.m.
On 9 March 2017 14:41:37 CET, Neels Hofmeyr <nhofmeyr(a)sysmocom.de> wrote:
I once used a script that connects to nitb's vty, prints the lchan
summary and
exits. I ran that once a second. It often managed to crash the NITB!
I do that often, once even forgot and left it running in a screen a few days.
It didn't crash. I cat show lchan to nc and have to make it wait for 1 second
otherwise it sometimes misses to vty output, (minimum wait is 1 second with nc) Sometimes
i would like to poll lchan faster.
Anyway, not an ideal solution for monitoring the lchans, as you know.
Fairwaves have something using the control port, but maybe expanding the meas feed is a
better way to do this kind of thing?
10:14 p.m.
Hi Neels,
On Thu, Mar 09, 2017 at 02:41:37PM +0100, Neels Hofmeyr wrote:
We could, but I would rather use such a massive change to do it
properly, i.e. implement the 'immediate reaction' to value changes in
the respective programs.
Yes, but not now [TM]. I at least think I need to spend lots of more
time thinking about this (in the bathtub or elsewhere).
I once used a script that connects to nitb's vty,
prints the lchan summary and
exits. I ran that once a second. It often managed to crash the NITB! ... I
never actually investigated it, but would have been good to do so.
this should not happen, and I don't see a reason why it would. There's
no concurrency and no race conditions, everything runs synchronously in
one thread. I've not observed such behavior so far (luckily). but it
should be a relatively trivial scriptable test case: "vty load
generation" ;)
So would the central interface simply remember the
port numbers for us? No,
you're talking about a "config server" that the individual applications
connect
to. On the one hand nice, on the other yet another osmo-entity. Still, +1.
I don't really see where the problem lies with 'yet another process'.
People have been using zebra+quagga for some 20 years by now, and they
always came as a suite of processes. And that was before systemd took
care of re-spawning them ;)
I mean seriously, on my laptop I have 195 kernel threads cluttering my
'ps' output already, why would I care about a hand ful of osmocom
processes?
Would osmo-programs require the config server to run?
yes.
Would they be able to launch the config daemon
automatically -- portably?
I would probably not bother but simply print a meaningful error message
and assume that people get their systemd rules right in terms of the
dependencies?
GET SET
NOTIFY
ENUM, integers with ranges, etc.
machine-parseable description of a "MIB/config item", together with
syntax, ranges, enum values, help texts, etc (ideally in the C source
code, possibly in comments?)
Add to the wishlist: an indicator whether changing the value immediately
changes the application's behavior. A boolean unfortunately doesn't cut it,
e.g. some settings take effect when a specific BTS' OML link is restarted.
Yes, those are bugs / shortcomings of the code and should be
adressed by people caring sufficiently about it? We can re-generate the
BCCH/SACCH filling at any time and update them in the BTS. We can also
re-configure the OML Managed Objects at runtime. The protoocl permits
all of that.
With global structs like gsm_network, I could imagine
a completely automagic
code generation from API comments. The description could contain the global
gsm_network instance's symbol name, and the machine built code could provide
the NITB's side as well as the config server's side, in the NITB storing those
values directly in the global gsm_network instance. Or maybe let the automagic
code create a replacement for the config part of the gsm_network struct. That's
all easy for singleton items.
Yes, I would go for separate config structs, and hand a newly-allocated
one to the application, so the application can check for differences
between old and new config, if neded, before applying the new config.
More complex would be things like fetching a
subscriber with a given IMSI first
and applying changes there. Arbitrary complexity of actions to update a value
are thinkable, so we probably won't get around writing it manually? Semi?
That's not configuration data, is it? I would not try to do subscriber
management within the MIB.
But hold on, there are different cases: we don't
want to store individual IMSIs
in the config server (that's what the HLR is for), while we do want to store
individual BTS configurations centrally.
exactly.
For GET/SET/NOTIFY of a BTS element, we would need the
API to have an array
index argument, analogous to 'bts 0', 'bts 1'. For 'trx 0' etc we
even need
nested array indexes. With a path scheme? GET('bts[2].trx[0].arfcn').
yes. That's one part that OpenWRT's uci does ver nicely, but it has
weak types (or none at all?) and is unsuitable in many other ways.
Also, I think uci has limited path depth, which isn't nice either.
But for operations on a subscriber, we would instead
merely send instructions
to the HLR to do/store/say things -- if we want to allow managing subscribers
from the central "vty" application, and if we want to allow querying the
current state of entities (like 'show lchan summary'), we need another class of
items that don't GET/SET things from/to the central config server, but tell an
application to do or say something. Call them COMMAND, and they would feature
arbitrary arguments. In the VTY we just implemented whatever we want to
happen, in this scheme we should explicitly differentiate config items from
commands.
yes. There are many details to be resolved, though. We could simply
require each application to provide pretty-print functions for all of
its 'introspectable structs's and then build a library/plugin that the
"VTY application" can load to print even the "raw" binary structs.
We'd
just have to prefix them with some kind of type annotation (in the
protocol, not at runtime in all processes). This way even the printf()
effort is kept out of the respective applications, further reducing load
that adminstrative interfaces can put on them.
When we have separated BSC from NITB for good,
centrally issued COMMANDs could
also make up for the removed information match-up: in the NITB we always knew
which BTS a subscriber was calling from with all of the detailed BTS state.
With a separate BSC we don't anymore -- but the central "vty" could run
queries
with MSC and BSC to match up that information conveniently again.
possibly, but that sounds rather more comples.
All of this is
brainstorming and vaporware
It would be nice to dabble up some hacky code to discuss about, to sharpen our
teeth on; without spending too much time on it and having in mind to discard it
for something better anytime? Sounds a bit like re-inventing dbus. (I don't know
much about dbus though,
would it make sense to use dbus?? Is it a giant kraken of dependencies?)
I don't think dbus is a configuration management system, but an IPC
system. And yes, we'd need some kind of IPC, but I would ignore/defer
the exact IPC method/transport for now, as it is not the key "idea"
under discuission.
Seems to me that with all the wishlist features we
have collected now, the
implementation of such a system would actually be quite complex and not a small
effort.
yes. The question is if we can find something that is possible to do in
stages/milestones, rather than changing everything at once. Or some
elegant hack that provide at least some of the features we need at
limited effort.
I think the most important feature is that all our
config items become a
machine and human API at the same time -- and that is a killer feature for all
those folks asking for web interfaces.
exactly. Whether we do this still inside the processes or in an external
central process is a minor detail.
Also, what I'd really like is to see the generalized read/parse and save
code, avoiding all the possible manual mistakes we have at places.
Finally, being able to do away with the "vty-additions.xml" files but
having more documentation at the same place where the config items are
defined/declared is better and will make it feasible to have more
documentation in the vty reference manuals.
What I find very surprising is that nobody seems to have come up with
something like this already. But what can be found is miles far away
from what we're thinking of. There's libconfuse, uci, libcli, clish,
... - all of them seem to have some interesting ideas, but fall quite
short of a proper system. And then there's things like SNMP and
net-snmp which are ugly and complex.
The IETF has come up with OpenConfig (and NETCONF and RESTCONF
protocols) which goes very much into the direction I've been thinking
of. But it seems they've mostly been working on architecture as well as
OpenConfig models for regular IP-type network element configuration,
(writtein in YANG, a language they invented for that purpose) and
protocols, but I haven't seen any FOSS implemenataions yet?
http://jedelman.com/home/openconfig-data-models-and-apis/ is a good
intro into this part of the networking universe. Fully buzzword
compatible ;) And again very heavy and complex. If there were many
implementations out there, it might be worth investigating - but there
aren't.
Regards,
Harald
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
8:20 p.m.
Hi Harald,
A very interesting topic indeed.
On Mar 9, 2017 4:15 AM, "Harald Welte" <laforge(a)gnumonks.org> wrote:
So what I had in mind for quite some time (actually since netconf 1.2
about one year ago), is to have some kind of an external "VTY/MIB
daemon" (or even separate daemons for each) which maintains a
hierarchical database of configuration values. The MIB deamon simply
offers an API (via client library) to GET or SET the individual values,
or to NOTIFY an application about a changed value. This API is both
used by the actual Osmocom programs to obtain their configuration (and
obtain updates to it during runtime), as well as by the "VTY daemon"
providing interactive shell access to it. Finally, other external
applications could use the same interface/client library to do the same.
A proxy to SNMP or REST interfaces can be imagined, for even more
interfacing with the outside world.
Right now VTY and control interface serve three functions:
1. Configuration ("configure terminal")
2. Object manipulation (eg "subscriber IMSI .. active ...")
3. Information query with polling and traps (eg "show ..." and control
interface)
A centralized configuration DB massively works with use case (1), but
doesn't really fit use cases (2) and (3) IMHO. You can do it, but then
it'll serve just as a proxy to the actual application and may become a
quite bloated and complicated piece of code.
Running another instance of yet another daemon is also something I
personally would really like to avoid looking from a user/admin
perspective. Number of processes we're already running on a NITB BTS is
already getting out of hand :(
So stealing an idea from FreeSWITCH and building on top of it:
Why don't we keep the configuration DB inside each app, but expose a
machine readable API only? Then VTY becomes just one of many applications
talking to this API on par with web-ui, scripts and whatnot.
This API could be something like JSON or something more binary (Erlang
terms binary format, BERT, etc). We can even make it more REST-like with
only GET/SET/NOTIFY or more RPC like with a richer set of primitives. There
are pros and cons in all these approaches, but the key point is that:
1. This works with all 3 use cases above
2. It's less intrusive into the current code base (I think)
3. Allows to have a single CLI/VTY app talking to any Osmocom app. Btw, the
app can know default ports of standard apps and allow connection by neg let
name like:
$ osmo-cli -n osmo-bts
4. The app can still do TAB completion and help in a VTY way - FreeSWITCH
handles this perfectly with its fs_cli.
What do you think? It's a completely different approach than a config DB,
so I wanted to through this idea in without too much details to see if it
sticks.
Please excuse typos. Written with a touchscreen keyboard.
--
Regards,
Alexander Chemeris
CTO/Founder Fairwaves, Inc.
https://fairwaves.co
9:11 p.m.
On Thu, Mar 09, 2017 at 10:20:11PM +0300, Alexander Chemeris wrote:
Right now VTY and control interface serve three
functions:
1. Configuration ("configure terminal")
2. Object manipulation (eg "subscriber IMSI .. active ...")
3. Information query with polling and traps (eg "show ..." and control
interface)
A centralized configuration DB massively works with
use case (1), but
doesn't really fit use cases (2) and (3) IMHO. You can do it, but then
it'll serve just as a proxy to the actual application and may become a
quite bloated and complicated piece of code.
I have some vague ideas here but need to think further about it before I
think it's ready for an actual proposal.
Running another instance of yet another daemon is also
something I
personally would really like to avoid looking from a user/admin
perspective. Number of processes we're already running on a NITB BTS is
already getting out of hand :(
I don't see the problem. On modern systems it is simply a matter of a
systemd job configuring that daemon as a dependency for the other
daemons. And if you prefer some other system like upstart, it is
basically the same. I don't think any user cares about the number of
processes, at least not until you're talking about hundreds of them.
Why don't we keep the configuration DB inside each
app, but expose a
machine readable API only? Then VTY becomes just one of many applications
talking to this API on par with web-ui, scripts and whatnot.
It's another idea, indeed. I guess to a large extent the same work is
required for both my or your proposal: You need some data structures
that describe the syntax of the configuration values, their permitted
ranges, types, help texts, etc. The difference is then just where this
configuration is stored / "owned". I like the idea to have it external,
as some paramters can then easily be shared between processes. Let's
say the 'gsm_network' configuration, i.e. things like MCC, MNC. You
curently have to enter that on both the SGSN and the MSC, but this could
be stored just as one 'network' config object that multiple processes
refer to.
What do you think? It's a completely different
approach than a config DB,
so I wanted to through this idea in without too much details to see if it
sticks.
Well, see above, I think it's more or less the same code, but running
inside the same process vs. running it externaly?
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
10 Mar
10 Mar
10:11 a.m.
Hi.
I won't claim in-depth experience with MIB frameworks, just want to
contribute few buzzwords into brainstorming:
NETCONF - rfc 6241 https://tools.ietf.org/html/rfc6241
YANG - rfc rfc 6020 https://tools.ietf.org/html/rfc6020
The former is IETF protocol for configuring network devices [1]. The
latter is data modeling language describing device's configuration and
state, manipulated by netconf [2].
From brief overview those seems to match well with Harald's outline.
There might be few pieces still missing but it seems doable. I see
following advantages in adopting netconf & yang:
- it's ietf standard
- there are open source (including GPL) implementations we can play with
[3, 4]
- it's already widely supported by telecom vendors so chances are in
operators using Osmocom stack, will be some people familiar with it
- it's SDN/NFV compatible which makes it future-proof
- there's proposed extension RESTCONF which make integration with creepy
web things like SOAP much easier [5]
- and most important for me personally - there's Emacs mode for it [6] :-)
We're not the first to feel the need for proper MIB/configuration
subsystem, and the task indeed looks huge. So the more we can rely on
work already done in this field, the better. Even if we have to write
some things, by relying on open and widely adopted standards we increase
our chances of getting external contributions which is always nice.
[1] https://trac.ietf.org/trac/netconf
[2] http://www.yang-central.org/twiki/bin/view/Main/WebHome
[3] http://ensuite.sourceforge.net/
[4] https://github.com/CESNET/libnetconf
[5] https://tools.ietf.org/html/draft-bierman-netconf-restconf-04
[6] https://www.emacswiki.org/emacs/yang-mode.el
--
Max Suraev <msuraev(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
26 Apr
26 Apr
3:52 p.m.
Hi.
A little follow-up on this topic after OsmoDevCon. I've stumbled upon following
project http://www.clicon.org/
From a quick glance over their tutorial at
http://www.clicon.org/clicon_tutorial.pdf
it seems to fulfill almost all the
properties discussed at OsmoDevCon:
* autogeneration of cli and conf from single source
* transactions support
* both human-oriented cli and machine-oriented protocol from a single source
Added bonus is that it's available under GPL (dual-license actually):
https://github.com/clicon/clicon
The only downside I see so far (besides the sheer amount of work for migration) is
that this project is not widely used (yet?). Another thing I have not understood
without digging further is how to apply some configs/commands in runtime without
restarting the backend (OsmoBSC, OsmoMSC etc).
All-in-all it seems like smth definitely worth looking into.
--
Max Suraev <msuraev(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
2567
days inactive
2615
days old
8 comments
6 participants
participants (6)
-
Alexander Chemeris -
Harald Welte -
k＠rhizomatica.org -
Keith Whyte -
Max -
Neels Hofmeyr