So, we've been discussing IRL about the osmo-hnbgw implementation of
interacting with nftables from a separate thread.
The idea now is that a separate thread gets the results from nftables, caches
them and enqueues them, so that the main thread can update the hnbgw counters
in-sync. A second separate thread does nft add/del rule maintenance.
There is another complication, and I want to make sure we agree on the solution:
Problem: to delete a rule, I need to retrieve its 'handle' first.
With the amount of threading we want, this becomes a bit complex.
So. We 'add rule' and 'delete rule', as hNodeB attach and detach.
But, to be able to 'delete rule', we need to retrieve an id that nftables
assigned to this rule when it was added, the 'handle':
add rule inet osmo-hnbgw gtpu-ul ip saddr 1.2.3.4 counter comment "ul:1-2-3-4";
...
list table inet osmo-hnbgw
-->
table inet osmo-hnbgw {
chain gtpu-ul {
ip saddr 1.2.3.4 counter packets 5 bytes 100 comment "ul:1-2-3-4" # handle 23;
} ^^^^^^^^^
}
...
delete rule inet osmo-hnbgw gtpu-ul handle 23;
^^^^^^^^^
AFAICT I cannot set a handle right from the start. I can also not delete a rule
by just stating it again as it is, nor by using the 'comment' as a handle. nft
wants a 'handle', period. Does anyone know better?
Every time we read counters, osmo-hnbgw also retrieves the handles for each
rule, and correlates via the 'comment'. Similarly to counters, we can only get
all handles at once, via a complete table dump (possibly inefficient).
It works out fine in most practical cases. But the complex corner case is when
an hNodeB detaches quickly, before the counters had a chance to run and update
the hNodeB's state, including the handle.
In the current agreement on our implementation with two threads for nft, with
queues back to the main thread, it would go something like this, to avoid
races:
main() nft_maintenance_thread() nft_read_counters_thread()
| | |
|---add hNodeB-------->| |
|<--ok-----------------| |
X | |
|---del hNodeB-------->| |
| [rule handle unknown!] |
..|<--EAGAIN-------------| |
. | | |
. |---do counters--------------------------------->|
. | | [work]
. | | [work]
. | | [work]
. |<--update-counters-and-handles------------------|
. | | |
..|---del hNodeB-------->| |
| [delete rule] |
|<--done---------------| |
Simplifying a bit...
main() nft_maintenance_thread() nft_read_counters_thread()
| | |
|---add hNodeB-------->| |
|<--ok-----------------| |
X | |
[want: del hNodeB] | |
[rule handle unknown!] | |
[set some "del" flag] | |
. | | |
. |---do counters--------------------------------->|
. | | [work]
. | | [work]
. | | [work]
. |<--update-counters-and-handles------------------|
. | | |
[del flag!] | |
|---del hNodeB-------->| |
| [delete rule] |
|<--done---------------| |
I'm not liking very much the amount of complexity spawning here, but
this last approach is ok I guess, do you agree?
i.e. when the handle is not set yet, set a flag on the hNodeB,
and trigger the 'delete' the next time the counter thread reports back a handle
for the hNodeB.
~N
--
- Neels Hofmeyr <nhofmeyr(a)sysmocom.de> https://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Siemensstr. 26a
* 10551 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
Dear mailing list,
as announced in https://osmocom.org/news/246, the rpm packages for
CentOS 8 / AlmaLinux 8 are unmaintained.
sysmocom is currently still ensuring that the rpm binary packages build
from the rpm spec files we have in our git repositories. But this is
also significant effort that we would rather spend elsewhere, and so we
are considering to stop providing rpm builds altogether and possibly
removing these spec files from our git repositories.
This affects the official Osmocom binary package repositories (nightly
and latest) for CentOS 8/AlmaLinux 8 and openSUSE Tumbleweed.
Stable versions of Osmocom programs are also in openSUSE's official
repositories and can be installed from there.
So, is anyone using the official Osmocom rpm repositories?
As OsmoDevCon is coming up this weekend, we can also discuss it there,
feel free to talk to me about it.
Best regards,
Oliver
--
- Oliver Smith <osmith(a)sysmocom.de> https://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Siemensstr. 26a
* 10551 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
Dear Osmocom developers,
In our project we have been using the gsmtap pseudo-header to store UMTS and LTE signalling traces in pcapng format. This is very efficient compared to our previous implementation based on DLT_USER, and thanks to the gsmtap dissector, it allows any *shark app to read our output file.
When looking for the proper type and sub_type for NR signalling traces, I discovered they are not yet part of the standard. I found several threads of discussion about this, from about 2017 onwards, and I understand it is a matter of time and resources if the version 3 of GSMTAP has not been released.
The most interesting feature I have read about is the ability to extend the header with a TLV approach. EARFCN, eNodeB-ID, PCI, are useful metadata we would like to add to our traces.
In an ideal world, I would like things to be put in motion so that a new version of GSMTAP could see the light. More humbly, I ask you what kind of help is needed. Are you waiting for a draft specification? Are all requirements well known?
I suppose that LTE and NR are not the main focus today for liboscmocore development, but there are many projects out there waiting for the GSMTAP format to be updated.
Thanks!
Mauro
Hi all,
we are currently having lots of discussions on (non-)blocking I/O. I'd like to
put these thoughts out there for that discussion, because there seems to be a
misunderstanding in terms.
Blocking never goes away, it is just reduced to other orders of magnitude.
Types of "blocking":
- blocking or non-blocking pipes: will writing to a file or socket stop the
program until the pipe is ready for writing? (basic "OS level" I/O)
- synchronous or asynchronous event handling: will the program stop until a
remote side has responded, or can the program handle other events in the
meantime? (one job queue, one worker == osmo_select_main())
- sequential or parallelized event handling: can events be handled
concurrently, or just one after the other?
(one or more job queues, more than one worker)
- concurrent access of resources: a given resource is not thread-safe, hence
one thread needs to wait for the other to release a resource lock.
(This is always present, the aim is to hit a sweet spot of least locking.)
In osmo programs I have worked on, we do the first two, but not the other two.
Asynchronous event handling is the bare minimum for a server program to be
functional. Non-blocking pipes is a common addition that is easy to do.
From then on we enter the world of parallelization, and things get very complex
very quickly. It is possible to cause more blocking than before. It is possible
to significantly increase the load, instead of improving.
I am familiar with parallelized non-blocking event handling and I/O, from
realtime audio+video+control hacking. We do not use any of these techniques in
osmo programs I have worked on -- for good reasons, I thought.
The spectrum, from most blocking to most non-blocking:
- single-threaded, single queue with async defer;
- task queue with multiple worker threads;
- scheduling based on fairness or urgency;
- map/reduce across a cloud, in a functional language.
We're almost all the way to the blocking end of the parallelization spectrum.
So far I thought that this was a conscious choice. Async-but-blocking is low
complexity, with large benefits in maintainability and stability.
Example:
If we have pending, say, 10 incoming packets on three different links, we
handle each packet one by one when it is its turn. If one subscriber's incoming
measurement report triggers longish handover calculations, any events like an
MGCP ACK or SCCP CC for some other subscribers will have to wait in line, even
though they might take a thousand fold less time to complete.
OsmoBSC works well in that fashion, even for hundreds of cells and multiple
MSC: compared to audio+video+control, CNI signalling has huge tolerances on
timing. This is why 3GPP separates control-plane from user-plane.
It is important to balance all of these aspects!
---
It was mentioned somewhere that our VTY is both asynchronous and non-blocking.
I do not agree at all and would like to explain, as an example of the above.
Our vty server is NOT asynchronous. When a VTY request comes in, the vty
function must directly vty_out() the response. We cannot defer the VTY response
asynchronously like any other protocol can (see example below).
Our VTY structures, and the program-specific internal state that VTY
manipulates and queries, are not thread-safe. The VTY server cannot be
parallelized as it is now.
A contrived example:
Let's say we wanted to query nft counters from a VTY command:
* read VTY command from user,
* do some nft command asynchronously,
* and print back the result when nft is done.
Naively, we could store the struct vty * somewhere, and exit the vty handling
function. When nft is done some time later, just vty_out() the result to that
struct vty * that we still have from earlier. But there are problems:
If the user closed the telnet session in the meantime, this struct vty * is
stale and the program will crash. We need a cancel mechanism to avoid that.
Also when a VTY command function is done, we directly transmit the next VTY
prompt. vty-test scripts (`expect`) won't function properly when more response
data arrives after the prompt is received; human users may be confused.
So our VTY server is *both* Synchronous and Blocking. It is not trivial to make
it async (like all of our other protocols are) and non-blocking (which we have
nowhere in osmo-cni yet).
---
These are the kinds of mechanisms I care about in our discussions:
- "blocking" on what time scale?
- tradeoff with code complexity and maintainability.
- tradeoff with code stability and determinism.
- tradeoff with system performance load due to additional management and caching.
One does not simply put things in threads.
There are very non-trivial aspects that *always* come with it,
one of them is super good, most of them are pretty bad.
~N
--
- Neels Hofmeyr <nhofmeyr(a)sysmocom.de> https://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Siemensstr. 26a
* 10551 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
