From laforge at osmocom.org  Fri Sep  4 19:08:20 2020
From: laforge at osmocom.org (Harald Welte)
Date: Fri, 4 Sep 2020 21:08:20 +0200
Subject: OsmoDevCon  / COVID-19
Message-ID: <20200904190820.GA609235@nataraja>

Dear fellow Osmocom developers,

as you all know, we've sadly had to postpone OsmoDevCon 2020 back in
April this year.  At the time, we discussed to re-visit the situation
in October 2020.

While legally it is no problem at all to host an event with ~ 20
participants in Berlin/Germany (specific regulations really only start
from 50+ participants) - I'm not entirely convinced it would be the
smartest move.

Legality and public health regulations are only one part of the equation
- common sense and profound care for the key members of our community
for sure are more relevant considerations to me.

I'm not 100% in favour and not 100% against.  Hence, I would like to get
your input.  Should we

a) try to get an event organized on-site in Berlin?  We'd have to move
   to a larger venue than IN-Berlin with proper ventilation and sufficient
   space so we can keep physical distance, but I think that's
   manageable for sysmocom as organizer.

b) simply postpone to 2021?  I'm convinced the situation will not change
   significantly (in a positive way) until late April 2021, so it's not
   really a "solution" as it will likely mean we have to think of late
   2021 or 2022.

c) plan some kind of online conference?  To be honest, I think this
   model works fine for events where a single speaker wants to give
   lectures to hundreds or thousands of participants.  But OsmoDevCon
   is much more interactive.  We could record or live-stream some talks
   or screencasts from home, sure.  But that only captures one part of
   the event.  We could also try to set a date for a collaborative
   mumble, or the like - for the "hallway track".

What are your thoughts?  Let's avoid cross-posting the discussion to all
of the mailing lists and simply have it on openbsc at lists.osmocom.org.

Regards,
	Harald
-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From 246tnt at gmail.com  Fri Sep  4 19:50:51 2020
From: 246tnt at gmail.com (Sylvain Munaut)
Date: Fri, 4 Sep 2020 21:50:51 +0200
Subject: OsmoDevCon / COVID-19
In-Reply-To: <20200904190820.GA609235@nataraja>
References: <20200904190820.GA609235@nataraja>
Message-ID: <CAHL+j0_TkbDFO-kx5TBb==F0xfw7ypD4cuoAVVhR3FXHAaxmRg@mail.gmail.com>

Hi,

I think a 2020 event doesn't make all that much sense.

I'm also not a big fan of online events, at this point well we have IRC.
At least my experience with video / audio conferencing has never been
good and it gets tiring very quickly.

So I would target a 2021 version.

I'm not sure how things are in Germany but at least here, with
reasonable precautions, the spread seems to be in regression, so
hopefully by April 2021 we'd have a fairly good view of what an
osmodevcon 2021 would look like.
As for the venue, I'm not convinced this would make that much of a
difference. I mean 4 days, with about 14h per day with the same
people, not sure how 1.5m distance effectiveness would be with
everyone moving around anyway.


Cheers,

     Sylvain


From nhofmeyr at sysmocom.de  Mon Sep  7 10:58:16 2020
From: nhofmeyr at sysmocom.de (Neels Hofmeyr)
Date: Mon, 7 Sep 2020 12:58:16 +0200
Subject: OsmoDevCon / COVID-19
In-Reply-To: <CAHL+j0_TkbDFO-kx5TBb==F0xfw7ypD4cuoAVVhR3FXHAaxmRg@mail.gmail.com>
References: <20200904190820.GA609235@nataraja>
 <CAHL+j0_TkbDFO-kx5TBb==F0xfw7ypD4cuoAVVhR3FXHAaxmRg@mail.gmail.com>
Message-ID: <20200907105816.GA28150@my.box>

what he said.
~N

On Fri, Sep 04, 2020 at 09:50:51PM +0200, Sylvain Munaut wrote:
> Hi,
> 
> I think a 2020 event doesn't make all that much sense.
> 
> I'm also not a big fan of online events, at this point well we have IRC.
> At least my experience with video / audio conferencing has never been
> good and it gets tiring very quickly.
> 
> So I would target a 2021 version.
> 
> I'm not sure how things are in Germany but at least here, with
> reasonable precautions, the spread seems to be in regression, so
> hopefully by April 2021 we'd have a fairly good view of what an
> osmodevcon 2021 would look like.
> As for the venue, I'm not convinced this would make that much of a
> difference. I mean 4 days, with about 14h per day with the same
> people, not sure how 1.5m distance effectiveness would be with
> everyone moving around anyway.
> 
> 
> Cheers,
> 
>      Sylvain


From mychaela.falconia at gmail.com  Thu Sep 10 22:21:14 2020
From: mychaela.falconia at gmail.com (Mychaela Falconia)
Date: Thu, 10 Sep 2020 14:21:14 -0800
Subject: Alternatives to CalypsoBTS?
Message-ID: <CA+uuBqavNt2r-oE=ZcmvXB5HOe2K1uPRrW1Qqjimjc_=X8xviw@mail.gmail.com>

Hello GSM network side folks,

I assume that most of you are probably aware of the existence of a
certain hack called CalypsoBTS: it's a rather unbelievable hack that
takes a piece of hw meant to be a GSM MS and turns it into a poor
man's GSM BTS, all inherent asymmetry in the GSM air interface be
damned.  As a manufacturer of Calypso-based GSM MS devices I
occasionally get approached by people who seek to acquire a Calypso
device seemingly for the sole purpose of running that CalypsoBTS hack,
and every time I get approached by someone of that sort, I always
shake my head in bewilderment - isn't there a better way to get your
own little toy BTS running than to misuse GSM MS hardware?

The purpose of the present post is to solicit advice from the community
as to what I should tell those poor souls who are seeking to set up
their own toy BTS and are looking to do it via the CalypsoBTS route -
is there some better way that we as a community can steer them toward
instead?

It is my understanding - please correct me if I'm wrong - that the
least expensive way to set up your own GSM BTS for toy purposes (as
opposed to running a real operational GSM network which people will
depend on for real communication) is to use a generic SDR device of
one of several types that are supported by osmo-bts-trx/osmo-trx - but
this is where my knowledge in this area ends, as this particular mode
of GSM toying has never been an area of interest for me.  But for
those people who (unlike me) *are* interested in setting up their own
toy BTS in the least expensive way, what SDR hardware should we as the
community recommend for them?  What is the cheapest option that will
be good enough - especially if the criteria for "good enough" are
compared to CalypsoBTS?  What would be the least expensive option that
is just good enough to be advocated as a better alternative to the
CalypsoBTS hack?

In the case of my current FCDEV3B hardware the price tag seems to be
an effective deterrent against people misusing it as CalypsoBTS:
FCDEV3B is expensive, and someone whose actual need is to run their
own BTS rather than MS can easily buy a suitable SDR device for the
same price or less, it seems.  But the issue is beginning to rear its
ugly head again because I have another development board in the works
(not here yet, probably won't have the hw until December), and there
is a possibility (nothing is certain yet) that it might be a bit less
expensive than FCDEV3B.

Is there an SDR-based option (or any other non-CalypsoBTS option) for
running a toy BTS with osmo-bts-trx and the Osmocom CNI stack behind
it that would cost $250 or less in hardware?  The $250 number comes
from my anticipated-around-December new FreeCalypso development board
- there is a chance that it might be that cheap (but again, absolutely
nothing is certain at the present moment) - and if there is no SDR or
other option for running your own toy BTS for the same or lower hw
price, then I fear that I am going to be flooded with support requests
from people asking for help with using my new board as CalypsoBTS,
which I absolutely dread.  Hence my present attempt to pre-emptively
seek some better solution for those people.

(It would be nice if that stream of support requests from people
seeking to run the CalypsoBTS hack could be redirected to Sysmocom or
some other commercial entity who could make some money helping those
people, but my experience is that these people are not the kind who
would ever pay for commercial support, so no hope there...)

It may also be worth mentioning that the filter replacement hack
(removing or replacing Rx SAW filters that are meant to limit the GSM
MS device's Rx capability to only specific GSM downlink bands) will
not be possible on the new FreeCalypso GSM MS board that will be
coming around December - that design uses an integrated RF FEM (front
end module) instead of discrete antenna switch and SAW filter
components.  But I've also heard that plenty of people run the
CalypsoBTS hack without doing any filter rework, just letting the
strong signal from a nearby GSM MS force its way through wrong SAW
filters and not caring about the 40 dB or so of attenuation being
incurred - I cringe at the thought, but that's what people do...

M~


From domi at tomcsanyi.net  Fri Sep 11 06:07:36 2020
From: domi at tomcsanyi.net (Tomcsanyi, Domonkos)
Date: Fri, 11 Sep 2020 08:07:36 +0200
Subject: Alternatives to CalypsoBTS?
In-Reply-To: <CA+uuBqavNt2r-oE=ZcmvXB5HOe2K1uPRrW1Qqjimjc_=X8xviw@mail.gmail.com>
References: <CA+uuBqavNt2r-oE=ZcmvXB5HOe2K1uPRrW1Qqjimjc_=X8xviw@mail.gmail.com>
Message-ID: <F7DB1848-C3E7-4751-9E9A-584433EBE281@tomcsanyi.net>

Hi Mychaela,

I have long been running my tests using a USRP, because on the long run it is always worth to buy something that is better in quality. However indeed people interested in cheap hacks would not like the price tag of the device.
Throwing a quick glimps at the OsmoTRX wiki page I think the cheapest hardware that should work well is the LimeSDR - coming in at around 300 USD. I think most of the bugs and issues with it are more or less sorted - however this is only based on what I see on the mailing list, I am not actively following the code developments.
BTW I have run without any hardware modification a calypsoBTS with 2 phones. It worked suprisingly well, even a single voice call was possible reliably. I know it is complete abuse of the hardware, but otoh it was a unique and awesome experience :).

Cheers,
Domi

> 2020. szept. 11. d?tummal, 0:21 id?pontban Mychaela Falconia <mychaela.falconia at gmail.com> ?rta:
> 
> ?Hello GSM network side folks,
> 
> I assume that most of you are probably aware of the existence of a
> certain hack called CalypsoBTS: it's a rather unbelievable hack that
> takes a piece of hw meant to be a GSM MS and turns it into a poor
> man's GSM BTS, all inherent asymmetry in the GSM air interface be
> damned.  As a manufacturer of Calypso-based GSM MS devices I
> occasionally get approached by people who seek to acquire a Calypso
> device seemingly for the sole purpose of running that CalypsoBTS hack,
> and every time I get approached by someone of that sort, I always
> shake my head in bewilderment - isn't there a better way to get your
> own little toy BTS running than to misuse GSM MS hardware?
> 
> The purpose of the present post is to solicit advice from the community
> as to what I should tell those poor souls who are seeking to set up
> their own toy BTS and are looking to do it via the CalypsoBTS route -
> is there some better way that we as a community can steer them toward
> instead?
> 
> It is my understanding - please correct me if I'm wrong - that the
> least expensive way to set up your own GSM BTS for toy purposes (as
> opposed to running a real operational GSM network which people will
> depend on for real communication) is to use a generic SDR device of
> one of several types that are supported by osmo-bts-trx/osmo-trx - but
> this is where my knowledge in this area ends, as this particular mode
> of GSM toying has never been an area of interest for me.  But for
> those people who (unlike me) *are* interested in setting up their own
> toy BTS in the least expensive way, what SDR hardware should we as the
> community recommend for them?  What is the cheapest option that will
> be good enough - especially if the criteria for "good enough" are
> compared to CalypsoBTS?  What would be the least expensive option that
> is just good enough to be advocated as a better alternative to the
> CalypsoBTS hack?
> 
> In the case of my current FCDEV3B hardware the price tag seems to be
> an effective deterrent against people misusing it as CalypsoBTS:
> FCDEV3B is expensive, and someone whose actual need is to run their
> own BTS rather than MS can easily buy a suitable SDR device for the
> same price or less, it seems.  But the issue is beginning to rear its
> ugly head again because I have another development board in the works
> (not here yet, probably won't have the hw until December), and there
> is a possibility (nothing is certain yet) that it might be a bit less
> expensive than FCDEV3B.
> 
> Is there an SDR-based option (or any other non-CalypsoBTS option) for
> running a toy BTS with osmo-bts-trx and the Osmocom CNI stack behind
> it that would cost $250 or less in hardware?  The $250 number comes
> from my anticipated-around-December new FreeCalypso development board
> - there is a chance that it might be that cheap (but again, absolutely
> nothing is certain at the present moment) - and if there is no SDR or
> other option for running your own toy BTS for the same or lower hw
> price, then I fear that I am going to be flooded with support requests
> from people asking for help with using my new board as CalypsoBTS,
> which I absolutely dread.  Hence my present attempt to pre-emptively
> seek some better solution for those people.
> 
> (It would be nice if that stream of support requests from people
> seeking to run the CalypsoBTS hack could be redirected to Sysmocom or
> some other commercial entity who could make some money helping those
> people, but my experience is that these people are not the kind who
> would ever pay for commercial support, so no hope there...)
> 
> It may also be worth mentioning that the filter replacement hack
> (removing or replacing Rx SAW filters that are meant to limit the GSM
> MS device's Rx capability to only specific GSM downlink bands) will
> not be possible on the new FreeCalypso GSM MS board that will be
> coming around December - that design uses an integrated RF FEM (front
> end module) instead of discrete antenna switch and SAW filter
> components.  But I've also heard that plenty of people run the
> CalypsoBTS hack without doing any filter rework, just letting the
> strong signal from a nearby GSM MS force its way through wrong SAW
> filters and not caring about the 40 dB or so of attenuation being
> incurred - I cringe at the thought, but that's what people do...
> 
> M~


From laforge at gnumonks.org  Fri Sep 11 09:29:52 2020
From: laforge at gnumonks.org (Harald Welte)
Date: Fri, 11 Sep 2020 11:29:52 +0200
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
References: <CAAtamJbY4mvdVQLPLygpToA=WzXJ7du4nYxC2jqpAe3dcGL9TQ@mail.gmail.com>
 <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
Message-ID: <20200911092952.GQ1165387@nataraja>

Hi Xaver,

On Fri, Sep 11, 2020 at 07:51:50AM +0200, Xaver Zu wrote:

> I created a short Wiki page. Please see if it's enough as a link to a
> commit message.
> https://osmocom.org/projects/sdr/wiki/PCIeSDR

Thanks.  For sure it is sufficient in terms of content.  I did some minor reformatting
while reading.

However, I found a major problem. You state:

> The higher-level driver (in userspace) includes a DSP, a calibration stage, and the gateware update. It is in the form of a closed source dynamic library named libsdr.so. The C API for Linux / x86 is in the form of a header file libsdr.h which also serves as brief documentation.

This is incompatible with the strong copyleft licensing (AGPLv3) of osmo-trx!
You cannot directly link with a proprietary library.

The only way I can see a PCIeSDR backend for osmo-trx would be via osmo-trx-ipc,
which provides a general-purpose sharde memory interface to another process.  The
other process can then use the non-free library, if you want that.

-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at osmocom.org  Fri Sep 11 10:50:02 2020
From: laforge at osmocom.org (Harald Welte)
Date: Fri, 11 Sep 2020 12:50:02 +0200
Subject: Question on osmo-trx mcbts
Message-ID: <20200911105002.GU1165387@nataraja>

Hi Thomas,

it's been some time, hope you're doing fine!

Today on IRC we were collectively wondering about one detail in the osmo-trx
mcbts implementation that has been there from day one:

Your're creating the Channelizer and Synthesis class for four channels of 800kHz
within the 3.2Mbps wideband-channel.  So far so good.

But why does the code ever only use up to three of them?

And why is there a specific re-ordering, see radioInterfaceMulti.cpp in
getLogicalChan() ?

It would be great if you could share your thoughts on that.

To me, it looks like the constraint to 3 channels is arbitrary and we should just
as well be able to use all four?

Thanks,
	Harald

-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at gnumonks.org  Fri Sep 11 15:13:32 2020
From: laforge at gnumonks.org (Harald Welte)
Date: Fri, 11 Sep 2020 17:13:32 +0200
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
References: <CAAtamJbY4mvdVQLPLygpToA=WzXJ7du4nYxC2jqpAe3dcGL9TQ@mail.gmail.com>
 <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
Message-ID: <20200911151332.GD1165387@nataraja>

Hi Xaver,

[posting this on the mailing list, as it may be of interest to others]

On Fri, Sep 11, 2020 at 04:55:35PM +0200, Xaver Zu wrote:
> This is a dynamic linking.

this is true, but the nature of the linking does not matter in terms of
what creates the derivative work.

> Is this a problem even if I do not distribute the resulting binary? 

The AGPL (contrary to the GPL) conditions start when users remotely interact
with your software (See Section 13 of AGPLv3).  This means even operating
a network with remote users already means you need to provide the complete
and corresponding source code to the entire work under AGPLv3, which is
impossible if you have a proprietary dependency.

So if you build the software, and you make sure that nobody but yourself
every uses the resulting GSM network, you are fine. But as soon as any
third party uses it, you would be putting yourself in the impossible position
to providing the source code to a proprietary library.

> I can simply add a warning to the file that the compiled program cannot be distributed.

One might consider this a "further restriction" which by itself is a
violation of AGPLv3:

	All other non-permissive additional terms are considered "further
	restrictions" within the meaning of section 10. [...] You may not impose
	any further restrictions on the exercise of the rights granted or
	affirmed under this License.

But even beyond that: How many users do you think willread that warning
in the source code vs. how many will simply run it (and possibly allow
others to register).  Even if you added a message to start-up, and to
the user manual, I still would think many people don't look at that.

I don't think we should distribute something that "tricks" people into
committing license violations or putting them in an impossible position,
legally speaking.

> Is it a violation that someone assembles and uses it on their own machine?

It is not a violation as long as they are they only person using the
resulting GSM network (or using its VTY or its TRXC/TRXD interface).

I would compare distributing source code like that to shipping a product
that is almost impossible to use safely, but very easy to use in a
[legally] unsafe way.

> Is it a violation to even have sources and build instructions for it?

Maybe not, but see above, I don't want to make it too easy for people to
shoot themselves into their foot.

> Can we agree to leave it as a starting point until someone will have the
> time to do it using IPC?

I'm very sorry (really), but we will not merge this code to the
mainline osmo-trx code base on osmocom.org.

Kind regards,
	Harald

-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From mychaela.falconia at gmail.com  Fri Sep 11 17:07:53 2020
From: mychaela.falconia at gmail.com (Mychaela Falconia)
Date: Fri, 11 Sep 2020 09:07:53 -0800
Subject: Support for new SDR devices
In-Reply-To: <20200911151332.GD1165387@nataraja>
References: <CAAtamJbY4mvdVQLPLygpToA=WzXJ7du4nYxC2jqpAe3dcGL9TQ@mail.gmail.com>
 <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
Message-ID: <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>

Let me see if I understand the PCIeSDR proprietary libsdr.so situation
correctly.  There is a hardware vendor (Amarisoft) who produces a
piece of hardware, but refuses to provide FLOSS support for it, and
instead provides an essential layer (required for the device to work)
only in the form of a proprietary binary library with no source - is
this understanding correct?

Does this vendor have competitors?  Are there other companies who make
SDR devices in PCIe form factor, but who provide all necessary support
pieces in full source form under an Osmocom-compatible license?  Maybe
LimeSDR PCIe?  If such alternative non-proprietary vendors exist, why
not vote with your dollars against proprietary blobs by giving your
business to better vendors rather than Amarisoft?

OTOH if you personally (Xaver) already invested into Amarisoft
hardware and don't want your work to go to waste because of license
worship, I encourage you to maintain your own fork of OsmoTRX outside
of osmocom.org and thus out of reach of their license police.  I will
never use it myself or even look at it because this kind of SDR is not
my area of interest, but I am encouraging you as a matter of general
principle, just like my predecessor Che Guevara would have.

M~


From mychaela.falconia at gmail.com  Fri Sep 11 17:54:37 2020
From: mychaela.falconia at gmail.com (Mychaela Falconia)
Date: Fri, 11 Sep 2020 09:54:37 -0800
Subject: Alternatives to CalypsoBTS?
In-Reply-To: <F7DB1848-C3E7-4751-9E9A-584433EBE281@tomcsanyi.net>
References: <CA+uuBqavNt2r-oE=ZcmvXB5HOe2K1uPRrW1Qqjimjc_=X8xviw@mail.gmail.com>
 <F7DB1848-C3E7-4751-9E9A-584433EBE281@tomcsanyi.net>
Message-ID: <CA+uuBqaW=j7g+vz95e9f1y3UE=vXwCb6bxN+xCJ0QmdFvMu0JQ@mail.gmail.com>

Hi Domi,

> Throwing a quick glimps at the OsmoTRX wiki page I think the cheapest
> hardware that should work well is the LimeSDR - coming in at around 300 USD.

Looking around myself, I came to the same general conclusion - but
there are several different members of the Lime family.  It looks like
LimeSDR-Mini is the cheapest of all of them, but does it have some
problem compared to the original LimeSDR-USB?  Does it have a worse
clock that isn't good enough to run a GSM BTS?  And what about
LimeNET-Micro?  It appears to be priced the same as the original
LimeSDR-USB, but has a built-in RPi microcomputer that the software
would run on, and its GPSDO seems like a better clock option than all
of the predecessors - but I could be missing something...

M~


From xaver1zu at gmail.com  Fri Sep 11 18:09:16 2020
From: xaver1zu at gmail.com (Xaver Zu)
Date: Fri, 11 Sep 2020 20:09:16 +0200
Subject: Support for new SDR devices
In-Reply-To: <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
References: <CAAtamJbY4mvdVQLPLygpToA=WzXJ7du4nYxC2jqpAe3dcGL9TQ@mail.gmail.com>
 <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
Message-ID: <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>

I use it in a laboratory with 1-2 clients. It's about backing up that job
when I've already invested time. It seems that the master branch is
impossible for this. I have it in a private fork.

Is it possible in its current form to have it in any branch?
Can it at least stay in the gerrit?

Basically the main problem is that I link with the libsdr.so library but I
can't publish the source codes. I don't have them.
Could I solve this with an open source (GPL) reimplementation of that
library?



p? 11. 9. 2020 v 19:07 odes?latel Mychaela Falconia <
mychaela.falconia at gmail.com> napsal:

> Let me see if I understand the PCIeSDR proprietary libsdr.so situation
> correctly.  There is a hardware vendor (Amarisoft) who produces a
> piece of hardware, but refuses to provide FLOSS support for it, and
> instead provides an essential layer (required for the device to work)
> only in the form of a proprietary binary library with no source - is
> this understanding correct?
>
> Does this vendor have competitors?  Are there other companies who make
> SDR devices in PCIe form factor, but who provide all necessary support
> pieces in full source form under an Osmocom-compatible license?  Maybe
> LimeSDR PCIe?  If such alternative non-proprietary vendors exist, why
> not vote with your dollars against proprietary blobs by giving your
> business to better vendors rather than Amarisoft?
>
> OTOH if you personally (Xaver) already invested into Amarisoft
> hardware and don't want your work to go to waste because of license
> worship, I encourage you to maintain your own fork of OsmoTRX outside
> of osmocom.org and thus out of reach of their license police.  I will
> never use it myself or even look at it because this kind of SDR is not
> my area of interest, but I am encouraging you as a matter of general
> principle, just like my predecessor Che Guevara would have.
>
> M~
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200911/3f59a1ea/attachment.htm>

From denver at ossguy.com  Fri Sep 11 18:23:17 2020
From: denver at ossguy.com (Denver Gingerich)
Date: Fri, 11 Sep 2020 18:23:17 +0000
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
References: <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
Message-ID: <20200911182316.GY25289@ossguy.com>

On Fri, Sep 11, 2020 at 08:09:16PM +0200, Xaver Zu wrote:
> Basically the main problem is that I link with the libsdr.so library but I
> can't publish the source codes. I don't have them.
> Could I solve this with an open source (GPL) reimplementation of that
> library?

Yes!  Please do let us know where you publish this if you do write such a reimplementation.

Denver
https://jmp.chat/


From mychaela.falconia at gmail.com  Fri Sep 11 18:36:07 2020
From: mychaela.falconia at gmail.com (Mychaela Falconia)
Date: Fri, 11 Sep 2020 10:36:07 -0800
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
References: <CAAtamJbY4mvdVQLPLygpToA=WzXJ7du4nYxC2jqpAe3dcGL9TQ@mail.gmail.com>
 <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
Message-ID: <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>

Hi Xaver,

> It's about backing up that job when I've already invested time.

Yes, I totally empathize with you on this one!

> Is it possible in its current form to have it in any branch?
> Can it at least stay in the gerrit?

Given their stance on license worship, I highly doubt that the owners
of osmocom.org domain and infrastructure would allow license-violating
stuff to be hosted anywhere under their domain.  But I can give you
hosting on freecalypso.org if you like - I am a proud pirate. :-)

M~


From xaver1zu at gmail.com  Fri Sep 11 18:49:29 2020
From: xaver1zu at gmail.com (Xaver Zu)
Date: Fri, 11 Sep 2020 20:49:29 +0200
Subject: Support for new SDR devices
In-Reply-To: <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>
References: <CAAtamJbY4mvdVQLPLygpToA=WzXJ7du4nYxC2jqpAe3dcGL9TQ@mail.gmail.com>
 <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>
Message-ID: <CAAtamJYRRkpBvQOuPzOkVvudApKhgtSy_LHdViroqPWCn5RGmw@mail.gmail.com>

p? 11. 9. 2020 v 20:36 odes?latel Mychaela Falconia <
mychaela.falconia at gmail.com> napsal:

> Hi Xaver,
>
> > It's about backing up that job when I've already invested time.
>
> Yes, I totally empathize with you on this one!
>
> > Is it possible in its current form to have it in any branch?
> > Can it at least stay in the gerrit?
>
> Given their stance on license worship, I highly doubt that the owners
> of osmocom.org domain and infrastructure would allow license-violating
> stuff to be hosted anywhere under their domain.  But I can give you
> hosting on freecalypso.org if you like - I am a proud pirate. :-)
>
> M~
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200911/b1fb9aee/attachment.htm>

From xaver1zu at gmail.com  Fri Sep 11 18:55:53 2020
From: xaver1zu at gmail.com (Xaver Zu)
Date: Fri, 11 Sep 2020 20:55:53 +0200
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJYRRkpBvQOuPzOkVvudApKhgtSy_LHdViroqPWCn5RGmw@mail.gmail.com>
References: <CAAtamJbY4mvdVQLPLygpToA=WzXJ7du4nYxC2jqpAe3dcGL9TQ@mail.gmail.com>
 <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>
 <CAAtamJYRRkpBvQOuPzOkVvudApKhgtSy_LHdViroqPWCn5RGmw@mail.gmail.com>
Message-ID: <CAAtamJZAG=Cw6WiVjkiCT7EwPwp1UhxoYmJUXLTaVTx1341OVA@mail.gmail.com>

Please refrain from any personal attacks, we are guests here. People in
this community have a right to their position. The GPL license seems to be
on their side. I don't feel any frustration.

I want to solve factual problems objectively. Formal problems formally.

Denver please, if you have any doubts we could discuss about it.

Xaver


p? 11. 9. 2020 v 20:49 odes?latel Xaver Zu <xaver1zu at gmail.com> napsal:

>
>
> p? 11. 9. 2020 v 20:36 odes?latel Mychaela Falconia <
> mychaela.falconia at gmail.com> napsal:
>
>> Hi Xaver,
>>
>> > It's about backing up that job when I've already invested time.
>>
>> Yes, I totally empathize with you on this one!
>>
>> > Is it possible in its current form to have it in any branch?
>> > Can it at least stay in the gerrit?
>>
>> Given their stance on license worship, I highly doubt that the owners
>> of osmocom.org domain and infrastructure would allow license-violating
>> stuff to be hosted anywhere under their domain.  But I can give you
>> hosting on freecalypso.org if you like - I am a proud pirate. :-)
>>
>> M~
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200911/9a2e6702/attachment.htm>

From denver at ossguy.com  Fri Sep 11 18:59:31 2020
From: denver at ossguy.com (Denver Gingerich)
Date: Fri, 11 Sep 2020 18:59:31 +0000
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJZAG=Cw6WiVjkiCT7EwPwp1UhxoYmJUXLTaVTx1341OVA@mail.gmail.com>
References: <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>
 <CAAtamJYRRkpBvQOuPzOkVvudApKhgtSy_LHdViroqPWCn5RGmw@mail.gmail.com>
 <CAAtamJZAG=Cw6WiVjkiCT7EwPwp1UhxoYmJUXLTaVTx1341OVA@mail.gmail.com>
Message-ID: <20200911185931.GZ25289@ossguy.com>

On Fri, Sep 11, 2020 at 08:55:53PM +0200, Xaver Zu wrote:
> Please refrain from any personal attacks, we are guests here. People in
> this community have a right to their position. The GPL license seems to be
> on their side. I don't feel any frustration.
> 
> I want to solve factual problems objectively. Formal problems formally.
> 
> Denver please, if you have any doubts we could discuss about it.

I'm not sure what sort of doubts you're referring to off-hand - could you be a bit more specific as to the type(s) of doubts you're particularly interested in knowing more about?

Denver
https://jmp.chat/


From mychaela.falconia at gmail.com  Fri Sep 11 19:08:36 2020
From: mychaela.falconia at gmail.com (Mychaela Falconia)
Date: Fri, 11 Sep 2020 11:08:36 -0800
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJZAG=Cw6WiVjkiCT7EwPwp1UhxoYmJUXLTaVTx1341OVA@mail.gmail.com>
References: <CAAtamJbY4mvdVQLPLygpToA=WzXJ7du4nYxC2jqpAe3dcGL9TQ@mail.gmail.com>
 <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>
 <CAAtamJYRRkpBvQOuPzOkVvudApKhgtSy_LHdViroqPWCn5RGmw@mail.gmail.com>
 <CAAtamJZAG=Cw6WiVjkiCT7EwPwp1UhxoYmJUXLTaVTx1341OVA@mail.gmail.com>
Message-ID: <CA+uuBqbr-ciwchbqcFS6Z3HWe3w2qhpCXWtn_iKXXWmm0YQtjw@mail.gmail.com>

Hi Xaver,

> Please refrain from any personal attacks, we are guests here.

What personal attacks are you talking about?  I offered you hosting on
freecalypso.org for whatever materials you may have that aren't
acceptable to osmocom.org for whatever reasons - that's an offer, not
an attack, personal or otherwise.  If you aren't interested in my
offer, that is totally fine with me - spares me the work of setting
things up for you - but please don't view me as any kind of attacker
against you.

M~


From xaver1zu at gmail.com  Fri Sep 11 19:11:45 2020
From: xaver1zu at gmail.com (Xaver Zu)
Date: Fri, 11 Sep 2020 21:11:45 +0200
Subject: Support for new SDR devices
In-Reply-To: <20200911185931.GZ25289@ossguy.com>
References: <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>
 <CAAtamJYRRkpBvQOuPzOkVvudApKhgtSy_LHdViroqPWCn5RGmw@mail.gmail.com>
 <CAAtamJZAG=Cw6WiVjkiCT7EwPwp1UhxoYmJUXLTaVTx1341OVA@mail.gmail.com>
 <20200911185931.GZ25289@ossguy.com>
Message-ID: <CAAtamJZFoL_e=OvFQfE0O4xnonZfEgHWRs+ctmgy3htxUYdRkA@mail.gmail.com>

If I felt any doubts between the lines that you didn't have, then I
apologize.

Do you think that this path is possible and correct?
I still think so.

Xaver

p? 11. 9. 2020 v 20:59 odes?latel Denver Gingerich <denver at ossguy.com>
napsal:

> On Fri, Sep 11, 2020 at 08:55:53PM +0200, Xaver Zu wrote:
> > Please refrain from any personal attacks, we are guests here. People in
> > this community have a right to their position. The GPL license seems to
> be
> > on their side. I don't feel any frustration.
> >
> > I want to solve factual problems objectively. Formal problems formally.
> >
> > Denver please, if you have any doubts we could discuss about it.
>
> I'm not sure what sort of doubts you're referring to off-hand - could you
> be a bit more specific as to the type(s) of doubts you're particularly
> interested in knowing more about?
>
> Denver
> https://jmp.chat/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200911/ea79651b/attachment.htm>

From denver at ossguy.com  Fri Sep 11 19:52:57 2020
From: denver at ossguy.com (Denver Gingerich)
Date: Fri, 11 Sep 2020 19:52:57 +0000
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJZFoL_e=OvFQfE0O4xnonZfEgHWRs+ctmgy3htxUYdRkA@mail.gmail.com>
References: <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>
 <CAAtamJYRRkpBvQOuPzOkVvudApKhgtSy_LHdViroqPWCn5RGmw@mail.gmail.com>
 <CAAtamJZAG=Cw6WiVjkiCT7EwPwp1UhxoYmJUXLTaVTx1341OVA@mail.gmail.com>
 <20200911185931.GZ25289@ossguy.com>
 <CAAtamJZFoL_e=OvFQfE0O4xnonZfEgHWRs+ctmgy3htxUYdRkA@mail.gmail.com>
Message-ID: <20200911195257.GA25289@ossguy.com>

On Fri, Sep 11, 2020 at 09:11:45PM +0200, Xaver Zu wrote:
> If I felt any doubts between the lines that you didn't have, then I
> apologize.
> 
> Do you think that this path is possible and correct?
> I still think so.

Yes, I think reimplementing libsdr.so under an AGPL-compatible license (such as GPLv3) is both possible and the correct thing to do.  I haven't looked at libsdr.so at all, mind you, so I don't have any idea how long it will take.  Hopefully it is not too long for you.

Denver
https://jmp.chat/


From xaver1zu at gmail.com  Fri Sep 11 20:36:04 2020
From: xaver1zu at gmail.com (Xaver Zu)
Date: Fri, 11 Sep 2020 22:36:04 +0200
Subject: Fwd: Support for new SDR devices
In-Reply-To: <CAAtamJbpDct8wDytjUqj2M-TEAFN9=dDaXA9kS+a=KReLKtdVw@mail.gmail.com>
References: <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <CA+uuBqZ7jzSuQA++smODsXunK+==fsMOXZYU1W_04eGDC5GpSA@mail.gmail.com>
 <CAAtamJYRRkpBvQOuPzOkVvudApKhgtSy_LHdViroqPWCn5RGmw@mail.gmail.com>
 <CAAtamJZAG=Cw6WiVjkiCT7EwPwp1UhxoYmJUXLTaVTx1341OVA@mail.gmail.com>
 <20200911185931.GZ25289@ossguy.com>
 <CAAtamJZFoL_e=OvFQfE0O4xnonZfEgHWRs+ctmgy3htxUYdRkA@mail.gmail.com>
 <20200911195257.GA25289@ossguy.com>
 <CAAtamJbpDct8wDytjUqj2M-TEAFN9=dDaXA9kS+a=KReLKtdVw@mail.gmail.com>
Message-ID: <CAAtamJb66yCateWxTvR-2wqADiTnzpsiefHHaGde7548-Y0a=w@mail.gmail.com>

That library does a lot of work. Configures HW, sends samples to FPGA via
DMA, does calibration, normalization, format conversions ... Highly
optimized for Intel CPU. If everything were to be done in an equivalently
closed version, it would take several years of work for me. I would like to
start with a simple proof-of concept. Implementing only the functions used
in osmo-trx-pciesdr. In the first phase, these can be only dummy functions
writing or reading zeros.

I assume that the GPL violation problem will be solved from the moment I
will be able to build the library and link it with osmo-trx and run it.
The next step is just a matter of effort from anyone in the community.

Xaver
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200911/c2bab41d/attachment.htm>

From laforge at osmocom.org  Fri Sep 11 20:33:48 2020
From: laforge at osmocom.org (Harald Welte)
Date: Fri, 11 Sep 2020 22:33:48 +0200
Subject: LimeSDR (was Re: Alternatives to CalypsoBTS?)
In-Reply-To: <CA+uuBqaW=j7g+vz95e9f1y3UE=vXwCb6bxN+xCJ0QmdFvMu0JQ@mail.gmail.com>
References: <CA+uuBqavNt2r-oE=ZcmvXB5HOe2K1uPRrW1Qqjimjc_=X8xviw@mail.gmail.com>
 <F7DB1848-C3E7-4751-9E9A-584433EBE281@tomcsanyi.net>
 <CA+uuBqaW=j7g+vz95e9f1y3UE=vXwCb6bxN+xCJ0QmdFvMu0JQ@mail.gmail.com>
Message-ID: <20200911203348.GE1165387@nataraja>

Hi Mychaela,

On Fri, Sep 11, 2020 at 09:54:37AM -0800, Mychaela Falconia wrote:
> > Throwing a quick glimps at the OsmoTRX wiki page I think the cheapest
> > hardware that should work well is the LimeSDR - coming in at around 300 USD.
> 
> Looking around myself, I came to the same general conclusion - but
> there are several different members of the Lime family.  It looks like
> LimeSDR-Mini is the cheapest of all of them, but does it have some
> problem compared to the original LimeSDR-USB?  

LimeSDR-USB is the oldest and best supported device.  LimeSDR mini does not
have a clock input for an external reference clock.  Yes, with some board
rework it can be added, but even then it has much stricter requirements on
the clock (IIRC phase noise), as it doesn't have any onboard PLL but directly
feeds that input to the Transceiver chip which is very sensitive to phase noise.

> Does it have a worse clock that isn't good enough to run a GSM BTS?  

Any SDR without a a built-in or external OCXO, Rubidium or GPS-DO has a clock
that is insufficient for operating a GSM BTS.

> And what about LimeNET-Micro?  It appears to be priced the same as the original
> LimeSDR-USB, but has a built-in RPi microcomputer that the software
> would run on, and its GPSDO seems like a better clock option than all
> of the predecessors - but I could be missing something...

OsmoTRX + OsmoBTS will run on it (and we actually even provide Raspbian
binar packages in our nightly and latest feeds, for people who don't
want to build everything from source).

Please note that - as far as I know - the CPU power of the RPi compute
module is insufficient for operating multiple software
transceivers/carriers within one wideband channel ('multi-arfcn mode').
So you are constrained to 1TRX operation.  If you don't need that, the
LimeNET-Micro might be a good choice.  If you do, go for a LimeSDR-USB
within the Lime device portfolio, but add a GPS-DO like a BG7TBL if you
don't already have a 10MHz reference around in your lab.

-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at osmocom.org  Fri Sep 11 20:35:29 2020
From: laforge at osmocom.org (Harald Welte)
Date: Fri, 11 Sep 2020 22:35:29 +0200
Subject: Question on osmo-trx mcbts
In-Reply-To: <20200911105002.GU1165387@nataraja>
References: <20200911105002.GU1165387@nataraja>
Message-ID: <20200911203529.GF1165387@nataraja>

Hi Tom,

sorry for the noise.

On Fri, Sep 11, 2020 at 12:50:02PM +0200, Harald Welte wrote:
> But why does the code ever only use up to three of them?

tnt has meanwhile solved that mystery (due to the inherent property of
a polyphase filter bank)

> And why is there a specific re-ordering, see radioInterfaceMulti.cpp in
> getLogicalChan() ?

We suspect it is to have a monotonically increasing frequency from logical
transceivers 0, 1, 2.

Regards,
	Harald
-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at gnumonks.org  Fri Sep 11 21:00:22 2020
From: laforge at gnumonks.org (Harald Welte)
Date: Fri, 11 Sep 2020 23:00:22 +0200
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
References: <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
Message-ID: <20200911210022.GG1165387@nataraja>

Hi Xaver,

On Fri, Sep 11, 2020 at 08:09:16PM +0200, Xaver Zu wrote:
> Is it possible in its current form to have it in any branch?

I would prefer not to have it in a branch of the official repository.

> Can it at least stay in the gerrit?

I see no problem with having it in gerrit.  I will add a related review
comment that the patch was rejected due to licensing concerns.   I don't
think copyright prevents us from publicly documenting our development
process and reltaed decisions.

> Could I solve this with an open source (GPL) reimplementation of that
> library?

Yes.  But, as other people stated, it is likely a significant effort,
and there are other SDR devices that ship with fully open source
software, sometimes not only software but even FPGA gateware or even
open source hardware.

By all means, don't let me discourage you from implementing an open
source alternative to that library.  I just think there are other
SDR devices available, which most likely have a much larger user base
(at least in terms of FOSS projects like Osmocom, srsLTE, ...)

Regards,
	Harald
-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From tom at tsou.cc  Sat Sep 12 02:23:33 2020
From: tom at tsou.cc (Tom Tsou)
Date: Fri, 11 Sep 2020 22:23:33 -0400
Subject: Question on osmo-trx mcbts
In-Reply-To: <20200911203529.GF1165387@nataraja>
References: <20200911105002.GU1165387@nataraja>
 <20200911203529.GF1165387@nataraja>
Message-ID: <CAATyM+b_Czu=WGtGRtqJJbeYJVbGvyM60GZ45EMiC2TqAK23Fw@mail.gmail.com>

Hi Harald,

Indeed, it's been awhile since I've looked at that code!

> On Fri, Sep 11, 2020 at 12:50:02PM +0200, Harald Welte wrote:
> > But why does the code ever only use up to three of them?
>
> tnt has meanwhile solved that mystery (due to the inherent property of
> a polyphase filter bank)

Yes, among the physical channels there will be one centered at
baseband and one aliased across the Nyquist boundary - the latter is
unusable. Wider bandwidth / higher channel count was possible though
limited by practicality (configuration was difficult) and RF effects
(see below).

> > And why is there a specific re-ordering, see radioInterfaceMulti.cpp in
> > getLogicalChan() ?
>
> We suspect it is to have a monotonically increasing frequency from logical
> transceivers 0, 1, 2.

Correct. The mapping itself is arbitrary. Consecutive numbering was
one simple approach.

I'll also add that one of the most difficult aspects of the
multi-carrier implementation had little to do with software. B210 was
the target device when the code was developed; the on board AD9361
RFIC was never recommended by ADI for multi-carrier operation. The DC
and aliasing effects were quite visible when using shifted carriers in
addition to the reduction in dynamic range. Conformance tests for
spectrum mask were challenging.

  -Tom


From mychaela.falconia at gmail.com  Sat Sep 12 02:39:19 2020
From: mychaela.falconia at gmail.com (Mychaela Falconia)
Date: Fri, 11 Sep 2020 18:39:19 -0800
Subject: LimeSDR (was Re: Alternatives to CalypsoBTS?)
In-Reply-To: <20200911203348.GE1165387@nataraja>
References: <CA+uuBqavNt2r-oE=ZcmvXB5HOe2K1uPRrW1Qqjimjc_=X8xviw@mail.gmail.com>
 <F7DB1848-C3E7-4751-9E9A-584433EBE281@tomcsanyi.net>
 <CA+uuBqaW=j7g+vz95e9f1y3UE=vXwCb6bxN+xCJ0QmdFvMu0JQ@mail.gmail.com>
 <20200911203348.GE1165387@nataraja>
Message-ID: <CA+uuBqa3S=A8TbVAYeSiuZeo7wMa_GV8HFe5pK7BB-nuZ3MYWA@mail.gmail.com>

Hi Harald,

Thanks for the explanation regarding different LimeSDR devices.  Just
in case I wasn't clear, the present inquiry was *not* for me - instead
I was trying to make a pre-emptive move, trying to prepare a canned
answer for that inevitable case when some very green newbie asks about
getting the anticipated cheaper-than-FCDEV3B Calypso board for the
purpose of misusing it as CalypsoBTS.  I was (and still am to some
extent) looking for a canned answer along the lines of "no, please
don't misuse Calypso devices as a poor man's BTS, they are meant to be
MS rather than BTS, if you need absolute lowest cost BTS, please use
$This_lowend_SDR instead" - and I was looking to see what that
$This_lowend_SDR metavariable should be set to.

Unfortunately the very existence of that CalypsoBTS hack has seriously
muddied the waters we all have to swim in - the world would have been
a cleaner place had that hack never been invented.  I don't know if
you have noticed it or not, but over the last few years there has been
a very significant influx of very green noobs coming to OsmocomBB not
for the purpose of doing research or tinkering from GSM MS side, but
who seek old Motorola phones for the *sole* purpose of turning them
into a poor's man BTS via that CalypsoBTS hack, and who seem to have
no interest whatsoever in GSM MS side.

Remember that guy a few years ago who was adamantly asking for a port
of OsmocomBB to Qualcomm phones and who thought it would be a slam
dunk because there apparently exist some leaked QC sources?  A careful
reading of his posts, parsing to see exactly what he sought out of OBB
and out of that hypothetical Qualcomm port, reveals that he was *not*
seeking a FOSS or research-oriented implementation of GSM MS (or of
3G/4G UE) on a newer platform, instead he was seeking to turn his
QC-based phone into a BTS in the spirit of CalypsoBTS!  I was truly
disgusted and sickened to my stomach when I saw what we was really
after - but he was not alone by any means, there are a great plenty of
them.

I assume that this attitude that leads people to seeking Motorola
phones for the sole purpose of turning them into CalypsoBTS stems out
of ignorance, rather than intentional hostility toward those who work
on the MS side of the air interface - but ignorance needs to be
treated with education, we need to educate those green newbies that
GSM MS devices make a *very* poor BTS and that they should get
something more appropriate for the BTS role - and my question was
exactly what should be recommended to those greenest of newbies.

> Please note that - as far as I know - the CPU power of the RPi compute
> module is insufficient for operating multiple software
> transceivers/carriers within one wideband channel ('multi-arfcn mode').
> So you are constrained to 1TRX operation.  If you don't need that, the
> LimeNET-Micro might be a good choice.

But this 1TRX operation is still better than CalypsoBTS, or is it not?
Remember that the goal is to convince the super-newbie to follow some
path other than CalypsoBTS...

You may not be able to relate because you are not in my shoes, but as
a maker of Calypso GSM MS devices and as the world's most active
supporter of that chipset in the present time, I stand as the front-
line target for those people seeking a Calypso device for the purpose
of misusing it as a BTS - I am typically the first person they reach
out to, usually in extremely terse inquiries in half-broken English,
typically asking for subsidized hardware, and only revealing their
true intentions several emails later, after I have already wasted a
ton of emotional energy on them.  So yeah, I seriously resent the very
existence of CalypsoBTS.

M~


From xaver1zu at gmail.com  Sun Sep 13 08:49:56 2020
From: xaver1zu at gmail.com (Xaver Zu)
Date: Sun, 13 Sep 2020 10:49:56 +0200
Subject: Support for new SDR devices
In-Reply-To: <20200911210022.GG1165387@nataraja>
References: <d02fbfec-4007-14e8-f4fa-d08f00fb4372@sysmocom.de>
 <CAAtamJZZAHJUK1oL6RfTUi8oQ-CwznJUpSJ=Af1w+9O+C-xBjw@mail.gmail.com>
 <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <20200911210022.GG1165387@nataraja>
Message-ID: <CAAtamJbOHFHsmyZfhGSt7MMRs1aDsLJ4Fb-nwWjOJfMcEkRvkA@mail.gmail.com>

> Yes.  But, as other people stated, it is likely a significant effort,
> and there are other SDR devices that ship with fully open source
> software, sometimes not only software but even FPGA gateware or even
> open source hardware.

Which SDRs with PCIe do you remember?
I know about XTRX. Master osmo-trx does not support it. And I know about
limesdr-pcie, I'm not convinced that it makes sense to buy in this state of
development.

I don't know about GPL compatible PCIe SDR with Analog Devices AD93XX
family chip. Results of SDR based on the Analog Devices versus Lime chip
are in favor of Analog Devices. I did a simple DSB modulation test. Analog
Devices has a cleaner spectrum, better carrier suppression. This is what I
noticed first on the spectrum analyzer.

Does anyone know of an SDR combining PCIe and an AD93XX chip that has the
potential to be supported by osmo-trx?
Does anyone know about such a CPRI for PCIe? I don't need it but libsdr.so
supports it. I think this can be of benefit to the community.

Xaver

p? 11. 9. 2020 v 23:00 odes?latel Harald Welte <laforge at gnumonks.org>
napsal:

> Hi Xaver,
>
> On Fri, Sep 11, 2020 at 08:09:16PM +0200, Xaver Zu wrote:
> > Is it possible in its current form to have it in any branch?
>
> I would prefer not to have it in a branch of the official repository.
>
> > Can it at least stay in the gerrit?
>
> I see no problem with having it in gerrit.  I will add a related review
> comment that the patch was rejected due to licensing concerns.   I don't
> think copyright prevents us from publicly documenting our development
> process and reltaed decisions.
>
> > Could I solve this with an open source (GPL) reimplementation of that
> > library?
>
> Yes.  But, as other people stated, it is likely a significant effort,
> and there are other SDR devices that ship with fully open source
> software, sometimes not only software but even FPGA gateware or even
> open source hardware.
>
> By all means, don't let me discourage you from implementing an open
> source alternative to that library.  I just think there are other
> SDR devices available, which most likely have a much larger user base
> (at least in terms of FOSS projects like Osmocom, srsLTE, ...)
>
> Regards,
>         Harald
> --
> - Harald Welte <laforge at gnumonks.org>
> http://laforge.gnumonks.org/
>
> ============================================================================
> "Privacy in residential applications is a desirable marketing option."
>                                                   (ETSI EN 300 175-7 Ch.
> A6)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200913/a7cca6cf/attachment.htm>

From vyanitskiy at sysmocom.de  Thu Sep 17 20:51:54 2020
From: vyanitskiy at sysmocom.de (Vadim Yanitskiy)
Date: Fri, 18 Sep 2020 03:51:54 +0700
Subject: Structure alignment problems in PCUIFv9 and PCUIFv10
Message-ID: <fa73207f-4bac-b83b-399b-3684cad64962@sysmocom.de>

Hi all,

we seem to have problems with structure alignment in the new version of
the PCUIF protocol:

  PCUIFv9:  sizeof(struct gsm_pcu_if) ->  212;  212 % 4 == 0
  PCUIFv10: sizeof(struct gsm_pcu_if) -> 1006; 1006 % 4 != 0

I think we would need to add/remove some padding.  The question is
whether we should make sure that all structures are aligned, or having
the top level struct gsm_pcu_if aligned would be enough?

Even in PCUIFv9 not all structures are properly aligned:

  sizeof(struct gsm_pcu_if_data_cnf_dt) -> 21; 21 % 4 -> 1,
  sizeof(struct gsm_pcu_if_rts_req)     -> 13; 13 % 4 -> 1,
  sizeof(struct gsm_pcu_if_rach_ind)    -> 15; 15 % 4 -> 3,
  sizeof(struct gsm_pcu_if_pag_req)     -> 11; 11 % 4 -> 3,
  sizeof(struct gsm_pcu_if_susp_req)    -> 11; 11 % 4 -> 3.

I devise by 4 because the widest member is uint32_t in all cases.

Kind regards,
Vadim.

-- 
- Vadim Yanitskiy <vyanitskiy at sysmocom.de>    http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte


From ivan.babanov at gmail.com  Fri Sep 18 20:44:43 2020
From: ivan.babanov at gmail.com (Ivan Babanov)
Date: Fri, 18 Sep 2020 23:44:43 +0300
Subject: working dahdi config for Ericsson RBS 2000
Message-ID: <CACWjWChEvUcMBZc9ra440KxRh2czs9x23gP7JepoYAPVWq0Oxw@mail.gmail.com>

Hello!

Can anybody share a working /etc/dahdi/system.conf file? I'm trying to
bring up RBS 2000 with osmo-bsc and I'm not sure that I have correct
configuration.
1. Which signaling should be used? CAS? CCS? RBS?
2. There is the same thing with framing. hdb3?
3. parameter "dslot=1,4" pointed in RBS2000 wiki looks wrong. dahdi_cnf
does not work with it at least now.

Thank you
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200918/d066bdd0/attachment.htm>

From vyanitskiy at sysmocom.de  Fri Sep 18 17:52:17 2020
From: vyanitskiy at sysmocom.de (Vadim Yanitskiy)
Date: Sat, 19 Sep 2020 00:52:17 +0700
Subject: Structure alignment problems in PCUIFv9 and PCUIFv10
In-Reply-To: <fa73207f-4bac-b83b-399b-3684cad64962@sysmocom.de>
References: <fa73207f-4bac-b83b-399b-3684cad64962@sysmocom.de>
Message-ID: <631054d3-fb99-a440-d94a-ff4e1e621d17@sysmocom.de>

Hi all,

we seem to have problems with structure alignment in the new version of
the PCUIF protocol:

  PCUIFv9:  sizeof(struct gsm_pcu_if) ->  212;  212 % 4 == 0
  PCUIFv10: sizeof(struct gsm_pcu_if) -> 1006; 1006 % 4 != 0

I think we would need to add/remove some padding.  The question is
whether we should make sure that all structures are aligned, or having
the top level struct gsm_pcu_if aligned would be enough?

Even in PCUIFv9 not all structures are properly aligned:

  sizeof(struct gsm_pcu_if_data_cnf_dt) -> 21; 21 % 4 -> 1,
  sizeof(struct gsm_pcu_if_rts_req)     -> 13; 13 % 4 -> 1,
  sizeof(struct gsm_pcu_if_rach_ind)    -> 15; 15 % 4 -> 3,
  sizeof(struct gsm_pcu_if_pag_req)     -> 11; 11 % 4 -> 3,
  sizeof(struct gsm_pcu_if_susp_req)    -> 11; 11 % 4 -> 3.

I devise by 4 because the widest member is uint32_t in all cases.

Kind regards,
Vadim.

-- 
- Vadim Yanitskiy <vyanitskiy at sysmocom.de>    http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte



From laforge at osmocom.org  Fri Sep 18 21:22:42 2020
From: laforge at osmocom.org (Harald Welte)
Date: Fri, 18 Sep 2020 23:22:42 +0200
Subject: working dahdi config for Ericsson RBS 2000
In-Reply-To: <CACWjWChEvUcMBZc9ra440KxRh2czs9x23gP7JepoYAPVWq0Oxw@mail.gmail.com>
References: <CACWjWChEvUcMBZc9ra440KxRh2czs9x23gP7JepoYAPVWq0Oxw@mail.gmail.com>
Message-ID: <20200918212242.GJ1611880@nataraja>

Hi Ivan,

On Fri, Sep 18, 2020 at 11:44:43PM +0300, Ivan Babanov wrote:

> Can anybody share a working /etc/dahdi/system.conf file? 

as usual, it depends on your configuration.  The RBS can be configured to do E1 or T1,
and in case of E1 it can be configured to do CRC4 or not.  That needs to 
reflect on your DAHDI side.  Similarly, the E1 timing source (which is very critical, as
the DAHDI cards don't have a clock source stable enough for the RBS to be reliably happy with)

> 1. Which signaling should be used? CAS? CCS? RBS?
> 2. There is the same thing with framing. hdb3?

For a E1 line with CRC4 enabled, you need:

span=1,1,0,ccs,hdb3,crc4

> 3. parameter "dslot=1,4" pointed in RBS2000 wiki looks wrong. dahdi_cnf
> does not work with it at least now.

IIRC it's called 'dchan', not 'dslot'.  But we configure that from
osmo-bsc now, so you can simply not specify any D-channels from
system.conf.  Simply use "bchan=1-31".

-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at gnumonks.org  Wed Sep 16 16:58:31 2020
From: laforge at gnumonks.org (Harald Welte)
Date: Wed, 16 Sep 2020 18:58:31 +0200
Subject: Support for new SDR devices
In-Reply-To: <CAAtamJbOHFHsmyZfhGSt7MMRs1aDsLJ4Fb-nwWjOJfMcEkRvkA@mail.gmail.com>
References: <0f66b036-f789-b630-5fd4-0660dfef57ce@sysmocom.de>
 <20200817101701.GM2916512@nataraja>
 <CAAtamJZtoU2H=KPsiBKvOJsEXRm316VV0zM8KTJ09w+Bp2ODdQ@mail.gmail.com>
 <20200911092952.GQ1165387@nataraja>
 <CAAtamJaOLfOC1aUTaswaThYe7cpcJwevRKh72E=sYngqg5ctwQ@mail.gmail.com>
 <20200911151332.GD1165387@nataraja>
 <CA+uuBqZ-2XGZf5WSDJ5vA+Xs3OhfFENDVpqMe3hst5gncFrqxg@mail.gmail.com>
 <CAAtamJYvpuyywACuwGwjjvSU980xX_sZ8gXHa5m6zoqnPEx_nQ@mail.gmail.com>
 <20200911210022.GG1165387@nataraja>
 <CAAtamJbOHFHsmyZfhGSt7MMRs1aDsLJ4Fb-nwWjOJfMcEkRvkA@mail.gmail.com>
Message-ID: <20200916165831.GN1611880@nataraja>

Hi Xaver,

On Sun, Sep 13, 2020 at 10:49:56AM +0200, Xaver Zu wrote:
> > Yes.  But, as other people stated, it is likely a significant effort,
> > and there are other SDR devices that ship with fully open source
> > software, sometimes not only software but even FPGA gateware or even
> > open source hardware.
> 
> Which SDRs with PCIe do you remember?

I was indeed thinking about LimeSDR-PCIe and XTRX.

> I know about XTRX. Master osmo-trx does not support it. 

well, neither does osmo-trx have for PCIe-SDR :P  There were some patches submitted
by Fairwaves to gerrit.osmocom.org (https://gerrit.osmocom.org/c/osmo-trx/+/15685)
which received reasonable feedback, but Fairwaves did not have the time/capacity
to follow-up and re-submit.

Given that you have demonstrated you can implement an entire osmo-trx backend
and submit it to osmcoom and go through the review cycle, maybe you can convince
Fairwaves that they should support you with a free or subsidized XTRX if you 
move their patches ahead and get them merged?

> And I know about limesdr-pcie, I'm not convinced that it makes sense
> to buy in this state of development.

I'm not following the detailed status of this device.  We do have one at
sysmocom, but our focus has been on the USB-attached devices.

> I don't know about GPL compatible PCIe SDR with Analog Devices AD93XX
> family chip. 

I'm not aware of any, but I'm not the expert on available SDR hardware,
to be honest.

> Results of SDR based on the Analog Devices versus Lime chip
> are in favor of Analog Devices. 

That reflects the general impression I get from many people so far.

> Does anyone know of an SDR combining PCIe and an AD93XX chip that has the
> potential to be supported by osmo-trx?

> Does anyone know about such a CPRI for PCIe? I don't need it but libsdr.so
> supports it. I think this can be of benefit to the community.

The main problem regarding CPRI interfaces is that there are no CPRI
radio heads with publicly documented "command and control" interfaces.
While the I/Q sample formats on CPRI/eCPRI are standardized, the
command+control is not.  So unless somebody reverse engineers those
interfaces within CPRI, there will not be any radio head you can use.

And without any usable radio head, why would anyone build a CPRI PCIe
interface card.

Some people within Osmocom had started to investigate commercially
available CPRI radio heads e.g. from Ericsson, but I think it's
primarily a "lack of resources / contributions" situation.  Several
people in Osmocom do have the equipment and even purchased some R&S
CPRI test equipment.

I'm sure if a radio head was available with (published or reverse
engineered) command and control, people would be interested in building
a PCIe CPRI interface.  sysmocom might even be interested in
contributing financially to such a development at such a point in time.

Regards,
	Harald
-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at gnumonks.org  Mon Sep 21 12:57:34 2020
From: laforge at gnumonks.org (Harald Welte)
Date: Mon, 21 Sep 2020 14:57:34 +0200
Subject: Structure alignment problems in PCUIFv9 and PCUIFv10
In-Reply-To: <fa73207f-4bac-b83b-399b-3684cad64962@sysmocom.de>
References: <fa73207f-4bac-b83b-399b-3684cad64962@sysmocom.de>
Message-ID: <20200921125734.GN2023771@nataraja>

On Fri, Sep 18, 2020 at 03:51:54AM +0700, Vadim Yanitskiy wrote:
> we seem to have problems with structure alignment in the new version of
> the PCUIF protocol:
> 
>   PCUIFv9:  sizeof(struct gsm_pcu_if) ->  212;  212 % 4 == 0
>   PCUIFv10: sizeof(struct gsm_pcu_if) -> 1006; 1006 % 4 != 0

the total size of the struct doesn't really matter that much.
> 
> I think we would need to add/remove some padding.  The question is
> whether we should make sure that all structures are aligned, or having
> the top level struct gsm_pcu_if aligned would be enough?

I think what is important is that the individual fields / members are aligned
at the natural alignment boundary of the most common architectures (so,
let's say to DWORD boundary).  Of course, for an uint8_t it may not be as relevant as for an
unaligned uint32_t in the middle of a struct.

Otherwise each access to a member will cause unaligned accesses, which may
be more expensive depending on your architecture.  Even though ARMV7 suppors
unaligned loads/stores, they are apparently still slower than aligned ones.

For an INFO_IND that doesn't matter, but for primitives/message we exchange
at high frequency it may matter.

Regards,
	Harald
-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From vyanitskiy at sysmocom.de  Wed Sep 23 10:40:39 2020
From: vyanitskiy at sysmocom.de (Vadim Yanitskiy)
Date: Wed, 23 Sep 2020 17:40:39 +0700
Subject: Application specific VTY attributes
Message-ID: <b79fc27a-e7e5-86c1-01a5-fd7828b3f8e8@sysmocom.de>

Hi Philipp,

while reading your comments to:

https://gerrit.osmocom.org/c/osmo-bsc/+/19670
https://gerrit.osmocom.org/c/osmo-mgw/+/20250

I realized that it would be better to discuss here.

My initial idea was that the absence of attributes (in the command
definition and thus in the VTY reference) implicitly indicates that a
given VTY command in the 'config' node comes into effect immediately.
This way we would only need to add attributes to commands requiring
program restart or reestablishment of some link(s) / connection(s).

However, in some applications *most* of the commands may require full
program restart. Or the opposite: most commands may apply immediately.
Adding same attributes to every command is annoying, so I came up with a
concept of default 'applies when' policy: what if we add a new field to
'struct cmd_node' indicating default attributes of all commands
belonging to it?

  struct cmd_node foo_node = {
    .node = FOO_NODE,
    .prompt = "%s(config-net)# ",
    .vtysh = 1,
    .usrattr = (1 << FOO_VTY_ATTR_RESTART_FULL),  // (!)
  };

This way a DEFUN() command definition would inherit foo_node.usrattr,
and a DEFUN_USRATTR() command definition would override it.

dexter wrote:
> I just wonder if there could be a VTY_ATTR_RESTART_FULL predefined in
> libosmore. Almost any of the osmo program will need it. Maybe also an
> VTY_ATTR_RESTART_IMMEDIATE?

ACK.  I think they could be even moved to generic attributes:

  /*! Attributes (flags) for \ref cmd_element */
  enum {
    CMD_ATTR_DEPRECATED          = (1 << 0),
    CMD_ATTR_HIDDEN              = (1 << 1),
    CMD_ATTR_APPLY_IMMEDIATE     = (1 << 2),
    CMD_ATTR_APPLY_FULL_RESTART  = (1 << 3),
  };

dexter wrote:
> Maybe we could have some standard letters defined in libosmocore
> for standard situations:
> F = Full restart required
> I = Applied Immediately

ACK.  We can also agree that generic (pre-defined) attributes use upper
case letters, while the application specific ones use lower case?

What do you think about these ideas?  If you agree, and nobody has any
objections, I can quickly implement them in libosmocore.  Would be also
nice to know what other developers think. Neels, Harald, Pau, Daniel?

Best regards,
Vadim.

-- 
- Vadim Yanitskiy <vyanitskiy at sysmocom.de>    http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte


From pmaier at sysmocom.de  Wed Sep 23 13:03:10 2020
From: pmaier at sysmocom.de (Philipp Maier)
Date: Wed, 23 Sep 2020 15:03:10 +0200
Subject: Application specific VTY attributes
In-Reply-To: <b79fc27a-e7e5-86c1-01a5-fd7828b3f8e8@sysmocom.de>
References: <b79fc27a-e7e5-86c1-01a5-fd7828b3f8e8@sysmocom.de>
Message-ID: <43ea8ddc-0316-066b-8f70-4da09478d112@sysmocom.de>

Hello Vadim.

> My initial idea was that the absence of attributes (in the command
> definition and thus in the VTY reference) implicitly indicates that a
> given VTY command in the 'config' node comes into effect immediately...

I also think that this makes sense, because as far as I can see this 
predefinition would also affect interactive VTY commands. Those always 
have an immediate effect. Also we prefer if commands apply immediately 
when the implementation permits it.

> 
>    struct cmd_node foo_node = {
>      .node = FOO_NODE,
>      .prompt = "%s(config-net)# ",
>      .vtysh = 1,
>      .usrattr = (1 << FOO_VTY_ATTR_RESTART_FULL),  // (!)
>    };
> 
> This way a DEFUN() command definition would inherit foo_node.usrattr,
> and a DEFUN_USRATTR() command definition would override it.

Makes sense I think, however, this will be more important in the future. 
Right now we need to look at each VTY command individually and check 
what changes it causes and when. I have now done the common part of 
osmo-bts, but for the moment I am using comments until the API is stable 
again.

> ACK.  We can also agree that generic (pre-defined) attributes use upper
> case letters, while the application specific ones use lower case?

Probably makes sense, there will also be never collisions should we add 
more predefined attributes later.

> What do you think about these ideas?  If you agree, and nobody has any
> objections, I can quickly implement them in libosmocore.  Would be also
> nice to know what other developers think. Neels, Harald, Pau, Daniel?

I do not have any counter-arguments. I think your ideas make sense.

I still question myself if CMD_ATTR_APPLY_FULL_RESTART should be used on 
every CONFIGURE TERMINAL command. A full restart will always cause the 
changes to be applied, or is the policy only to set the attribute of the 
minimal condition that causes the change to apply?.

Best regards,
Philipp
-- 
Philipp Maier <pmaier at sysmocom.de>              http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte


From nhofmeyr at sysmocom.de  Wed Sep 23 13:58:31 2020
From: nhofmeyr at sysmocom.de (Neels Hofmeyr)
Date: Wed, 23 Sep 2020 15:58:31 +0200
Subject: Application specific VTY attributes
In-Reply-To: <b79fc27a-e7e5-86c1-01a5-fd7828b3f8e8@sysmocom.de>
References: <b79fc27a-e7e5-86c1-01a5-fd7828b3f8e8@sysmocom.de>
Message-ID: <20200923135831.GA20875@my.box>

On Wed, Sep 23, 2020 at 05:40:39PM +0700, Vadim Yanitskiy wrote:
>   struct cmd_node foo_node = {
>     .usrattr = (1 << FOO_VTY_ATTR_RESTART_FULL),  // (!)
>   };

this way we could no longer see that a newly added command failed to set the
correct user attribute. i.e. I add a new DEFUN, I fail to remember setting
attributes, and thus it shows some attribute which is wrong. If each command
gets explicit attributes, errors like that are unlikely: If I add a DEFUN and
forget to set the right attribute, there will be no attribute, and a user
reading the vty reference will ask us to fix, without the need to try and fail
first.

> dexter wrote:
> > I just wonder if there could be a VTY_ATTR_RESTART_FULL predefined in
> > libosmore. Almost any of the osmo program will need it. Maybe also an
> > VTY_ATTR_RESTART_IMMEDIATE?
> 
> ACK.  I think they could be even moved to generic attributes:
> 
>   /*! Attributes (flags) for \ref cmd_element */
>   enum {
>     CMD_ATTR_DEPRECATED          = (1 << 0),
>     CMD_ATTR_HIDDEN              = (1 << 1),
>     CMD_ATTR_APPLY_IMMEDIATE     = (1 << 2),
>     CMD_ATTR_APPLY_FULL_RESTART  = (1 << 3),
>   };

libosmocore defined attrs would be useful. That would also require some
_VTY_ATTR_LAST enum value so that programs can add own attributes that don't
overlap.

Not so sure about these CMD attributes including the left shift. It would
duplicate things, because we do need a plain integer enum to trivially hit the
right attribute indexes. See also osmo_fsm state and event values.

I know that the VTY code disregards osmo_ completely, but I'd still favor
prepending OSMO_* to new enum values.

{
btw, each and every osmo_fsm caller and now also each and every vty attribute
defining code defines a left-shift macro like

	#define S(x) (1 << (x))

it's a bit silly to repeat this definition all over the place.
Added to libosmocore, it would have to be called OSMO_S()...
makes everything longer. maybe nevermind, just thinking aloud.
}

> dexter wrote:
> > Maybe we could have some standard letters defined in libosmocore
> > for standard situations:
> > F = Full restart required
> > I = Applied Immediately
> 
> ACK.  We can also agree that generic (pre-defined) attributes use upper
> case letters, while the application specific ones use lower case?

My opinion is not strong here.

I guess the argument is that if we want to add a letter to libosmocore later,
we have to check each and every osmo-* program to avoid collisions?

I think the effort to do this explicitly is minor, so we can do it like cmdline
options: each program assigns its own letters completely?
Then again, 'F' should mean the same thing in various osmo-* programs?

But consider, having attributes of upper case and lower case of the same letter
could be somewhat confusing to users. Might as well just use numbers, like for
additives on a fast food menu...

libosmocore assigns letters, applications assign numbers and special
characters??

~N
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200923/c483844e/attachment.bin>

From vyanitskiy at sysmocom.de  Wed Sep 23 15:48:23 2020
From: vyanitskiy at sysmocom.de (Vadim Yanitskiy)
Date: Wed, 23 Sep 2020 22:48:23 +0700
Subject: Application specific VTY attributes
In-Reply-To: <20200923135831.GA20875@my.box>
References: <b79fc27a-e7e5-86c1-01a5-fd7828b3f8e8@sysmocom.de>
 <20200923135831.GA20875@my.box>
Message-ID: <658e5dd2-e656-c37e-9257-232c9a3670a7@sysmocom.de>

Hi Neels,

On 9/23/20 8:58 PM, Neels Hofmeyr wrote:
> libosmocore defined attrs would be useful. That would also require
> some _VTY_ATTR_LAST enum value so that programs can add own attributes
> that don't overlap.

it needs to be clarified that we have two kinds of attributes: global
(the ones defined in libosmocore, like CMD_ATTR_DEPRECATED) and
application specific. Both are stored in separate bit-masks, not
together.  So there can be no overlap between the flag values.

My proposal is to add CMD_ATTR_APPLY_{FULL_RESTART,IMMEDIATE} as global
attributes, and still reserve 32 bits for application specific ones.

> I know that the VTY code disregards osmo_ completely, but I'd still
> favor prepending OSMO_* to new enum values.

ACK, makes sense.  We can also prepend 'OSMO_' to the existing symbols
and add backwards-compatibility defines.  Fortunately, only two
attributes exist at the moment: CMD_ATTR_{DEPRECATED,HIDDEN}.

> it's a bit silly to repeat this definition all over the place.
> Added to libosmocore, it would have to be called OSMO_S()...
> makes everything longer. maybe nevermind, just thinking aloud.

I don't think it's a big deal to add a one-liner define.  AFAIR, Max
tried to get such macros merged to libosmocore, but his patch was rejected.

> My opinion is not strong here.
>
> I guess the argument is that if we want to add a letter to libosmocore
> later, we have to check each and every osmo-* program to avoid
> collisions?
>
> ...
>
> libosmocore assigns letters, applications assign numbers and special
> characters??

As I already mentioned, the attribute values themselves do not overlap,
but if we assign flag letters to the globally defined attributes, then
there is a chance that the letters would overlap.

What I also forgot to point out is that we can have up to 8 global
attributes and up to 32 application specific attributes, because:

  struct cmd_element {
    // ...
    unsigned char attr;   /*!< Command attributes (global) */
    unsigned int usrattr; /*!< Command attributes (program specific)*/
  };

Given that we're not going to have more than 8 global attributes, and
not going to assign anything to both CMD_ATTR_{DEPRECATED,HIDDEN}, I
think reserving something like '*' and '!' would be enough.

Best regards,
Vadim.

-- 
- Vadim Yanitskiy <vyanitskiy at sysmocom.de>    http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200923/fd1bacb5/attachment.bin>

From nhofmeyr at sysmocom.de  Fri Sep 25 10:38:02 2020
From: nhofmeyr at sysmocom.de (Neels Hofmeyr)
Date: Fri, 25 Sep 2020 12:38:02 +0200
Subject: Fwd: OpenAirInterface releases 5G core network code
Message-ID: <20200925103802.GA4864@my.box>

Possibly interesting to note this OAI announcement.

Unfortunately none of the links are direct, but all of them are ugly tracking
links.  Also ads for facetwoogle. So I had to blank out a lot. Also interesting
to note how they don't self-host, but rather spread their code across *both*
hosted-gitlab and github. Also interesting how the From: address is a gmail one
vs. the contact at openairinterface.org further below. In a massive herd effort,
folks are fudging up the internet, no news there at least.

~N

----- Forwarded message from OpenAirInterface Software Alliance <osalliance5g at gmail.com> -----

Date: Fri, 25 Sep 2020 05:40:17 +0000
From: OpenAirInterface Software Alliance <osalliance5g at gmail.com>
To: nhofmeyr at sysmocom.de
Subject: OpenAirInterface releases 5G core network code
X-Mailer: MailChimp Mailer - **CIDbb030ed5232925faf6b6**




** OpenAirInterface releases 5G core network code
------------------------------------------------------------

Dear Community,

The OpenAirInterface Software Alliance is pleased to announce the release of the OAI 5G Core Network code. An initial set of features is already available and developments with a solid team are underway to build the rest of the 3GPP 5G SBA core. The project home page is here (<blanked-tracking-link>)  and gives a good overview of the development phases and the roadmap.

The AMF (<blanked-tracking-link>) and the SMF (<blanked-tracking-link>) code is available on GITLAB. While working to release the UPF code, we are working with the SPGW-U (<blanked-tracking-link>)  on the GITHUB.

A CI bench is already operational and in place. Any new features or developments introduced by community developers will automatically launch the CI pipeline and changes will only merge after having thoroughly been tested against third party testers and the OAI 5G RAN test benches.

We cordially invite the community to contribute to the developments of the OAI 5G core network and reach out to contact at openairinterface.org for any questions and contribution proposals.

The OpenAirInterface Software Alliance Team*

[removing a ton of tracking links]

----- End forwarded message -----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20200925/c49e3f3d/attachment.bin>

From keith at rhizomatica.org  Sat Sep 26 09:26:14 2020
From: keith at rhizomatica.org (Keith)
Date: Sat, 26 Sep 2020 11:26:14 +0200
Subject: Fwd: OpenAirInterface releases 5G core network code
In-Reply-To: <20200925103802.GA4864@my.box>
References: <20200925103802.GA4864@my.box>
Message-ID: <182b26f5-ac69-1b5c-31db-deedde013a6d@rhizomatica.org>


On 25/09/2020 12:38, Neels Hofmeyr wrote:
> Possibly interesting to note this OAI announcement.

Thanks. interesting indeed. a tl;dr. maybe? - "we have some cr*p we'd
like to push onto the world, and as you open source "community" people
are so gullible, here's a begging email to get you to help build it for
us (for free), or at least track your interest in our real operation -
data mining.

Why would one give any more time than what it takes to write this email
to this bunch of mostly white middle aged european men? + (the one token
woman - from Fujitsu)
https://www.openairinterface.org/?page_id=2304 (yes, that link will
track you too)

>
> Unfortunately none of the links are direct, but all of them are ugly tracking
> links.  Also ads for facetwoogle. So I had to blank out a lot. Also interesting
> to note how they don't self-host, but rather spread their code across *both*
> hosted-gitlab and github. 
Thanks for the redaction. Hey, but... isn't it of far more importance to
dedicate our time and resources to developing extractive unjust
eco-unfriendly technology with almost limitless potential for abusive
usage, than fiddling around with silly decentralised things like self
hosting.
> Also interesting how the From: address is a gmail one
> vs. the contact at openairinterface.org further below. In a massive herd effort,
> folks are fudging up the internet, no news there at least.

Interesting. I might have expected the domain to point at gmail then,
but openairinterface.org actually announces an MX of the same name in
AS12876 - "ONLINE_NET_DEDICATED_SERVERS_NL"

Still, there are many popular way to have your email end up on google's
servers. (Like writing to this email list for example.)

IMHO, the "internet" was pretty well fudged up since last century, it's
just that we are only starting to really notice now. There is so much to
be done to build a less dangerous inter-network, that the last thing I
will be doing is touching anything that even smells vaguely of 5G.

Sorry for the rant on this list, but well it is Saturday and there's a
pandemic on, a migration crisis and looming climate disasters, which
shouldn't overshadow the fact that some many people have been sick and
starving for decades anyway. but hey - let's build a whole bunch of
unneeded technology and use words like community, alliance,
democratising and last but not least, least abuse the sh*t once again
out of the word "open".

:-(






From laforge at osmocom.org  Sat Sep 26 17:50:13 2020
From: laforge at osmocom.org (Harald Welte)
Date: Sat, 26 Sep 2020 19:50:13 +0200
Subject: libosmocore stderr log performance / blocking
Message-ID: <20200926175013.GA18730@nataraja>

Hi!

>From time to time we've been discussing logging related stalling of osmocom
programs, particularly in the context of the stderr log target.

In general, when we log to stderr, we perform a blocking, buffered
write.  So any slow output on stderr will slow down the entire osmocom program.

I've taken some time to play with this using bpftrace and usdt (static userspace
probes) in order to determine the latency of log statements in osmocom programs.

My test program doesn't do anything else but an endless LOGP() of verying length
(from 1..512 characters).  It was executed on a Lenovo Thinkpad X260
(i7-6600U, 16GB RAM, SATA SSD with btrfs on top of dm-crypt).

I've compared
* launching the program in uxterm
* launching the program in gnome-terminal
* launching the program in screen (detached)
* launching the program in screen (attached; printing via uxterm)
* launching the program with 2>/dev/null
* launching the program with 2>/file/on/btrfs/on/ssd
* launching the program from systemd (with journald)
* replacing stderr logging with gsmtap log target (without wqueue)

The resulting histograms of the time spent for writing the logs are
below.

The git repo containing the test program, the libosmocore usdt patch and
the bpftrace program can be found at git://git.osmocom.org/laforge/osmo-logbench.git

== launching the program in uxterm

@usecs: 
[2, 4)                30 |                                                    |
[4, 8)            199963 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)            82294 |@@@@@@@@@@@@@@@@@@@@@                               |
[16, 32)            6897 |@                                                   |
[32, 64)            1649 |                                                    |
[64, 128)             57 |                                                    |
[128, 256)           221 |                                                    |
[256, 512)          8588 |@@                                                  |
[512, 1K)          13062 |@@@                                                 |
[1K, 2K)             144 |                                                    |
[2K, 4K)              14 |                                                    |
[4K, 8K)               3 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)            28 |                                                    |



== launching the program in gnome-terminal

@usecs: 
[2, 4)             32387 |@@@                                                 |
[4, 8)            457668 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)            44408 |@@@@@                                               |
[16, 32)             466 |                                                    |
[32, 64)              63 |                                                    |
[64, 128)              9 |                                                    |
[128, 256)             3 |                                                    |
[256, 512)            53 |                                                    |
[512, 1K)              0 |                                                    |
[1K, 2K)               0 |                                                    |
[2K, 4K)               0 |                                                    |
[4K, 8K)               1 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)            39 |                                                    |
[64K, 128K)          125 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)               1 |                                                    |

So we can see that gnome-terminal tends to be faster on most stderr-writes,
but at the expens of introducing a few long-delays in the 32..128ms bucket.

== launching the program in screen (detached)

@usecs: 
[2, 4)            107649 |@@@@@@@@                                            |
[4, 8)            699672 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)            15262 |@                                                   |
[16, 32)             622 |                                                    |
[32, 64)            5840 |                                                    |
[64, 128)          42728 |@@@                                                 |
[128, 256)         14000 |@                                                   |
[256, 512)            68 |                                                    |
[512, 1K)              3 |                                                    |
[1K, 2K)               0 |                                                    |
[2K, 4K)               1 |                                                    |
[4K, 8K)               0 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)              50 |                                                    |

* launching the program in screen (attached; printing via uxterm)

@usecs: 
[2, 4)                 2 |                                                    |
[4, 8)             67887 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)            30378 |@@@@@@@@@@@@@@@@@@@@@@@                             |
[16, 32)           58881 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@       |
[32, 64)            5188 |@@@                                                 |
[64, 128)            301 |                                                    |
[128, 256)            61 |                                                    |
[256, 512)          3963 |@@@                                                 |
[512, 1K)           7603 |@@@@@                                               |
[1K, 2K)             407 |                                                    |
[2K, 4K)              42 |                                                    |
[4K, 8K)              20 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)               1 |                                                    |

so intrestingly, attaching the screen will significantly slow down stderr. It
seems screen doesn't do sufficient buffering internally, or it blocks the osmocom
program until not only it has updated its internal buffer, but also written to
the output of the attached uxterm?

== launching the program with 2>/dev/null

@usecs: 
[1]              2202080 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2, 4)             26966 |                                                    |
[4, 8)              1107 |                                                    |
[8, 16)              305 |                                                    |
[16, 32)              10 |                                                    |
[32, 64)               6 |                                                    |
[64, 128)              0 |                                                    |
[128, 256)             0 |                                                    |
[256, 512)             1 |                                                    |
[512, 1K)              0 |                                                    |
[1K, 2K)               0 |                                                    |
[2K, 4K)               0 |                                                    |
[4K, 8K)               0 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)             153 |                                                    |

unsurprisingly, this turns out to be rather fast

== launching the program with 2>/file/on/btrfs/on/ssd

@usecs: 
[2, 4)           2275464 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)            265612 |@@@@@@                                              |
[8, 16)             9384 |                                                    |
[16, 32)             368 |                                                    |
[32, 64)               4 |                                                    |
[64, 128)             24 |                                                    |
[128, 256)             3 |                                                    |
[256, 512)             1 |                                                    |
[512, 1K)              0 |                                                    |
[1K, 2K)               0 |                                                    |
[2K, 4K)               0 |                                                    |
[4K, 8K)               0 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)              95 |                                                    |

I've tried creating some load using a cocnrrent dd of the underlying block device,
but couldn't make out any difference.  I guess this shows how good the I/O
subsystem is.  Yes, some writes are slower than writing to /dev/null, but 

== launching the program from systemd (with journald)

[2, 4)           1060607 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)             75133 |@@@                                                 |
[8, 16)              724 |                                                    |
[16, 32)             107 |                                                    |
[32, 64)               8 |                                                    |
[64, 128)              8 |                                                    |
[128, 256)             0 |                                                    |
[256, 512)             0 |                                                    |
[512, 1K)              0 |                                                    |
[1K, 2K)               0 |                                                    |
[2K, 4K)               0 |                                                    |
[4K, 8K)               0 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)           38 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)               1 |                                                    |

This is also pretty fast.  There are some writes in the 64..128ms range, though.
Those are likely going to cause problems in timing critical programs like osmo-bts,
osmo-trx or osmo-mgw.

* replacing stderr logging with gsmtap log target (without wqueue)

@usecs: 
[4, 8)            776978 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)             1771 |                                                    |
[16, 32)              60 |                                                    |
[32, 64)               0 |                                                    |
[64, 128)              1 |                                                    |
[128, 256)             0 |                                                    |
[256, 512)             0 |                                                    |
[512, 1K)              0 |                                                    |
[1K, 2K)               0 |                                                    |
[2K, 4K)               0 |                                                    |
[4K, 8K)               0 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)               0 |                                                    |
[2G, 4G)              82 |                                                    |

Ignoring the strange (and certainly bogus) artefacts in the 2..4s bin at the bottom,
writes to the GSMTAP UDP socket tend to be quite fast, much faster than anything
via stderr.  Even faster than stderr-to-dev-null.

== conclusion

all-in-all, it looks not too bad at least on a relatively unloaded
couple-of-years-old laptop.  I would have expected worse.

We can definitely see that there are non-neglectable delays created by
both systemd/journald as well as gnome-terminal.

One idea would be to play with setvbuf(3) to increase stderr buffering
inside libc.

The other idea would be to do our own buffering using osmo_wqueue() on
the raw, unbuffered stderr-fd, which would then be in non-blocking mode
and handled vis osmocom select.  We'd have to add some kind of limit for
that buffer in order to avoid OOM situations.

-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at osmocom.org  Sat Sep 26 17:57:01 2020
From: laforge at osmocom.org (Harald Welte)
Date: Sat, 26 Sep 2020 19:57:01 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200926175013.GA18730@nataraja>
References: <20200926175013.GA18730@nataraja>
Message-ID: <20200926175701.GB18730@nataraja>

On Sat, Sep 26, 2020 at 07:50:14PM +0200, Harald Welte wrote:
> One idea would be to play with setvbuf(3) to increase stderr buffering
> inside libc.

I just tried; at least for gnome-terminal it did not at all change the
results, even if a 1MByte buffer was used in setvbuf().  I thoght this could
be related to our use of fflush() in stderr logging.  But even when removing
the fflush() I could still see those 64..128ms stalls when running the test
program via gnome-terminal.  Not sure what's going on, but definitely we
can stop looking at setvbuf() as a solution.

-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at gnumonks.org  Sat Sep 26 19:52:51 2020
From: laforge at gnumonks.org (Harald Welte)
Date: Sat, 26 Sep 2020 21:52:51 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200926175013.GA18730@nataraja>
References: <20200926175013.GA18730@nataraja>
Message-ID: <20200926195251.GC18730@nataraja>

Dear list,

I did some more experiments.

On Sat, Sep 26, 2020 at 07:50:14PM +0200, Harald Welte wrote:

> == launching the program in gnome-terminal
> 
> @usecs: 
> [2, 4)             32387 |@@@                                                 |
> [4, 8)            457668 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
> [8, 16)            44408 |@@@@@                                               |
> [16, 32)             466 |                                                    |
> [32, 64)              63 |                                                    |
> [64, 128)              9 |                                                    |
> [128, 256)             3 |                                                    |
> [256, 512)            53 |                                                    |
> [512, 1K)              0 |                                                    |
> [1K, 2K)               0 |                                                    |
> [2K, 4K)               0 |                                                    |
> [4K, 8K)               1 |                                                    |
> [8K, 16K)              0 |                                                    |
> [16K, 32K)             0 |                                                    |
> [32K, 64K)            39 |                                                    |
> [64K, 128K)          125 |                                                    |
> [128K, 256K)           0 |                                                    |
> [256K, 512K)           0 |                                                    |
> [512K, 1M)             0 |                                                    |
> [1M, 2M)               0 |                                                    |
> [2M, 4M)               0 |                                                    |
> [4M, 8M)               0 |                                                    |
> [8M, 16M)              0 |                                                    |
> [16M, 32M)             0 |                                                    |
> [32M, 64M)             0 |                                                    |
> [64M, 128M)            0 |                                                    |
> [128M, 256M)           0 |                                                    |
> [256M, 512M)           0 |                                                    |
> [512M, 1G)             0 |                                                    |
> [1G, 2G)               1 |                                                    |
> 
> So we can see that gnome-terminal tends to be faster on most stderr-writes,
> but at the expens of introducing a few long-delays in the 32..128ms bucket.
>
> The other idea would be to do our own buffering using osmo_wqueue() on
> the raw, unbuffered stderr-fd, which would then be in non-blocking mode
> and handled vis osmocom select.  We'd have to add some kind of limit for
> that buffer in order to avoid OOM situations.

I just implemented that.  Results are promising.

This is with gnome-terminal where above we had the significant number of 64..128ms
blocking delays.  Now:

@usecs:
[1]               404920 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2, 4)            308383 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@             |
[4, 8)              3191 |                                                    |
[8, 16)             1434 |                                                    |
[16, 32)            1438 |                                                    |
[32, 64)              65 |                                                    |
[64, 128)             22 |                                                    |
[128, 256)             9 |                                                    |
[256, 512)             9 |                                                    |


so no more than 512us for any log output line (which is within the process,
allocating a msgb + enqueueing it to the write_queue).  We have to be a bit
careful here, in order to prevent re-entrancy problems.  For example, write_queue
currently wants to log if the queue is full, which obviosly causes infinite
recursion.

We make logging more expensive this way, as we add a msgb allocation + memcpy
to each log statement.  At the same time, we remove the potential of any
write-blocking, which is probably worth it.

There are some other side-effects, though: stderr-logging no longer works
for small programs tat don't ever call osmo_select_main().  So maybe we need
to support both the old blocking as well as the new non-blocking method?

But how do we make sure everyone gets the best default behavior?  Start
with the blocking/buffered I/O and switch to the write_queue backend on
the first call to osmo_select_main()?

What do you think?

-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From pespin at sysmocom.de  Sun Sep 27 11:57:21 2020
From: pespin at sysmocom.de (Pau Espin Pedrol)
Date: Sun, 27 Sep 2020 13:57:21 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200926195251.GC18730@nataraja>
References: <20200926175013.GA18730@nataraja> <20200926195251.GC18730@nataraja>
Message-ID: <6b106423-043c-52f5-a96a-4f82f9ecff43@sysmocom.de>

Hi,

LGTM. Indeed we should keep allowing use of logging API without
requiring osmo_select_main(). I submitted several comments to the
related gerrit patch you submitted.

-- 
- Pau Espin Pedrol <pespin at sysmocom.de>         http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte


From laforge at osmocom.org  Mon Sep 28 19:40:46 2020
From: laforge at osmocom.org (Harald Welte)
Date: Mon, 28 Sep 2020 21:40:46 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200926195251.GC18730@nataraja>
References: <20200926175013.GA18730@nataraja> <20200926195251.GC18730@nataraja>
Message-ID: <20200928194046.GB3871@nataraja>

Dear list,

Vadim requested to also compare syslog based logging.  In theory, I would have expected it
to perform similar to gsmtap, given that it also doesn't do anything else but sending UDP
packets.

However, the performance looks slightly worse than with gsmtap:

@usecs:
[2, 4)              1167 |                                                    |
[4, 8)           2056285 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)             8403 |                                                    |
[16, 32)            1880 |                                                    |
[32, 64)               6 |                                                    |
[64, 128)              6 |                                                    |
[128, 256)             0 |                                                    |
[256, 512)             2 |                                                    |
[512, 1K)              1 |                                                    |
[1K, 2K)             738 |                                                    |
[2K, 4K)               2 |                                                    |
[4K, 8K)               0 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)              49 |                                                    |

Note the significant count of samples in the 1..2ms bucket.  That's still quite a lot,
compared to the usual 4..8us.

Reminder: With gsmtap loggingwe had no log lines with more than 128us delay.

The results are reproducible, I just double-checked that gsmtap defintely outperforms
syslog in terms of the maximum delay/latency caused by logging.

-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From nhofmeyr at sysmocom.de  Mon Sep 28 20:25:48 2020
From: nhofmeyr at sysmocom.de (Neels Hofmeyr)
Date: Mon, 28 Sep 2020 22:25:48 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200926195251.GC18730@nataraja>
References: <20200926175013.GA18730@nataraja> <20200926195251.GC18730@nataraja>
Message-ID: <20200928202548.GA28797@my.box>

On Sat, Sep 26, 2020 at 09:52:51PM +0200, Harald Welte wrote:
> > The other idea would be to do our own buffering using osmo_wqueue() on
[...]
> I just implemented that.  Results are promising.

I'm wondering, when logging to a wqueue, you are measuring the timing of adding
the log statement to the queue? Or also the write operation via select() later?

Working off the logging wqueue could also delay handling pending incoming
packets? So maybe it just moves latency, at least in high load situations with
a lot of incoming work pending?

When we do a select() for, say, an incoming message, and that creates a lot of
interesting logging. We'll only actually see the logging on some output in
subsequent select() loop iterations, right?

And if we get a program crash in between? Then the stderr (for example) won't
show the final logging leading up to the crash, right?

So maybe there should be some trivial cfg or cmdline switch that changes over
to non-wqueue logging at least for debugging?

~N


From pespin at sysmocom.de  Mon Sep 28 21:24:58 2020
From: pespin at sysmocom.de (Pau Espin Pedrol)
Date: Mon, 28 Sep 2020 23:24:58 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200928202548.GA28797@my.box>
References: <20200926175013.GA18730@nataraja>
 <20200926195251.GC18730@nataraja> <20200928202548.GA28797@my.box>
Message-ID: <6fdcd30e-0b1e-f2ec-00ce-f044f94b1542@sysmocom.de>

Hi,

Neels I agree with you, I already raised same concerns as comments in
the gerrit patch.

-- 
- Pau Espin Pedrol <pespin at sysmocom.de>         http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte


From laforge at osmocom.org  Tue Sep 29 09:04:39 2020
From: laforge at osmocom.org (Harald Welte)
Date: Tue, 29 Sep 2020 11:04:39 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200928202548.GA28797@my.box>
References: <20200926175013.GA18730@nataraja> <20200926195251.GC18730@nataraja>
 <20200928202548.GA28797@my.box>
Message-ID: <20200929090439.GC3871@nataraja>

Hi Neels,

On Mon, Sep 28, 2020 at 10:25:48PM +0200, Neels Hofmeyr wrote:
> On Sat, Sep 26, 2020 at 09:52:51PM +0200, Harald Welte wrote:
> > > The other idea would be to do our own buffering using osmo_wqueue() on
> [...]
> > I just implemented that.  Results are promising.
> 
> I'm wondering, when logging to a wqueue, you are measuring the timing of adding
> the log statement to the queue? Or also the write operation via select() later?

I'm only measuring the delay of adding the log statement to the queue.

> Working off the logging wqueue could also delay handling pending incoming
> packets? So maybe it just moves latency, at least in high load situations with
> a lot of incoming work pending?

For sure, that write will take some time (later). However,
* we only process one log write per select() loop, so we have some kind of fair
  sharing of opportunity with incoming packets
* I would definitely expect those writes to take less time, as contrary to right
  now, the file descriptors are non-blocking.

I'm happy to do some additional benchmarking if you'd like, but I a non-blocking
write - as the name implies - doesn't block and hence we shouldn't see any significant
delays.  I would expect this to be in the < 1ms order of magnitude, compared to the
multiple-ms to multiple hundreds of ms category.  That is of course assuming that
the CPU has time to schedule the process again after the syscall completes.  But
we have to assume that is the case; if your CPU is overloaded then of course you
will see higher latencies, no matter what.

> When we do a select() for, say, an incoming message, and that creates a lot of
> interesting logging. We'll only actually see the logging on some output in
> subsequent select() loop iterations, right?

yes, that's correct. but only one log statements per select() loop iteration to
ensure there is no starving by logging.

> And if we get a program crash in between? Then the stderr (for example) won't
> show the final logging leading up to the crash, right?

correct.

> So maybe there should be some trivial cfg or cmdline switch that changes over
> to non-wqueue logging at least for debugging?

fine with me.

-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From laforge at gnumonks.org  Tue Sep 29 09:26:00 2020
From: laforge at gnumonks.org (Harald Welte)
Date: Tue, 29 Sep 2020 11:26:00 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200929090439.GC3871@nataraja>
References: <20200926175013.GA18730@nataraja> <20200926195251.GC18730@nataraja>
 <20200928202548.GA28797@my.box> <20200929090439.GC3871@nataraja>
Message-ID: <20200929092600.GD3871@nataraja>

one more follow-up:

On Tue, Sep 29, 2020 at 11:04:49AM +0200, Harald Welte wrote:

> > And if we get a program crash in between? Then the stderr (for example) won't
> > show the final logging leading up to the crash, right?
> 
> correct.

The only way around that would be to pus the job of 'flushing the queue' to
a separate process.  So the log messages would be sent over a socket (or in
shared memory, if you want to optimize further), and that separate process then
would take care of flushing/draining the queue even after the process exited.

To get back down to earth, I think the stderr/file write_queue would
usually only contain one element to begin with.  Only in the exceptional
case of some stalling, we'd have some log messages queued up.

The fact that we currently only stall occasionally when using blocking writes
shows us how infrequent the existing stdlib buffer (or pipe buffer) is
insufficient.  That gives us an idea about how infrequent we would have
a significant size of a write queue (in excess of the current stdlib or
pipe buffering).

In the "ideal world" we would probably have a system that is somewhere
in between of tht two extremes of
* old behavior: always a synchronous, potentially blocking write
* new behavior: always asynchronous, queued non-blocking write

So what I think would be good is:

* If the write_queue is empty, attempt a synchronous, non-blocking write
  to stderr/file.
* Only if that write returns short (in the extreme case '0'), enqueue the
  [remainder of the] log message to the write_queue for delayed storage.

This way we should get the best of both worlds: More or less unmodified
behavior for 99% of all cases, while avoiding any delays in those few
situations where right now we are stalling/blocking due to a full stdio buffer.

Regards,
	Harald
-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)


From pespin at sysmocom.de  Tue Sep 29 11:07:41 2020
From: pespin at sysmocom.de (Pau Espin Pedrol)
Date: Tue, 29 Sep 2020 13:07:41 +0200
Subject: libosmocore stderr log performance / blocking
In-Reply-To: <20200929092600.GD3871@nataraja>
References: <20200926175013.GA18730@nataraja>
 <20200926195251.GC18730@nataraja> <20200928202548.GA28797@my.box>
 <20200929090439.GC3871@nataraja> <20200929092600.GD3871@nataraja>
Message-ID: <6fd95422-1145-6e41-839b-94045b030672@sysmocom.de>

Hi,

On 9/29/20 11:26 AM, Harald Welte wrote:
> So what I think would be good is:
> 
> * If the write_queue is empty, attempt a synchronous, non-blocking write
>   to stderr/file.
> * Only if that write returns short (in the extreme case '0'), enqueue the
>   [remainder of the] log message to the write_queue for delayed storage.
> 

Fine with this approach, I was going to suggest that too while reading
your last comments. The important point here is changing the fd to be
non-blocking, all the rest is basically about how to keep ordering and
delays at minimum once we hit a would-block situation.
In any case, I'd gor for: when pushing stuff to the fd from the queue,
also push until short-write is hit, not only 1 msgb per select().

> This way we should get the best of both worlds: More or less unmodified
> behavior for 99% of all cases, while avoiding any delays in those few
> situations where right now we are stalling/blocking due to a full stdio buffer.
> 

ACK.


-- 
- Pau Espin Pedrol <pespin at sysmocom.de>         http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte


From laforge at osmocom.org  Tue Sep 29 17:05:44 2020
From: laforge at osmocom.org (Harald Welte)
Date: Tue, 29 Sep 2020 19:05:44 +0200
Subject: libosmocore stderr log performance / blocking (systemd-journald)
In-Reply-To: <20200928194046.GB3871@nataraja>
References: <20200926175013.GA18730@nataraja> <20200926195251.GC18730@nataraja>
 <20200928194046.GB3871@nataraja>
Message-ID: <20200929170544.GH3871@nataraja>

Vadim also requested comparison of performance with his systemd-journald log
target in teh same benchmark.

Pleae find the results below:

systemd-journald (raw=false)

[4, 8)              1427 |                                                    |
[8, 16)           331242 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32)           95378 |@@@@@@@@@@@@@@                                      |
[32, 64)            2874 |                                                    |
[64, 128)            449 |                                                    |
[128, 256)            48 |                                                    |
[256, 512)             7 |                                                    |
[512, 1K)              1 |                                                    |
[1K, 2K)               0 |                                                    |
[2K, 4K)               0 |                                                    |
[4K, 8K)               0 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)               0 |                                                    |
[2G, 4G)              81 |                                                    |

systemd-journald (raw=true)

[4, 8)              5477 |                                                    |
[8, 16)           576683 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32)          143985 |@@@@@@@@@@@@                                        |
[32, 64)            3000 |                                                    |
[64, 128)            772 |                                                    |
[128, 256)           107 |                                                    |
[256, 512)             8 |                                                    |
[512, 1K)              0 |                                                    |
[1K, 2K)               0 |                                                    |
[2K, 4K)               0 |                                                    |
[4K, 8K)               1 |                                                    |
[8K, 16K)              0 |                                                    |
[16K, 32K)             0 |                                                    |
[32K, 64K)             0 |                                                    |
[64K, 128K)            0 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             0 |                                                    |
[1M, 2M)               0 |                                                    |
[2M, 4M)               0 |                                                    |
[4M, 8M)               0 |                                                    |
[8M, 16M)              0 |                                                    |
[16M, 32M)             0 |                                                    |
[32M, 64M)             0 |                                                    |
[64M, 128M)            0 |                                                    |
[128M, 256M)           0 |                                                    |
[256M, 512M)           0 |                                                    |
[512M, 1G)             0 |                                                    |
[1G, 2G)               0 |                                                    |
[2G, 4G)              48 |                                                    |

So we can see that both of these are actually quite good: Up to 1ms for almost all log writes
in the raw=false case, and more or less the same order of magnitude for the raw case.
Both are much better than blocking stderr + systemd picking up stderr (where latencies
up to 128ms are observed at times.

-- 
- Harald Welte <laforge at osmocom.org>            http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)