<div dir="ltr">Hi Harald,<div><br></div><div>I'm really surprising that nextepc can support such a large people.</div><div>I'd like to share my though about three good point.</div><div><br></div><div>1. ogs_expect()</div><div>I think that ogs_expect() is really good starting point to remove unnecessary ogs_assert().</div><div>So, I added my version in ogslib() repository as below.</div><div><br></div><div>#define ogs_expect(expr, fallback) \<br> do { \<br> if (ogs_likely(expr)) ; \<br> else { \<br> ogs_error("%s: Expectation `%s' failed.", OGS_FUNC, #expr); \<br> fallback; \<br> } \<br> } while (0)<br></div><div><br></div><div>Let me know if the above interface has a problem.</div><div><br></div><div>2. VTY/CTRL</div><div>I'm happy if nextepc can have such a good VTY/CTL. Indeed, this is actually needed if someone would like to test nextepc on a large scale.</div><div><br></div><div>3. Logging</div><div>I agree that IMSI, APN context is needed when logging. So, I think mme_log()/sgw_log()/... needs to be introduced. And also, I saw a fix of the freeDiameter logging. How about merging it to master branch. Please github pull request if you agree.</div><div><br></div><div>Much appreciated about your super work!</div><div><br></div><div>Best Regards,</div><div> Sukchan</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 27, 2019 at 8:00 PM Harald Welte <<a href="mailto:laforge@gnumonks.org">laforge@gnumonks.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all,<br>
<br>
During the Chaos Communication Camp 2019 (an international hacker camp<br>
with about 5500 participants) last week, there is a tradition to operate<br>
Osmocom based 2G and more recently also 3G networks.<br>
<br>
This time I operated a nextepc based 4G/LTE network next to the camp 2G/3G<br>
networks. In order to share one subscriber database, I have implemented<br>
osmo_dia2gsup, which can translate the S6a/S6d diameter into Osmocom GSUP<br>
protocol, so nextepc can be used without nextepc-hssd but with osmo-hlr instead.<br>
<br>
The network was operating six Ericsson RBS6402 in Band 7 (2600 MHz).<br>
<br>
Some more details can be found at<br>
<br>
<br>
Regarding the nextepc side:<br>
<br>
* 2439 uniqua IMSIs were seen<br>
** 147 unique IMSIs of CCC SIM cards (26242)<br>
** 2292 non-CCC IMSIs<br>
** 75 unique MCC-MNC tuples<br>
** 34 unique MCCs<br>
** The usual suspects (Europe + North America), but also...<br>
*** Malaysia, Indonesia, Australia, New Zealand, South Africa<br>
* 560 Attach accept (CCC SIM cards)<br>
* 46590 Attach reject (commercial operator SIM cards)<br>
* 629 PDN context (APN) activations<br>
* 235 handovers between cells (X2)<br>
* 64 crashes + restarts of nextepc-mme<br>
* 9 crashes + restarts of nextepc-pgw<br>
* 0 crashes + restarts of nextepc-sgw<br>
* 10 crashes + restarts of nextepc-pcrf<br>
<br>
In general, it worked quite nicely, and I have to congratulate Sukchan on<br>
his work at nextepc.<br>
<br>
I investigated some of the crashes, reported them to the issue tracker and<br>
attempted to fix some of them on-site. The actual codebase that was running<br>
can be found at <a href="https://github.com/laf0rge/nextepc/commits/laforge/cccamp19" rel="noreferrer" target="_blank">https://github.com/laf0rge/nextepc/commits/laforge/cccamp19</a><br>
<br>
>From my experience with operating such a "large" nextepc network for the first<br>
time, I have the following overall feedback, which basically boils down<br>
to three major areas:<br>
<br>
== the use of assert() ==<br>
<br>
ASSERT should never be triggered by anything that is received from another<br>
network entity. So if a eNB sends an unknown S1AP-ID, or if a SGW sends<br>
an unknown TEID, or if the NAS MAC validation fails, or a EMM message<br>
cannot be decoded - all of those must be handled gracefully without<br>
terminating the program. This 'fail fast' way of programming can be<br>
done when writing code in C++ (exceptions that are caught) or in erlang<br>
(one process per message, crashing that one doesn't bring the entire MME<br>
down).<br>
<br>
I've tried my best to review all ogs_assert() in the MME and came up<br>
with the following patch:<br>
<a href="https://github.com/laf0rge/nextepc/commit/3b528af8fd51c85769123338eb57a4635c9d699e" rel="noreferrer" target="_blank">https://github.com/laf0rge/nextepc/commit/3b528af8fd51c85769123338eb57a4635c9d699e</a><br>
which requires<br>
<a href="https://github.com/laf0rge/ogslib/commit/dc36ccbb080038306666931bdc97f6204fd5c011" rel="noreferrer" target="_blank">https://github.com/laf0rge/ogslib/commit/dc36ccbb080038306666931bdc97f6204fd5c011</a><br>
which introduces ogs_expect() and ogs_expect_or_return() macros that can<br>
be used in many places instead of ogs_assert().<br>
<br>
It would also be possible to use this kind of 'fail fast' approach in C<br>
programs, but then one would have to use longjmp() from the 'assert',<br>
and you would have to use some kind of hierarchical memory allocator so<br>
that in the 'exception handler' you can release any dynamic allocations<br>
that were made before.<br>
<br>
<br>
== the lack of introspection ==<br>
<br>
When you operate a network, it is vital to have some visibility. For the<br>
MME you want to inspect how many subscribers are currently attached, where<br>
they are attached (TAC), whether they currently have an UE Context (and at<br>
which eNB), which TMSI/GUTI was allocated, etc.<br>
<br>
Likewise, for both SGW and PGW you want to see which PDN contexts exist, from<br>
which peer IP adresses, which APN was used, what IP addresses have been allocated, etc.<br>
<br>
In the Osmocom world, we implement this introspection in two ways:<br>
* by means of the VTY interface (for the human user)<br>
* by means of the CTRL interface (for other programs)<br>
<br>
If I hadn't been busy with debugging various other issues, I would have actually<br>
attempted to add a basic VTY interface to nextepc-mmed.<br>
<br>
For sure there may be better ways to expose this state (ideally with the same<br>
piece of code providing access to both human users as well as external programs),<br>
but I'm not aware of any nice C language implementation in FOSS that one could<br>
use right away.<br>
<br>
<br>
== logging without context ==<br>
<br>
When looking at log file output, it is very important that this log file output<br>
always carry sufficient context. IF there are many subscribers acting in parallel,<br>
you need to know which subscriber / pdn context / ... a given log message relates<br>
to, otherwise the log message is rather useless.<br>
<br>
For example, if you get<br>
[mme] DEBUG: [MME] Authentication-Information-Answer (mme-fd-path.c:211)<br>
then even at DEBUG level you have no indication what so ever for which<br>
particular subscriber this AIA was received. I would normally expect<br>
that the UE is resolved from the DIAMETER session-id, and then the UEs<br>
identity (IMSI) can be printed.<br>
<br>
I also find it suboptimal that log lines often span multiple lines, which means<br>
you cannot simply 'grep' for something, as you always need to check some lines<br>
before and/or after. But I guess conrary to the lack of context, this<br>
is a matter of teste and one can have different opinions about it.<br>
<br>
<br>
I'll try to contribute as much as I can regarding bug fixes and<br>
enhancements. Thanks again for all the great work so far!<br>
<br>
Regards,<br>
Harald<br>
-- <br>
- Harald Welte <<a href="mailto:laforge@gnumonks.org" target="_blank">laforge@gnumonks.org</a>> <a href="http://laforge.gnumonks.org/" rel="noreferrer" target="_blank">http://laforge.gnumonks.org/</a><br>
============================================================================<br>
"Privacy in residential applications is a desirable marketing option."<br>
(ETSI EN 300 175-7 Ch. A6)<br>
</blockquote></div>