Hi all, for logging changes, I would welcome a comprehensive approach and maybe add an all new set of logging commands, to not confuse with the current logging behavior; keep and deprecate the old ones for users that were fine with those. (I was the one responsible for the death of 'logging level all everything', and I've stumped at least one user who now keeps patching it back in manually; let's not repeat that.) Maybe we can come up with more meaningful wording that more accurately indicate what is happening? But if the current command set suffices without confusion, we can also keep that, and maybe resurrect the 'logging level all everything'?? Here are the handles / use cases that come up -- I've embedded implementation choices and command names in this proposal, but it's intended as a blueprint to incorporate everyone else's bikesheds: A) The hardcoded logging landscape on program startup without config. (Maybe this isn't all that important, including it nevertheless.) 1) a command to reset all to hardcoded defaults? B) A "normal" logging behavior configured by the user. We need: 1) a command to set all categories at once 2) a command to tweak individual categories C) Temporarily reducing the logging on all categories: silence the log. Even though a category is "normally" set to INFO, after this command it can be forced down to only log e.g. NOTICE. But if a category is already on ERROR, leave it down there and do not lift it up to NOTICE. 1) a command to temporarily reduce all categories. 2) a command to temporarily reduce individual categories. D) Temporarily increasing the logging on all categories: show everything. Even though a category is "normally" set to ERROR, after this command it can be forced up to e.g. INFO. But if a category is already up at DEBUG, don't reduce it to NOTICE. 1) a command to temporarily increase all categories. 2) a command to temporarily increase individual categories. 3) we need to decide whether temporarily increasing is stronger/weaker than temporarily reducing. Depending on the implementation, it could also be "last one wins" (which IMHO would be best). T) Maybe we also want to unconditionally set temporary levels, whether they increase or reduce. E) Remove temporary logging to go back to the "normal" logging landscape. 1) a command to put everything back to "normal", 2) a command to put individual categories back to "normal", 3) a separate command to only remove temporary reduction, i.e lift the temporary changes only if they cause less logging than normal, 4) a separate command to only remove temporary verbosity, i.e. only lift the temporary changes if they cause more logging than normal. F) Set a specific log context, and silence all messages (regardless of their logging category setting) unless they match the context filter: logging filter imsi 123456789123456 Though this feature has probably never been properly implemented through all code paths, it is potentially very powerful for live debugging. Especially with the new FSMs around everywhere, there is a good chance that we can fairly easily improve on the current 'logging filter foo' experience. During congress, I really yearned for this to work well. 0) In the current implementation, on each individual log message, this decides whether the context matches, and if not, the message is discarded. 1) Instead of discarding, we may want a level to reduce to. 2) Or maybe we want to leave the logging landscape unchanged, but lift the level for all logs that match the set context? 3) disable individual filters 4) disable all filters. This is currently 'logging filter all 1' but that is confusing, we could use a meaningful synonym, and separate from filters. G) toggle all logging in the telnet vty 1) silence logging (currently 'logging filter all 0', confusing) 2) continue logging (currently 'logging filter all 1', confusing) A bit of a meaningless image of different logging layers: level ^ Hardcoded defaults.. | ... - ====== =.... Normal logging landscape== | == .== ... ==... -- Temporary levels-- | -- =.... ...== - = | = -- ---- -=== |________________________> categories Same image with actual current logging overlayed: level ^ | ... # =###== =.... | == .== ... ==... ## Resulting levels ## | ## #.... ...## # = | # ## #### #=#= |________________________> categories - the hardcoded defaults are immutable. - the normal landscape can be freely chosen. - Some categories may have temporary tweaks. Where there are temporary tweaks, they take precedence as-is. The vty command ("make stronger" or "make weaker") decided whether to put a temporary level in place or not. Can you guys explain the current patch proposals in terms of above features? (like, use the capital letters A-G for reference) Feel free to change this vty transcript proposal i've tossed up as I went: ! A.1) hardcoded defaults logging hard-reset rsl,chan,msc logging hard-reset ! B) set normal categories ! B.1) all logging set notice all ! B.2) individual logging set notice rsl,chan,msc ! C) Temporarily reducing ! C.1) all logging reduce notice all ! C.2) individual logging reduce error l1p,nm,cc ! D) Temporarily increasing ! D.1) all logging increase info all ! D.2) individual logging increase debug mm,rr ! D.3) last one wins logging reduce error l1p,mm,rr logging increase debug mm,rr logging reduce notice rr ! result: l1p=error mm=debug rr=notice ! T) set temporary level unconditionally logging set-temp error all logging set-temp error mm,rr ! logging (recrease|induce) ;) ! E.3) remove temporary verbosity no logging reduce nm,cc no logging reduce all ! E.4) remove temporary silencing no logging increase nm,cc no logging increase all ! E.2) snap back to normal levels logging reset rsl,nm,rr ! E.1) ...for all categories logging reset all ! F.0) context filter logging filter imsi 123456789123456 ! F.1) choose context filter behavior logging filter-reduce error logging filter-discard ! F.2) logging filter-increase debug ! F.3) disable filter no logging filter imsi ! F.4) no logging filter ! G.1) logging [on] logging 1 ! G.2) no logging logging 0 Using a .TEXT argument, we could easily allow a space separator for category lists, as in "l1p mm cc". (nice would be to invent a VTY argument type that allows any number of the same kind, so far we have weird constructs like seen in bsc_vty.c, cfg_bts_codec0_cmd thru cfg_bts_codec4_cmd each with a different nr of same argument) A test program should define logging categories and a context filter, issue each of above commands and then log once all across each level and category to see what logging we still see. As a final step, we should figure out how the current logging commands match these features, and make them still do the things they always did as best as makes sense. What do you guys think? About bikesheds: if we can't agree, we could put out a patch contest, where everyone that cares implements a proposal, and we vote in the end. I hope we don't waste time by deadlocking in the end ;) ~N

On Wed, Jul 25, 2018 at 05:31:59PM +0200, Pau Espin Pedrol wrote: But currently no way to get back to it, IIUC. Not important anyway. We actually supported temporary logging increase, even though I never grokked how it was supposed to work, hence I broke it by removing the logging level all everything. This is an attempt to bring it back in an obvious and finely tunable way. It is not really complex, is it!? A problem there is that it is changing the meaning of a logging command that has been there for a decade. Because I fail to understand why exactly it isn't doing what I want and why it was implemented this way, and because I accept that other users might rely on exactly the current behavior to do what they want, I would prefer keeping it unchanged and to add new command names that make proper sense to us (to not repeat my past mistake). no, can't do whatever config setup: This does not allow for the 'logging level all everything' use case: - I have setup a detailed logging landscape, i.e. not one default level for all categories, but a fine grained selection: some on error, some on notice, some on info, ... - Now for a minute I want to quickly see all debug logging. logging increase debug all - Later I want to go back to the detailed landscape. logging reset all I'd like to provide a way to define a fine grained selection of logging levels as the default, more than just one level across the board. IIUC that can't be done with the current 'default' and 'unset'? Besides it changing an old vty command and it still missing an important use case, I totally agree that your patch is a usability improvement :) ~N

On Thu, Jul 26, 2018 at 11:04:25AM +0200, Pau Espin Pedrol wrote: The information is always there, the default categories log_info_cat struct are const. Remeber, each log target has its own category-level settings, so obviously the hardcoded defaults remain available. It's a simple thing. And still not really important :) And same as my last comment, you are changing a decade old vty command. Last time I did this it was the wrong choice. Introducing a common default level actually also closes the door on fine-grained default levels. I mean the VTY UI you are introducing would have to be deprecated to introduce fine-grained defaults. It is actually not practical to have all categories on the same level, unless it is, say, ERROR. So the patch does what you want and is better than current 'logging level all xxx' indeed, but changes old UI and has limits. As much as I'd like to let it go ahead, I'm still -1, sorry. true. Any opinions on this one, Keith maybe? I had the same idea once, but it's not practical. Copy pasting and stuff is not something we want to require the user to do. ~N

Before this echoing of the same arguments over and over continues between you and me, Pau, let's stop this here and let me do my homework in the logging code. I'll try to clarify what I mean, but I see I need to prove my points. Introducing new commands that work properly is IMO the only viable solution to fix old broken UI. IIUC your patch adds new commands? So I'd like to clarify how the implementation affects what old commands did, and whether it permits implementing per-category defaults without problems. Still not clear, is it? I want to set various categories to various log levels to taste, then go to say DEBUG on some/all of them, then later go back to exactly the levels selected before. It's like your default log level, just not one across all categories, but one level per category. Your suggestion: level ^ | - Normal logging landscape== | ===================== Temporary levels-- | -- - | -- ---- - |________________________> categories What I'd like to support: level ^ | - ====== = Normal logging landscape== | == == == -- Temporary levels-- | -- = == - = | = -- ---- -=== |________________________> categories This includes the possibility of having all of them at the same level, so it seems we are just very verbosely arguing about the implementation and UI details. Please let go of the hardcoded defaults, they aren't important, I did say so a number of times now. Do I read a dismissive tone here? :P Everything is always inherently per logtarget! exactly If you introduce one common default level for all categories, I fear that it conflicts with the use case to have separate defaults for each category. I still need to prove that. Don't the code changes modify how 'logging level all foo' behaves? I have to do my homework to strengthen my argument here. If monitoring e.g. BTS or PCU on a slow ARM, opening multiple log targets might impact stability. But then you lose the specific logging selection that you want to keep. ~N

On 26/07/2018 16:20, Neels Hofmeyr wrote: I have now got used to using a number of pre-prepared expect scripts to open VTYs for various logging scenarios. Rather than changing the logging in an open VTY, I would probably open another and change it there, or copy and modify the script to setup that logging config. I haven't got around to it yet, but I am planning on making this just one script that accepts params like "msc" or "mgw" as well as logging configs concepts as I can never remember the ports now with all the programs, although I have this: sudo netstat -tapn | egrep 'LISTEN.*osmo' | awk '{print $4, $7}' | awk -F "[:/ ]" '{print $2,"- " $4}' | grep 42 | sort -n I did not particularly have a problem with "logging level all everything" and "logging level all debug" and yes, as Neels mentioned have been patching libosmocore to revert to allowing "logging level all everything" - Despite some strange resistance to belief from some parties, this actually worked (works) as a temporary ON/OFF switch to set all categorys to debug, then revert to individual per category levels. I've explained that before, here on the list with Max, I think. The semantics off "logging level all everything" was no more difficult that anything else in GSM's world of acronyms etc. It was simply a case of learning what it does. I think it might be nice to have a default VTY logging storable in the config as we do for stderr. Otherwise, I'm kind of OK with using expect to setup what I want. K.

On Fri, Jul 27, 2018 at 04:59:46PM +0200, Neels Hofmeyr wrote: I guess Keith referst to the good old expect program, used by many of us for decades for automating shell like interfaces. "apt-get install expect && man 1 expect" ? -- - Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)

On Wed, Jul 25, 2018 at 11:16:46AM +0200, Pau Espin Pedrol wrote: I find both 'logging level unset <level>' and 'logging level default <level>' confusing. Did you mean to say '<category>'? IMHO "level unset" doesn't spark a meaningful image. Sounds like "allow any level", or next best meaning is "back to hardcoded defaults". ~N

[...] Ok, I understand now. So our main difference is that in your patch we have one single default log level, while I aim to allow separate default log levels for each category (and on each target). Let's discuss in the other thread leaf and end this one. ~N

Sure, everything I say and plan is on the premise of completely independent log targets, each log target has its own configuration for every single aspect (sharing only the hardcoded default category levels from the .c files). ~N

On 30/07/18 20:43, Harald Welte wrote: I think that feature is useful and it's there already anyway, it just doesn't work correctly in current merged code base. As explained in the thread, semantically it doesn't make much sense and that's why I was explaining I was modifying my patchset to remove that, and use "logging level default <loglevel>" instead, which better describes the action (setting the default log level to <loglevel>, meaning unset categories/subystems will use that log level). These modifications are already presnet in th new version of the patchset I submited 1-2 weeks ago in gerrit. -- - Pau Espin Pedrol &lt;pespin(a)sysmocom.de&gt; http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte

Hi Harald, Nice to see you are back :) I didn't expect this thread would grow so much. Probably, because we started to discuss the new features... But let's focus on the initial thread topic. I mean the global log level of a logging target, please see: https://git.osmocom.org/libosmocore/tree/include/osmocom/core/logging.h#n243 At the moment, it can be set with 'logging level all LEVEL', but anyway it doesn't work as expected, please see: https://osmocom.org/issues/3409 and this change by Pau which still mixes new features with actual bugfixing (src/logging.c#479): https://gerrit.osmocom.org/10116/ For some reason, I do associate libosmocore's logging subsystem with iptables. Please don't think I am crazy, there are really some similarities: - Instead of filtering packets, we are filtering the logging messages; - Logging target (e.g. stderr, or a file) is like a chain; - The global log level of a logging target (I usually refer this as 'the default logging level') is like the default policy of a chain; - If a logging category (e.g. 'meas' or 'l1c') has a custom logging level (e.g. 'debug'), one may interpret this as a rule in a chain. If there is no rule for a given category the default logging level can be used (like default policy). Meanwhile, we do have this line in some sample configuration files, and if there is no this line, we always have the following result of calling 'show running-config' or 'write file': logging level all everything This is why we (at least me and Pau) would like to introduce an idea of a bit more self-explaining command: logging level default debug Well, I think it makes sense to have some global / default logging level (e.g. 'error'), and a set of per-subsystem levels. For example, I would like to debug some particular subsystem / category (e.g. DMNCC), and silence all other: logging level default error logging level mncc debug Another important feature (I think) is a possibility to unset either a particular subsystem / category, or all subsystems / categories, which would fall-back to value of the default logging level for the current target. For example, I am running a program with the following logging configuration: logging level default notice logging level mncc debug logging level pag error logging level l1c debug logging level l1d info ... and now I would like to debug DPAG at run-time, so I could unset all custom values from VTY, and set DPAG to 'debug': (VTY)# logging level default error (VTY)# logging level all unset (VTY)# logging level pag debug Result: logging level default error logging level pag debug Finally, let's conclude my ideas in short all in one place: - Introduce 'unset' keyword; - Introduce self-explaining 'logging level default ...' which is aimed to set the (default / global) log level of a given logging target (e.g. 'stderr'); - Modify the behaviour of 'logging level all ...' to affect all logging categories (e.g. 'logging level all unset' would unset all custom user-defined levels, 'logging level all error' would sett all to 'error'). P.S. Sorry for 'the default', I hope this explaination is better. With best regards, Vadim Yanitskiy.

With a bit of time passed, I am revisiting this. It seems that my plan to have a per-subsystem configurable default log level as well as a per-subsystem configurable elevated/silenced log level is pretty much off the table. So then, for me that means it opens the door to accept Pau's patch. There was other criticism though: For some programs (osmo-hlr, osmo-msc, osmo-bsc) it does actually make perfect sense to set everything to DEBUG in a lab environment. That's why the brokenness of 'logging level all debug' is so annoying. You think you've set each and every log level to debug, but for some obscure reason still some logging is missing. Similarly, if I want to silence all logging except ERRORs, 'logging level all error' is a very useful command -- if it works the way that it sounds. Above annoyance is recurring and hindering everything else by constant uncertainty of what is being logged and/or the need to consider each individual logging category all the time. The "default" would be a single global setting for all categories. I sense a conflict between the "default" level and the struct log_info compiled-in defaults. Would the "default" level override the compiled-in defaults? Would the "default" have no effect until the vty command is issued? Or would the "default" have no effect until we call 'unset-all' to "unset" the initial levels copied from struct log_info levels? Now that I've accepted that we will not have a per-category adjustable default level, I think we might as well just stay with the compiled-in log_info defaults, and not introduce another default level. I want to be able to change all categories' current levels to "DEBUG" or "ERROR" with a single command, but I don't really have a need for a common default to fall back to. (Instead I would again set 'all' to that level.) So I've swerved 180° and now think that even the proposed one common default level across all categories is more complexity than we need. Can we fix 'logging level all foo' separately, without introducing more defaults first? We can then still consider the default,unset feature on its own. ~N