Hi all, for logging changes, I would welcome a comprehensive approach and maybe add an all new set of logging commands, to not confuse with the current logging behavior; keep and deprecate the old ones for users that were fine with those. (I was the one responsible for the death of 'logging level all everything', and I've stumped at least one user who now keeps patching it back in manually; let's not repeat that.) Maybe we can come up with more meaningful wording that more accurately indicate what is happening? But if the current command set suffices without confusion, we can also keep that, and maybe resurrect the 'logging level all everything'?? Here are the handles / use cases that come up -- I've embedded implementation choices and command names in this proposal, but it's intended as a blueprint to incorporate everyone else's bikesheds: A) The hardcoded logging landscape on program startup without config. (Maybe this isn't all that important, including it nevertheless.) 1) a command to reset all to hardcoded defaults? B) A "normal" logging behavior configured by the user. We need: 1) a command to set all categories at once 2) a command to tweak individual categories C) Temporarily reducing the logging on all categories: silence the log. Even though a category is "normally" set to INFO, after this command it can be forced down to only log e.g. NOTICE. But if a category is already on ERROR, leave it down there and do not lift it up to NOTICE. 1) a command to temporarily reduce all categories. 2) a command to temporarily reduce individual categories. D) Temporarily increasing the logging on all categories: show everything. Even though a category is "normally" set to ERROR, after this command it can be forced up to e.g. INFO. But if a category is already up at DEBUG, don't reduce it to NOTICE. 1) a command to temporarily increase all categories. 2) a command to temporarily increase individual categories. 3) we need to decide whether temporarily increasing is stronger/weaker than temporarily reducing. Depending on the implementation, it could also be "last one wins" (which IMHO would be best). T) Maybe we also want to unconditionally set temporary levels, whether they increase or reduce. E) Remove temporary logging to go back to the "normal" logging landscape. 1) a command to put everything back to "normal", 2) a command to put individual categories back to "normal", 3) a separate command to only remove temporary reduction, i.e lift the temporary changes only if they cause less logging than normal, 4) a separate command to only remove temporary verbosity, i.e. only lift the temporary changes if they cause more logging than normal. F) Set a specific log context, and silence all messages (regardless of their logging category setting) unless they match the context filter: logging filter imsi 123456789123456 Though this feature has probably never been properly implemented through all code paths, it is potentially very powerful for live debugging. Especially with the new FSMs around everywhere, there is a good chance that we can fairly easily improve on the current 'logging filter foo' experience. During congress, I really yearned for this to work well. 0) In the current implementation, on each individual log message, this decides whether the context matches, and if not, the message is discarded. 1) Instead of discarding, we may want a level to reduce to. 2) Or maybe we want to leave the logging landscape unchanged, but lift the level for all logs that match the set context? 3) disable individual filters 4) disable all filters. This is currently 'logging filter all 1' but that is confusing, we could use a meaningful synonym, and separate from filters. G) toggle all logging in the telnet vty 1) silence logging (currently 'logging filter all 0', confusing) 2) continue logging (currently 'logging filter all 1', confusing) A bit of a meaningless image of different logging layers: level ^ Hardcoded defaults.. | ... - ====== =.... Normal logging landscape== | == .== ... ==... -- Temporary levels-- | -- =.... ...== - = | = -- ---- -=== |________________________> categories Same image with actual current logging overlayed: level ^ | ... # =###== =.... | == .== ... ==... ## Resulting levels ## | ## #.... ...## # = | # ## #### #=#= |________________________> categories - the hardcoded defaults are immutable. - the normal landscape can be freely chosen. - Some categories may have temporary tweaks. Where there are temporary tweaks, they take precedence as-is. The vty command ("make stronger" or "make weaker") decided whether to put a temporary level in place or not. Can you guys explain the current patch proposals in terms of above features? (like, use the capital letters A-G for reference) Feel free to change this vty transcript proposal i've tossed up as I went: ! A.1) hardcoded defaults logging hard-reset rsl,chan,msc logging hard-reset ! B) set normal categories ! B.1) all logging set notice all ! B.2) individual logging set notice rsl,chan,msc ! C) Temporarily reducing ! C.1) all logging reduce notice all ! C.2) individual logging reduce error l1p,nm,cc ! D) Temporarily increasing ! D.1) all logging increase info all ! D.2) individual logging increase debug mm,rr ! D.3) last one wins logging reduce error l1p,mm,rr logging increase debug mm,rr logging reduce notice rr ! result: l1p=error mm=debug rr=notice ! T) set temporary level unconditionally logging set-temp error all logging set-temp error mm,rr ! logging (recrease|induce) ;) ! E.3) remove temporary verbosity no logging reduce nm,cc no logging reduce all ! E.4) remove temporary silencing no logging increase nm,cc no logging increase all ! E.2) snap back to normal levels logging reset rsl,nm,rr ! E.1) ...for all categories logging reset all ! F.0) context filter logging filter imsi 123456789123456 ! F.1) choose context filter behavior logging filter-reduce error logging filter-discard ! F.2) logging filter-increase debug ! F.3) disable filter no logging filter imsi ! F.4) no logging filter ! G.1) logging [on] logging 1 ! G.2) no logging logging 0 Using a .TEXT argument, we could easily allow a space separator for category lists, as in "l1p mm cc". (nice would be to invent a VTY argument type that allows any number of the same kind, so far we have weird constructs like seen in bsc_vty.c, cfg_bts_codec0_cmd thru cfg_bts_codec4_cmd each with a different nr of same argument) A test program should define logging categories and a context filter, issue each of above commands and then log once all across each level and category to see what logging we still see. As a final step, we should figure out how the current logging commands match these features, and make them still do the things they always did as best as makes sense. What do you guys think? About bikesheds: if we can't agree, we could put out a patch contest, where everyone that cares implements a proposal, and we vote in the end. I hope we don't waste time by deadlocking in the end ;) ~N

On 25/07/18 16:33, Neels Hofmeyr wrote: No, I meant what I wrote. So after my proposal (from my last email): * "logging level default <level>": Sets default log level (the loglevel for unset categories, aka the logtarget loglevel) to <level>. * "logging level all <level>": Sets each (all) category to use log level <level>, without modifying the default log level (aka logtarget loglevel). * "logging level (all|<category>) unset": Unsets given category log level, and as a result log messages for that category are handled by default log level (aka logtarget loglevel). * "logging level (all|<category>) <level>": Sets given category to given log level. As a result, messages for that category are handled by this log level and not the default log level. Regarding your last comment: unsetting doesn't mean going to initially per-category set defaults during startup of the application. It means to fallback to default log level (aka logtarget loglevel). Initial hardcoded values set per-category are basically the same as if you went through the VTY setting "log level <category> <level>" before parsing the cfg file. -- - Pau Espin Pedrol &lt;pespin(a)sysmocom.de&gt; http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte

On Wed, Jul 25, 2018 at 05:31:59PM +0200, Pau Espin Pedrol wrote: But currently no way to get back to it, IIUC. Not important anyway. We actually supported temporary logging increase, even though I never grokked how it was supposed to work, hence I broke it by removing the logging level all everything. This is an attempt to bring it back in an obvious and finely tunable way. It is not really complex, is it!? A problem there is that it is changing the meaning of a logging command that has been there for a decade. Because I fail to understand why exactly it isn't doing what I want and why it was implemented this way, and because I accept that other users might rely on exactly the current behavior to do what they want, I would prefer keeping it unchanged and to add new command names that make proper sense to us (to not repeat my past mistake). no, can't do whatever config setup: This does not allow for the 'logging level all everything' use case: - I have setup a detailed logging landscape, i.e. not one default level for all categories, but a fine grained selection: some on error, some on notice, some on info, ... - Now for a minute I want to quickly see all debug logging. logging increase debug all - Later I want to go back to the detailed landscape. logging reset all I'd like to provide a way to define a fine grained selection of logging levels as the default, more than just one level across the board. IIUC that can't be done with the current 'default' and 'unset'? Besides it changing an old vty command and it still missing an important use case, I totally agree that your patch is a usability improvement :) ~N

On 25/07/18 18:17, Neels Hofmeyr wrote: Just to make sure: We already have a default differentiate log level on each target. Not only 1 for the whole app. It's true that we don't have one per category. -- - Pau Espin Pedrol &lt;pespin(a)sysmocom.de&gt; http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte

On 25/07/18 18:12, Neels Hofmeyr wrote: No, we cannot, we would probably require a new API to store them as "harcoded categories", and some VTY command to set them. If somebody thinks it's needed and really useful, patches welcome. I think it's not that useful and I want to keep the logging system as simple as possible while still providing useful ways to fine-grain loglvels. Same as my last above comment. My intention with this patchset was to have the logging working again trying not to touch that much, and avoid spending much time on it. If at some point we supported the feature, it's not the case anymore nowadays. If somebody wants to re-add that feature and change the logging system, patches welcome. I think current behavior (for logging level all) is imho totally broken and the patchset I'm proposing changes behavior of old commands only minimally. Someone can always add these feature on top of present patchset if there's enough use for it. This feature is not there now, so I'm not removing it with presented patchset. I personally think the scenario you presented can be workarounded pretty easily in different ways, so I personally think there's no urgent need for it: 1- Open a new VTY, then call logging level all debug 2- Write down the commands you used to set the logging, then paste them after running "logging level default debug; logging level all unset" to see all logging for a while. -- - Pau Espin Pedrol &lt;pespin(a)sysmocom.de&gt; http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte

Hi, find answers in place. On 26/07/18 16:20, Neels Hofmeyr wrote: Ok, no need for new API, since it's already done using a specific API (log_init). But we'd need to change the implementation to account in that function that for each category the value being copied is the "hardcoded default" instead of a regular level being set. Ok so let's leave it broken and useless then? Deprecate it and write an entire new system? No, thanks. What do you mean with fine-grained default levels? Having a way to go back to hardcoded initial values? You can have that as long as you store the values in log_init in a new var default_loglevel for each category. If you still want more temporary levels (increase/decrease or whatever you call it) per logtarget or even per category, you can add them later and check them before the other levels in should_log_to_target(), then change them with new VTY commands. It's compatible with old UI. All programs using the API continue to work fine as well as old cfg files. It adds new parameter "logging level default <loglevel>" and "logging level <category> unset", that's all. So it fixes broken logging level all and adds extra feature to make it useful. You can even just re-open the session you have opened, no need to open a second one. -- - Pau Espin Pedrol &lt;pespin(a)sysmocom.de&gt; http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte

Hi Pau, Neels, and all, It seems, the reason of my misunderstanding was that confusing 'all' keyword. I have been interpretting it as 'default', but actually it is far from this meaning... Yep, finally I am ;) In order to have a clean commit history, I suggest the following algorithm (one high level point represents one single change): 1. Introduce 'unset' keyword first; 2. Introduce 'default' logging category ('logging level default'): 2.1. Modify the 'logging level' command (by adding 'default'); 2.2. Correct memory allocation in log_vty_command_string(); 2.3. Write *exactly 'default'* (not 'all') to the configuration; 3. Fix the behaviour of 'default' (i.e. target's) logging category: 3.1. In other words, correct the loging in should_log_to_target(); 4. Correct the behaviour of 'logging level all LEVEL'; 5. Testing coverage ;) 6. <New features> ... Regarding to changing the behaviour of 'logging level all LEVEL' command, I think we need to make this step. Let's have a look at OsmoHLR configuration, shipped as an example: log stderr ! ... ! Set *default* log level 'debug' for stderr logging level all debug logging level linp error What would we get now (before merging the patch set)? The 'debug' becomes default, but since the logging is broken, the hardcoded values will be used for all categories, excluding 'linp'. The 'all' remains confusing... If one saves the current configuration, the result would be: log stderr ! ... ! Set *default* log level 'debug' for stderr logging level all debug logging level linp error logging level foo <hardcoded> logging level bar <hardcoded> ! all other categories... What would we get after this patch set? The first command is now interpreted as it should be interpreted. So, we have no 'default' logging level (i.e. it becomes 'unset'), all categories, excluding 'linp', are set to 'debug'. If one saves the current configuration now, the result would be similar, i.e. it will also dump all possible categories, but instead of <hardcoded> values 'debug' would be used: log stderr ! ... ! Set *default* log level 'debug' for stderr logging level default debug logging level linp error logging level foo debug logging level bar debug ! all other categories... How to avoid this after merging the patch set? Update the configuration examples, replacing 'all' by 'default'. Either, doing 'logging level all unset' before saving to file. This way one would get the original configuration untouched, i.e. without dumping all possible categories... Then we can think about new features, and this is what do I think about some of them (described by Neels): I like the idea of Neels about resetting the logging levels. Additionally to the hardcoded logging settings, would be great to have a possibility to reset to the logging settings from the current configuration... I assume the following syntax for that: logging level (all|default|<category>) reset Regarding to the 'temporary' staff, I think one can interpret the current ('running') modified configuration as 'temporary'. You're free to change everything, and if you like the current logging configuration - you can write it to a file. Otherwise, your idea about resetting fits here just fine, so one would be able to restore the current configuration from file at runtime. I think it's useful feature. I didn't even know about this :) Another interesting idea is to introduce a RegExp based filter (if not yet), like we have in the talloc context introspection command. Examples: logging filter regexp (failed|unable|couldn't) logging filter regexp ^BTS[0-9]+ BTW: I also find 'logging filter all 1' confusing ;) With best regards, Vadim Yanitskiy.

On Fri, Jul 27, 2018 at 01:42:11PM +0200, Keith wrote: So you're saying you're no longer doing that and going with opening separate VTY telnets instead? There's an interesting feature request. Maybe easier would be a vty command to adopt in the current telnet the logging config from stderr (or any other log target). What's "using expect"? ~N

Hi Vadim, as I'm back from holidays, I'm catching up with mails, including this thread here. On Wed, Jul 25, 2018 at 12:10:08AM +0700, Vadim Yanitskiy wrote: but what is the "default level"? I assume as you're talking about interactive changes, you are talking about logging to a VTY. What is the default that you fall-back to? Unlike file/syslog targets, the VTY doesn't have any configurable defaults To be honest, I don't think I have any idea at all what "logging level all" does, I have never used it. I find it quite confusing that there are other things than setting the per-subsystem levels. Setting per-subsystem categories makes sense to me, everything else seems strange... I agree this is more intuitive from the meaning of the word "all", but I think that would be only ever used to set a global "sane" logging level like NOTICE or ERROR. You certainly don't want to set DEBUG on all categories, as otherwise you're going to be spammed and will drown in too much information. What is the "default logging level" ? The level compiled-into the struct log_info? How would you ever change the default? I thought that's the compiled-in part? Resetting the current logging masks to those default compiled-in ones by a single command seems like a useful feature to me. My practical work-around for this is to simply close + re-open the VTY session... -- - Harald Welte &lt;laforge(a)gnumonks.org&gt; http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)

Hi Neels, On Wed, Jul 25, 2018 at 04:22:09PM +0200, Neels Hofmeyr wrote: ACK to set all to one level, or to set all to one set of (configurable) defaults, like a template? ACK. Not sure for what one would use that. You could switch off logging completely and re-enable it later? Sounds incredibly complicated to me. ?!? WTF? Why all that complexity? Honestly, I'm stopping at that point. Just finishing to read that one e-mail, let alone the entire thread will cost me hours. To me, it boils down to a hypthetical luxury problem, unless there's somebody who is actually funding/contributing the amount of time required to designing, debating, implementing, reviewing and merging and overhaul of the logging sub-system. -- - Harald Welte &lt;laforge(a)gnumonks.org&gt; http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)

On 30/07/18 20:43, Harald Welte wrote: I think that feature is useful and it's there already anyway, it just doesn't work correctly in current merged code base. As explained in the thread, semantically it doesn't make much sense and that's why I was explaining I was modifying my patchset to remove that, and use "logging level default <loglevel>" instead, which better describes the action (setting the default log level to <loglevel>, meaning unset categories/subystems will use that log level). These modifications are already presnet in th new version of the patchset I submited 1-2 weeks ago in gerrit. -- - Pau Espin Pedrol &lt;pespin(a)sysmocom.de&gt; http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte

With a bit of time passed, I am revisiting this. It seems that my plan to have a per-subsystem configurable default log level as well as a per-subsystem configurable elevated/silenced log level is pretty much off the table. So then, for me that means it opens the door to accept Pau's patch. There was other criticism though: For some programs (osmo-hlr, osmo-msc, osmo-bsc) it does actually make perfect sense to set everything to DEBUG in a lab environment. That's why the brokenness of 'logging level all debug' is so annoying. You think you've set each and every log level to debug, but for some obscure reason still some logging is missing. Similarly, if I want to silence all logging except ERRORs, 'logging level all error' is a very useful command -- if it works the way that it sounds. Above annoyance is recurring and hindering everything else by constant uncertainty of what is being logged and/or the need to consider each individual logging category all the time. The "default" would be a single global setting for all categories. I sense a conflict between the "default" level and the struct log_info compiled-in defaults. Would the "default" level override the compiled-in defaults? Would the "default" have no effect until the vty command is issued? Or would the "default" have no effect until we call 'unset-all' to "unset" the initial levels copied from struct log_info levels? Now that I've accepted that we will not have a per-category adjustable default level, I think we might as well just stay with the compiled-in log_info defaults, and not introduce another default level. I want to be able to change all categories' current levels to "DEBUG" or "ERROR" with a single command, but I don't really have a need for a common default to fall back to. (Instead I would again set 'all' to that level.) So I've swerved 180° and now think that even the proposed one common default level across all categories is more complexity than we need. Can we fix 'logging level all foo' separately, without introducing more defaults first? We can then still consider the default,unset feature on its own. ~N