lsst.log should default to stdout

XMLWordPrintable

Details

• Type: RFC
• Status: Implemented
• Resolution: Done
• Component/s:
• Labels:

Description

Currently, lsst.log sends all output to stderr. This is surprising since most log messages are not errors: generally only actual errors (e.g. python exceptions, stack traces) should be sent to stderr.

lsst.log should send all output to stdout. This helps separate logs and errors when piping output; for example, in python:

 subprocess.check_call(['jointcal.py', *args], stdout=open('jointcal.log', 'w'), stderr=open('log.err','w)) 

I might also suggest that errors at warn, error, and fatal levels also get sent to stderr, so they're more easily identified, though that would result in them being duplicated when logging to a terminal. I'm not sure we want them to only go to stderr, as that pulls them out of context, but they can get drowned in all the other messages.

Activity

Hide
Paul Price added a comment -

I have suggested that "we should only log to stdout if buffering is explicitly disabled." If you're telling me that the OS' buffering of stdout is always explicitly disabled by our logging system, then I'm satisfied.

Show
Paul Price added a comment - I have suggested that "we should only log to stdout if buffering is explicitly disabled." If you're telling me that the OS' buffering of stdout is always explicitly disabled by our logging system, then I'm satisfied.
Hide
John Parejko added a comment -

I've revised the proposal and scratched out the "shared stream" idea: thanks to Krzysztof Findeisen for making it clear why that couldn't work. I agree that we should ensure that the logs explicitly unbuffered: we could probably have a test for that (as Joshua Hoblitt says above, there's no actual standard for which of these are buffered or not).

Any other objections?

Show
John Parejko added a comment - I've revised the proposal and scratched out the "shared stream" idea: thanks to Krzysztof Findeisen for making it clear why that couldn't work. I agree that we should ensure that the logs explicitly unbuffered: we could probably have a test for that (as Joshua Hoblitt says above, there's no actual standard for which of these are buffered or not). Any other objections?
Hide
Andy Salnikov added a comment -

Sorry for being late for discussion, I have just returned from vacation.

One clarification — lsst.log is by default configured to send output to ConsoleAppender which by default sends its output to stdout. It is CmdLineTask which configures lsst.log to send appender output to stderr (here and here).

Technically it is not impossible to send messages to different streams based on severity, one could for example write specialized appender which sends its output to one or more streams depending on the message context. In specialized appender it is also possible to detect when both stdout and stderr refer to the same file to avoid duplicated messages. Buffering is mostly non-issue, by default log4cxx ConsoleAppender is configured to do flush() after each message, so even if app crashes right after logging error message nothing should be lost. (This would also take care of the stream synchronization if messages are sent to different streams).

Having said this I think easiest and least confusing solution is to just change CmdLineTask configuration to log to stdout (even though it means error messages are also sent there).

Show
Andy Salnikov added a comment - Sorry for being late for discussion, I have just returned from vacation. One clarification — lsst.log is by default configured to send output to ConsoleAppender which by default sends its output to stdout . It is CmdLineTask which configures lsst.log to send appender output to stderr ( here and here ). Technically it is not impossible to send messages to different streams based on severity, one could for example write specialized appender which sends its output to one or more streams depending on the message context. In specialized appender it is also possible to detect when both stdout and stderr refer to the same file to avoid duplicated messages. Buffering is mostly non-issue, by default log4cxx ConsoleAppender is configured to do flush() after each message, so even if app crashes right after logging error message nothing should be lost. (This would also take care of the stream synchronization if messages are sent to different streams). Having said this I think easiest and least confusing solution is to just change CmdLineTask configuration to log to stdout (even though it means error messages are also sent there).
Hide
Kian-Tat Lim added a comment -

John Parejko Can we adopt this? You can assign Andy Salnikov or me the implementation ticket; as he pointed out there are only two lines that need to be changed, although I have some ideas about adding additional options to allow more flexibility.

Show
Kian-Tat Lim added a comment - John Parejko Can we adopt this? You can assign Andy Salnikov or me the implementation ticket; as he pointed out there are only two lines that need to be changed, although I have some ideas about adding additional options to allow more flexibility.
Hide
John Parejko added a comment -

Implementation ticket: DM-12359

Show
John Parejko added a comment - Adopted as written; logs should go to stdout instead of stderr. Implementation ticket: DM-12359

People

• Assignee:
John Parejko
Reporter:
John Parejko
Watchers:
Andy Salnikov, John Parejko, Joshua Hoblitt, Kian-Tat Lim, Krzysztof Findeisen, Paul Price, Robert Lupton