Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-402

lsst.log should default to stdout

    Details

      Description

      Currently, lsst.log sends all output to stderr. This is surprising since most log messages are not errors: generally only actual errors (e.g. python exceptions, stack traces) should be sent to stderr.

      lsst.log should send all output to stdout. This helps separate logs and errors when piping output; for example, in python:

      subprocess.check_call(['jointcal.py', *args], stdout=open('jointcal.log', 'w'), stderr=open('log.err','w))
      

      I might also suggest that errors at warn, error, and fatal levels also get sent to stderr, so they're more easily identified, though that would result in them being duplicated when logging to a terminal. I'm not sure we want them to only go to stderr, as that pulls them out of context, but they can get drowned in all the other messages.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            I have suggested that "we should only log to stdout if buffering is explicitly disabled." If you're telling me that the OS' buffering of stdout is always explicitly disabled by our logging system, then I'm satisfied.

            Show
            price Paul Price added a comment - I have suggested that "we should only log to stdout if buffering is explicitly disabled." If you're telling me that the OS' buffering of stdout is always explicitly disabled by our logging system, then I'm satisfied.
            Hide
            Parejkoj John Parejko added a comment -

            I've revised the proposal and scratched out the "shared stream" idea: thanks to Krzysztof Findeisen for making it clear why that couldn't work. I agree that we should ensure that the logs explicitly unbuffered: we could probably have a test for that (as Joshua Hoblitt says above, there's no actual standard for which of these are buffered or not).

            Any other objections?

            Show
            Parejkoj John Parejko added a comment - I've revised the proposal and scratched out the "shared stream" idea: thanks to Krzysztof Findeisen for making it clear why that couldn't work. I agree that we should ensure that the logs explicitly unbuffered: we could probably have a test for that (as Joshua Hoblitt says above, there's no actual standard for which of these are buffered or not). Any other objections?
            Hide
            salnikov Andy Salnikov added a comment -

            Sorry for being late for discussion, I have just returned from vacation.

            One clarification — lsst.log is by default configured to send output to ConsoleAppender which by default sends its output to stdout. It is CmdLineTask which configures lsst.log to send appender output to stderr (here and here).

            Technically it is not impossible to send messages to different streams based on severity, one could for example write specialized appender which sends its output to one or more streams depending on the message context. In specialized appender it is also possible to detect when both stdout and stderr refer to the same file to avoid duplicated messages. Buffering is mostly non-issue, by default log4cxx ConsoleAppender is configured to do flush() after each message, so even if app crashes right after logging error message nothing should be lost. (This would also take care of the stream synchronization if messages are sent to different streams).

            Having said this I think easiest and least confusing solution is to just change CmdLineTask configuration to log to stdout (even though it means error messages are also sent there).

            Show
            salnikov Andy Salnikov added a comment - Sorry for being late for discussion, I have just returned from vacation. One clarification — lsst.log is by default configured to send output to ConsoleAppender which by default sends its output to stdout . It is CmdLineTask which configures lsst.log to send appender output to stderr ( here and here ). Technically it is not impossible to send messages to different streams based on severity, one could for example write specialized appender which sends its output to one or more streams depending on the message context. In specialized appender it is also possible to detect when both stdout and stderr refer to the same file to avoid duplicated messages. Buffering is mostly non-issue, by default log4cxx ConsoleAppender is configured to do flush() after each message, so even if app crashes right after logging error message nothing should be lost. (This would also take care of the stream synchronization if messages are sent to different streams). Having said this I think easiest and least confusing solution is to just change CmdLineTask configuration to log to stdout (even though it means error messages are also sent there).
            Hide
            ktl Kian-Tat Lim added a comment -

            John Parejko Can we adopt this? You can assign Andy Salnikov or me the implementation ticket; as he pointed out there are only two lines that need to be changed, although I have some ideas about adding additional options to allow more flexibility.

            Show
            ktl Kian-Tat Lim added a comment - John Parejko Can we adopt this? You can assign Andy Salnikov or me the implementation ticket; as he pointed out there are only two lines that need to be changed, although I have some ideas about adding additional options to allow more flexibility.
            Hide
            Parejkoj John Parejko added a comment -

            Adopted as written; logs should go to stdout instead of stderr.

            Implementation ticket: DM-12359

            Show
            Parejkoj John Parejko added a comment - Adopted as written; logs should go to stdout instead of stderr. Implementation ticket: DM-12359

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Watchers:
                Andy Salnikov, John Parejko, Joshua Hoblitt, Kian-Tat Lim, Krzysztof Findeisen, Paul Price, Robert Lupton
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel