Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-29

Standardize logging and remove pex_logging

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None
    • Location:
      This issue

      Description

      The Data Access team has concerns about how we're doing logging. They would like to use Python's built-in logging library for most standalone, pure-Python utilities. In addition, we're currently using pex_logging within C++ and Python code throughout much of the LSST Stack. This custom code should be replaced. lsst::log has been developed as a thin layer on top of log4cxx that will be more flexible and maintainable as it is based on a standard external package. We need to transition the Stack to this interface.

      Among the Data Access team's concerns that point in favor of using Python logging:

      • Desire to minimize dependencies on LSST Stack and log4cxx
      • Simplicity of usage and default configuration
      • Use with third-party packages (that typically use Python logging)
      • Familiarity

      I propose to decree the following policy, although I'm not going to do much of the actual coding work:

      • C++ code throughout the Stack and Qserv uses lsst::log. (If anything is missing or poorly performing in that package, it should be fixed.)
      • All Python code developed by LSST DM uses Python logging.
      • Python apps (like CmdLineTask) that use packages that contain C++ code explicitly direct Python logging to use lsst.log in code (not via a configuration file). (I'm still worried that this will require configuration of individual logging components in multiple places depending on whether they're C++ or Python.)
      • A standard default Python logging message format and a standard default lsst::log message format that are compatible with each other will be defined in a separate RFC. This includes things like always logging timestamps in UTC. (Note for pedants: I think these would be TAI if and only if we take the step of shifting all production machines' system times to TAI. Even then, I would expect that non-production machines like developer's laptops will never produce TAI.)
      • All current code must be ported from pex_logging to the above by the end of Winter 2016 (Summer 2015?).
      • The Python logging configuration file that is currently installed and used by default by Qserv will be removed.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            John Swinbank Thanks. I had an old checkout of ci_hsc.

            daf_ingest is covered by DM-8270.

            ctrl_provenance has noticket.

            Show
            tjenness Tim Jenness added a comment - John Swinbank Thanks. I had an old checkout of ci_hsc. daf_ingest is covered by DM-8270 . ctrl_provenance has noticket.
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            Is someone working on ctrl_provenance? There were plans on replace it a very long time ago. It even got to the point where orchestration stubbed that out until the replacement arrived, because it was supposed to happen "soon".

            Show
            spietrowicz Steve Pietrowicz added a comment - Is someone working on ctrl_provenance? There were plans on replace it a very long time ago. It even got to the point where orchestration stubbed that out until the replacement arrived, because it was supposed to happen "soon".
            Hide
            tjenness Tim Jenness added a comment - - edited

            I have created a ticket for ctrl_provenance and attached it to this RFC (DM-8992). I'm not sure whether porting to log actually matters though. Kian-Tat Lim can you comment on the status of this RFC and whether DM-8992 matters?

            Show
            tjenness Tim Jenness added a comment - - edited I have created a ticket for ctrl_provenance and attached it to this RFC ( DM-8992 ). I'm not sure whether porting to log actually matters though. Kian-Tat Lim can you comment on the status of this RFC and whether DM-8992 matters?
            Hide
            ktl Kian-Tat Lim added a comment -

            Given progress on RFC-203 and the resulting elimination of pex_logging from much of the stack, I propose that this RFC be modified to only require the replacement of pex_logging by lsst::log, lsst.log, or Python logging. Once this is complete, pex_logging will be made a legacy package.

            Show
            ktl Kian-Tat Lim added a comment - Given progress on RFC-203 and the resulting elimination of pex_logging from much of the stack, I propose that this RFC be modified to only require the replacement of pex_logging by lsst::log , lsst.log , or Python logging . Once this is complete, pex_logging will be made a legacy package.
            Hide
            tjenness Tim Jenness added a comment -

            Adopted with caveat that lsst.log is now formally allowed in Python code.

            Show
            tjenness Tim Jenness added a comment - Adopted with caveat that lsst.log is now formally allowed in Python code.

              People

              • Assignee:
                ktl Kian-Tat Lim
                Reporter:
                ktl Kian-Tat Lim
                Watchers:
                Andy Salnikov, Gregory Dubois-Felsmann, John Swinbank, Kian-Tat Lim, Robert Lupton, Russell Owen, Steve Pietrowicz, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End: