Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15361

Make lsst.log outputs appear in Jupyter notebook cells

    XMLWordPrintable

    Details

    • Type: Story
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: log, log4cxx, Notebooks
    • Labels:
      None
    • Team:
      Data Access and Database

      Description

      Messages sent to lsst.log are not visible from Jupyter, because Jupyter only looks at Python's stderr and stdout, which are not the same as system stderr and stdout.

      Following some discussion on Slack, the best approach appears to be writing an lsst.log and/or log4cxx handler that writes to either Python's stderr or stdout or some Jupyter cell output hook; the latter in particular could enable things like using rich-text output to highlight messages at different levels.

        Attachments

          Activity

          Hide
          swinbank John Swinbank added a comment -

          Fritz Mueller, Frossie Economou — I think this is likely a SQuaRE issue rather than a DAX one, but since the team is set to DAX maybe you can check if you agree with me?

          For what it's worth, this has been identified as a priority by Pipelines folks using the Notebook Aspect. I know everybody's snowed under, but please bear that in mind when scheduling.

          Show
          swinbank John Swinbank added a comment - Fritz Mueller , Frossie Economou — I think this is likely a SQuaRE issue rather than a DAX one, but since the team is set to DAX maybe you can check if you agree with me? For what it's worth, this has been identified as a priority by Pipelines folks using the Notebook Aspect. I know everybody's snowed under, but please bear that in mind when scheduling.
          Hide
          ktl Kian-Tat Lim added a comment -

          To start with, copying a StackExchange answer, I've used a thread and a pipe to take logging output (os file descriptor 1) and feed it into Python's sys.stderr, which Jupyter in turn places in the cell. No fancy level-dependent coloring, but it appears to be basically functional.

          Show
          ktl Kian-Tat Lim added a comment - To start with, copying a StackExchange answer, I've used a thread and a pipe to take logging output (os file descriptor 1) and feed it into Python's sys.stderr , which Jupyter in turn places in the cell. No fancy level-dependent coloring, but it appears to be basically functional.
          Hide
          Parejkoj John Parejko added a comment -

          Pinging this again, as Chris Morrison and I want to be able to see Butler warnings and errors in jupyter. Any progress?

          Show
          Parejkoj John Parejko added a comment - Pinging this again, as Chris Morrison and I want to be able to see Butler warnings and errors in jupyter. Any progress?
          Hide
          salnikov Andy Salnikov added a comment - - edited

          Maybe it is easier to configure lsst.log to append to the same file descriptor as Python sys.stderr?

          Can you try this piece of code in Jupiter:

          import lsst.log 
          import sys
           
          _LOG_PROP = """
          log4j.rootLogger = DEBUG, JUPYTER
          log4j.appender.JUPYTER=org.apache.log4j.FileAppender
          log4j.appender.JUPYTER.File=/dev/fd/{}
          log4j.appender.JUPYTER.layout=PatternLayout
          log4j.appender.JUPYTER.layout.ConversionPattern=%c %p: %m%n
          """
          lsst.log.configure_prop(_LOG_PROP.format(sys.stderr.fileno()))
          

          and see if there is any output from lsst.log in a notebook? (change DEBUG to INFO if it is too chatty)

          Show
          salnikov Andy Salnikov added a comment - - edited Maybe it is easier to configure lsst.log to append to the same file descriptor as Python sys.stderr? Can you try this piece of code in Jupiter: import lsst.log import sys   _LOG_PROP = """ log4j.rootLogger = DEBUG, JUPYTER log4j.appender.JUPYTER=org.apache.log4j.FileAppender log4j.appender.JUPYTER.File=/dev/fd/{} log4j.appender.JUPYTER.layout=PatternLayout log4j.appender.JUPYTER.layout.ConversionPattern=%c %p: %m%n """ lsst.log.configure_prop(_LOG_PROP.format(sys.stderr.fileno())) and see if there is any output from lsst.log in a notebook? (change DEBUG to INFO if it is too chatty)
          Hide
          Parejkoj John Parejko added a comment -

          Chris Morrison: can you please try the above in one of your astrometry notebooks and see if you get some log output?

          Show
          Parejkoj John Parejko added a comment - Chris Morrison : can you please try the above in one of your astrometry notebooks and see if you get some log output?
          Hide
          ktl Kian-Tat Lim added a comment - - edited

          I have a couple of attempts at this that seemed to work but that I have not yet been able to test extensively. If someone could pick up one of these solutions (including Andy's above) and incorporate it into a package, that would be great.

          https://gist.github.com/ktlim/4e9deae888b150bb6e17efb32c1e46c3 logs to cells

          https://gist.github.com/ktlim/8b30bfbe8645583d3427718f96054974 logs to a Jupyter widget

          Show
          ktl Kian-Tat Lim added a comment - - edited I have a couple of attempts at this that seemed to work but that I have not yet been able to test extensively. If someone could pick up one of these solutions (including Andy's above) and incorporate it into a package, that would be great. https://gist.github.com/ktlim/4e9deae888b150bb6e17efb32c1e46c3 logs to cells https://gist.github.com/ktlim/8b30bfbe8645583d3427718f96054974 logs to a Jupyter widget

            People

            Assignee:
            ktl Kian-Tat Lim
            Reporter:
            jbosch Jim Bosch
            Watchers:
            Andy Salnikov, Chris Walter, Colin Slater, Jim Bosch, John Parejko, John Swinbank, Kian-Tat Lim, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Created:
              Updated: