# Make lsst.log outputs appear in Jupyter notebook cells

XMLWordPrintable

#### Details

• Type: Story
• Status: In Progress
• Resolution: Unresolved
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
Data Access and Database

#### Description

Messages sent to lsst.log are not visible from Jupyter, because Jupyter only looks at Python's stderr and stdout, which are not the same as system stderr and stdout.

Following some discussion on Slack, the best approach appears to be writing an lsst.log and/or log4cxx handler that writes to either Python's stderr or stdout or some Jupyter cell output hook; the latter in particular could enable things like using rich-text output to highlight messages at different levels.

#### Activity

Hide
John Swinbank added a comment -

Fritz Mueller, Frossie Economou — I think this is likely a SQuaRE issue rather than a DAX one, but since the team is set to DAX maybe you can check if you agree with me?

For what it's worth, this has been identified as a priority by Pipelines folks using the Notebook Aspect. I know everybody's snowed under, but please bear that in mind when scheduling.

Show
John Swinbank added a comment - Fritz Mueller , Frossie Economou — I think this is likely a SQuaRE issue rather than a DAX one, but since the team is set to DAX maybe you can check if you agree with me? For what it's worth, this has been identified as a priority by Pipelines folks using the Notebook Aspect. I know everybody's snowed under, but please bear that in mind when scheduling.
Hide
Kian-Tat Lim added a comment -

To start with, copying a StackExchange answer, I've used a thread and a pipe to take logging output (os file descriptor 1) and feed it into Python's sys.stderr, which Jupyter in turn places in the cell. No fancy level-dependent coloring, but it appears to be basically functional.

Show
Kian-Tat Lim added a comment - To start with, copying a StackExchange answer, I've used a thread and a pipe to take logging output (os file descriptor 1) and feed it into Python's sys.stderr , which Jupyter in turn places in the cell. No fancy level-dependent coloring, but it appears to be basically functional.
Hide
John Parejko added a comment -

Pinging this again, as Chris Morrison and I want to be able to see Butler warnings and errors in jupyter. Any progress?

Show
John Parejko added a comment - Pinging this again, as Chris Morrison and I want to be able to see Butler warnings and errors in jupyter. Any progress?
Hide
Andy Salnikov added a comment - - edited

Maybe it is easier to configure lsst.log to append to the same file descriptor as Python sys.stderr?

Can you try this piece of code in Jupiter:

 import lsst.log  import sys   _LOG_PROP = """ log4j.rootLogger = DEBUG, JUPYTER log4j.appender.JUPYTER=org.apache.log4j.FileAppender log4j.appender.JUPYTER.File=/dev/fd/{} log4j.appender.JUPYTER.layout=PatternLayout log4j.appender.JUPYTER.layout.ConversionPattern=%c %p: %m%n """ lsst.log.configure_prop(_LOG_PROP.format(sys.stderr.fileno())) 

and see if there is any output from lsst.log in a notebook? (change DEBUG to INFO if it is too chatty)

Show
Andy Salnikov added a comment - - edited Maybe it is easier to configure lsst.log to append to the same file descriptor as Python sys.stderr? Can you try this piece of code in Jupiter: import lsst.log import sys   _LOG_PROP = """ log4j.rootLogger = DEBUG, JUPYTER log4j.appender.JUPYTER=org.apache.log4j.FileAppender log4j.appender.JUPYTER.File=/dev/fd/{} log4j.appender.JUPYTER.layout=PatternLayout log4j.appender.JUPYTER.layout.ConversionPattern=%c %p: %m%n """ lsst.log.configure_prop(_LOG_PROP.format(sys.stderr.fileno())) and see if there is any output from lsst.log in a notebook? (change DEBUG to INFO if it is too chatty)
Hide
John Parejko added a comment -

Chris Morrison: can you please try the above in one of your astrometry notebooks and see if you get some log output?

Show
John Parejko added a comment - Chris Morrison : can you please try the above in one of your astrometry notebooks and see if you get some log output?
Hide
Kian-Tat Lim added a comment - - edited

I have a couple of attempts at this that seemed to work but that I have not yet been able to test extensively. If someone could pick up one of these solutions (including Andy's above) and incorporate it into a package, that would be great.

https://gist.github.com/ktlim/8b30bfbe8645583d3427718f96054974 logs to a Jupyter widget

Show
Kian-Tat Lim added a comment - - edited I have a couple of attempts at this that seemed to work but that I have not yet been able to test extensively. If someone could pick up one of these solutions (including Andy's above) and incorporate it into a package, that would be great. https://gist.github.com/ktlim/4e9deae888b150bb6e17efb32c1e46c3 logs to cells https://gist.github.com/ktlim/8b30bfbe8645583d3427718f96054974 logs to a Jupyter widget

#### People

Assignee:
Kian-Tat Lim
Reporter:
Jim Bosch
Watchers:
Andy Salnikov, Chris Walter, Colin Slater, Jim Bosch, John Parejko, John Swinbank, Kian-Tat Lim, Tim Jenness