# Suggest logging migration in daf_persistence and daf_butlerUtils

## Details

• Type: Story
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Templates:
• Story Points:
2
• Team:
Process Middleware

## Activity

Hide
Hsin-Fang Chiang added a comment -

Jenkins passed building (including lsst_distrib, lsst_sims, lsst_ci), but failed at the stack demo , because the default logging output is stdout in lsst.log instead of stderr in pex.logging, the demo script bin/export-results redirects its std output to make the "detected-sources(_small).txt" file, and there are a few debug messages from daf.persistence and daf.butlerUtils when running that script. For running command line tasks and unit tests, log has been configured to go to stderr. The standalone script bin/export-results needs to configure log as well. This is done in the ticket branch of lsst_dm_stack_demo, but Jenkins doesn't get that ticket branch. I've tested it on my machine, to reproduce the error with master and pass the demo with the ticket branch.

Show
Hsin-Fang Chiang added a comment - Jenkins passed building (including lsst_distrib, lsst_sims, lsst_ci), but failed at the stack demo , because the default logging output is stdout in lsst.log instead of stderr in pex.logging , the demo script bin/export-results redirects its std output to make the "detected-sources(_small).txt" file, and there are a few debug messages from daf.persistence and daf.butlerUtils when running that script. For running command line tasks and unit tests, log has been configured to go to stderr. The standalone script bin/export-results needs to configure log as well. This is done in the ticket branch of lsst_dm_stack_demo , but Jenkins doesn't get that ticket branch. I've tested it on my machine, to reproduce the error with master and pass the demo with the ticket branch.
Hide
Kian-Tat Lim added a comment -

Looks OK. The BlockTimingLog replacement is not generating exactly the same message as before, but I don't think that's really a problem.

Show
Kian-Tat Lim added a comment - Looks OK. The BlockTimingLog replacement is not generating exactly the same message as before, but I don't think that's really a problem.
Hide
Hsin-Fang Chiang added a comment -

Now I see by the messages generated previously, you didn't mean the messages before this old daf_persistence commit but the pex.logging.BlockTimingLog messages. I corrected the messages to be exactly the same now (with format differences).

Use tests/butlerPickle.py as an example and set a low threshold/level, this is before the logging migration (pex_logging):

  daf.persistence.butler.read DEBUG: Starting read from None at PickleStorage(foo1.pickle)  daf.persistence.butler.read DEBUG: Ending read from None at PickleStorage(foo1.pickle) 

This is after the logging migration (lsst_log):

 DEBUG daf.persistence.butler: Starting read from None at PickleStorage(foo1.pickle) DEBUG daf.persistence.butler: Ending read from None at PickleStorage(foo1.pickle) 

The master daf_persistence doesn't really use BlockTimingLog, so I added one commit to get the messages printed with pex.logging, before the logging migration. The effective changes are the same.

I merged the lsst_dm_stack_demo patch and am running Jenkins.

Show
Hsin-Fang Chiang added a comment - Now I see by the messages generated previously, you didn't mean the messages before this old daf_persistence commit but the pex.logging.BlockTimingLog messages. I corrected the messages to be exactly the same now (with format differences). Use tests/butlerPickle.py as an example and set a low threshold/level, this is before the logging migration (pex_logging): daf.persistence.butler.read DEBUG: Starting read from None at PickleStorage(foo1.pickle) daf.persistence.butler.read DEBUG: Ending read from None at PickleStorage(foo1.pickle) This is after the logging migration (lsst_log): DEBUG daf.persistence.butler: Starting read from None at PickleStorage(foo1.pickle) DEBUG daf.persistence.butler: Ending read from None at PickleStorage(foo1.pickle) The master daf_persistence doesn't really use BlockTimingLog, so I added one commit to get the messages printed with pex.logging, before the logging migration. The effective changes are the same. I merged the lsst_dm_stack_demo patch and am running Jenkins.
Hide
Hsin-Fang Chiang added a comment -

Jenkins and demo passed

The afw and meas_astrom patches are no longer needed because those minor dependency updates have been done in other tickets. So this ticket include daf_persistence, daf_butlerUtils, and cat.

Show
Hsin-Fang Chiang added a comment - Jenkins and demo passed The afw and meas_astrom patches are no longer needed because those minor dependency updates have been done in other tickets. So this ticket include daf_persistence , daf_butlerUtils , and cat .
Hide
Hsin-Fang Chiang added a comment -

daf_persistence, daf_butlerUtils, and cat are merged.

Show
Hsin-Fang Chiang added a comment - daf_persistence , daf_butlerUtils , and cat are merged.

## People

• Assignee:
Hsin-Fang Chiang
Reporter:
Hsin-Fang Chiang
Reviewers:
Kian-Tat Lim
Watchers:
Hsin-Fang Chiang, Kian-Tat Lim, Nate Pease, Tim Jenness