Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6984

Suggest logging migration in daf_persistence and daf_butlerUtils

    Details

    • Type: Story
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cat, daf_persistence, obs_base
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Team:
      Process Middleware

      Description

      Use lsst::log instead of pex::logging in daf_persistence and daf_butlerUtils

        Issue Links

          Activity

          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Jenkins passed building (including lsst_distrib, lsst_sims, lsst_ci), but failed at the stack demo , because the default logging output is stdout in lsst.log instead of stderr in pex.logging, the demo script bin/export-results redirects its std output to make the "detected-sources(_small).txt" file, and there are a few debug messages from daf.persistence and daf.butlerUtils when running that script. For running command line tasks and unit tests, log has been configured to go to stderr. The standalone script bin/export-results needs to configure log as well. This is done in the ticket branch of lsst_dm_stack_demo, but Jenkins doesn't get that ticket branch. I've tested it on my machine, to reproduce the error with master and pass the demo with the ticket branch.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Jenkins passed building (including lsst_distrib, lsst_sims, lsst_ci), but failed at the stack demo , because the default logging output is stdout in lsst.log instead of stderr in pex.logging , the demo script bin/export-results redirects its std output to make the "detected-sources(_small).txt" file, and there are a few debug messages from daf.persistence and daf.butlerUtils when running that script. For running command line tasks and unit tests, log has been configured to go to stderr. The standalone script bin/export-results needs to configure log as well. This is done in the ticket branch of lsst_dm_stack_demo , but Jenkins doesn't get that ticket branch. I've tested it on my machine, to reproduce the error with master and pass the demo with the ticket branch.
          Hide
          ktl Kian-Tat Lim added a comment -

          Looks OK. The BlockTimingLog replacement is not generating exactly the same message as before, but I don't think that's really a problem.

          Show
          ktl Kian-Tat Lim added a comment - Looks OK. The BlockTimingLog replacement is not generating exactly the same message as before, but I don't think that's really a problem.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Now I see by the messages generated previously, you didn't mean the messages before this old daf_persistence commit but the pex.logging.BlockTimingLog messages. I corrected the messages to be exactly the same now (with format differences).

          Use tests/butlerPickle.py as an example and set a low threshold/level, this is before the logging migration (pex_logging):

            daf.persistence.butler.read DEBUG: Starting read from None at PickleStorage(foo1.pickle)
            daf.persistence.butler.read DEBUG: Ending read from None at PickleStorage(foo1.pickle)
          

          This is after the logging migration (lsst_log):

          DEBUG daf.persistence.butler: Starting read from None at PickleStorage(foo1.pickle)
          DEBUG daf.persistence.butler: Ending read from None at PickleStorage(foo1.pickle)
          

          The master daf_persistence doesn't really use BlockTimingLog, so I added one commit to get the messages printed with pex.logging, before the logging migration. The effective changes are the same.

          I merged the lsst_dm_stack_demo patch and am running Jenkins.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Now I see by the messages generated previously, you didn't mean the messages before this old daf_persistence commit but the pex.logging.BlockTimingLog messages. I corrected the messages to be exactly the same now (with format differences). Use tests/butlerPickle.py as an example and set a low threshold/level, this is before the logging migration (pex_logging): daf.persistence.butler.read DEBUG: Starting read from None at PickleStorage(foo1.pickle) daf.persistence.butler.read DEBUG: Ending read from None at PickleStorage(foo1.pickle) This is after the logging migration (lsst_log): DEBUG daf.persistence.butler: Starting read from None at PickleStorage(foo1.pickle) DEBUG daf.persistence.butler: Ending read from None at PickleStorage(foo1.pickle) The master daf_persistence doesn't really use BlockTimingLog, so I added one commit to get the messages printed with pex.logging, before the logging migration. The effective changes are the same. I merged the lsst_dm_stack_demo patch and am running Jenkins.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Jenkins and demo passed

          The afw and meas_astrom patches are no longer needed because those minor dependency updates have been done in other tickets. So this ticket include daf_persistence, daf_butlerUtils, and cat.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Jenkins and demo passed The afw and meas_astrom patches are no longer needed because those minor dependency updates have been done in other tickets. So this ticket include daf_persistence , daf_butlerUtils , and cat .
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          daf_persistence, daf_butlerUtils, and cat are merged.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - daf_persistence , daf_butlerUtils , and cat are merged.

            People

            • Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Hsin-Fang Chiang, Kian-Tat Lim, Nate Pease, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development