Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32683

Print refcat name in log when loading

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_algorithms
    • Labels:
      None

      Description

      Often I have wanted to be certain that I'm loading the refcat I think I am: there are a few different ways to configure it, and I've gotten them tangled. The refcat loader has an INFO level "Loading reference objects from region..." message that should also include the name of the refcat it's loading to help with such debugging. Lee Kelvin's jointcal debugging this morning was the final impetus to do this, but I think it will be generally useful.

        Attachments

          Activity

          Hide
          Parejkoj John Parejko added a comment -

          Lee Kelvin: this ticket should help you with your "what refcat did I load?" debugging. See the two separate commits: I cleaned up some of the gen3 config stuff along the way, so it was slightly more than completely trivial.

          Show
          Parejkoj John Parejko added a comment - Lee Kelvin : this ticket should help you with your "what refcat did I load?" debugging. See the two separate commits: I cleaned up some of the gen3 config stuff along the way, so it was slightly more than completely trivial.
          Show
          Parejkoj John Parejko added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35453/pipeline
          Hide
          lskelvin Lee Kelvin added a comment - - edited

          This works very nicely during my testing with some Merian DECam data, for example:

          old:

          ====== Now processing astrometry...
          Loading reference objects from region bounded by [128.82970545, 132.26135235], [-4.04593700, -0.61713353] RA Dec
          ...
          ====== Now processing photometry...
          Loading reference objects from region bounded by [128.82970545, 132.26135235], [-4.04593700, -0.61713353] RA Dec
          

          new:

          ====== Now processing astrometry...
          Loading reference objects from gaia_dr2_20200414 in region bounded by [128.82970545, 132.26135235], [-4.04593700, -0.61713353] RA Dec
          ...
          ====== Now processing photometry...
          Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [128.82970545, 132.26135235], [-4.04593700, -0.61713353] RA Dec
          

          For reference, the above log outputs were generated from runs instigated using the following command:

          pipetask --long-log run --register-dataset-types -j 12 \
          -b $REPO --instrument lsst.obs.decam.DarkEnergyCamera \
          -i DECam/defaults/merian,u/lskelvin/drp-pipe/testing \
          --output-run u/lskelvin/drp-pipe/jointcal-noticket \
          -p $DRP_PIPE_DIR/recipes/DECam/Merian.yaml#jointcal \
          -d "instrument='DECam' AND skymap='hsc_rings_v1' AND tract=9071 AND patch=0"
          

          Only a minor comment on the PR. During testing running the jointcal task, the new code ran 7s slower (~2%). I put this down to variable load on my test machine, and likely not something to worry about. Otherwise, this looks great, thanks! As Jenkins hasn't complained, I think this looks good to merge to me.

          Show
          lskelvin Lee Kelvin added a comment - - edited This works very nicely during my testing with some Merian DECam data, for example: old: ====== Now processing astrometry... Loading reference objects from region bounded by [128.82970545, 132.26135235], [-4.04593700, -0.61713353] RA Dec ... ====== Now processing photometry... Loading reference objects from region bounded by [128.82970545, 132.26135235], [-4.04593700, -0.61713353] RA Dec new: ====== Now processing astrometry... Loading reference objects from gaia_dr2_20200414 in region bounded by [128.82970545, 132.26135235], [-4.04593700, -0.61713353] RA Dec ... ====== Now processing photometry... Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [128.82970545, 132.26135235], [-4.04593700, -0.61713353] RA Dec For reference, the above log outputs were generated from runs instigated using the following command: pipetask --long-log run --register-dataset-types -j 12 \ -b $REPO --instrument lsst.obs.decam.DarkEnergyCamera \ -i DECam/defaults/merian,u/lskelvin/drp-pipe/testing \ --output-run u/lskelvin/drp-pipe/jointcal-noticket \ -p $DRP_PIPE_DIR/recipes/DECam/Merian.yaml#jointcal \ -d "instrument='DECam' AND skymap='hsc_rings_v1' AND tract=9071 AND patch=0" Only a minor comment on the PR. During testing running the jointcal task, the new code ran 7s slower (~2%). I put this down to variable load on my test machine, and likely not something to worry about. Otherwise, this looks great, thanks! As Jenkins hasn't complained, I think this looks good to merge to me.
          Show
          Parejkoj John Parejko added a comment - cleanup jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35492/pipeline

            People

            Assignee:
            Parejkoj John Parejko
            Reporter:
            Parejkoj John Parejko
            Reviewers:
            Lee Kelvin
            Watchers:
            Eli Rykoff, Ian Sullivan, John Parejko, Lee Kelvin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.