Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8230

forcedPhotCcd.py doesn't work on DECam data

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_base, obs_base
    • Labels:
      None
    • Templates:
    • Story Points:
      1
    • Epic Link:
    • Sprint:
      DRP F16-6
    • Team:
      Data Release Production

      Description

      forcedPhotCcd.py on DECam data with tract,visit,ccdnum data IDs fails with a registry lookup error. This is due to an incomplete workaround for a known issue: incomplete data IDs with skymap keys cannot be filled in by the registry, because the skymap keys are included in the registry queries but don't exist in the registry tables. Our usual workaround for datasets like forced_src (which typically require a registry lookup to fill in unspecified data ID keys) is to do registry lookup on a similar type without a skymap key (such as src) by calling Butler.subset.

      In obs_decam, however, the src template requires only visit, ccdnum, while forced_src also requires filter. Because the data ID we're passing to Butler.subset is complete for src, it skips the registry lookup and passes an incomplete (for forced_src) data ID on to later code, which fails trying to complete a forced_src data ID that includes tract.

      There are a few ways I can imagine fixing this:

      • We could modify the obs_decam template to remove filter. This is really just a band-aid, but it'd be easy, and since we're discovering this bug now I think it's unlikely there's any DECam CCD forced photometry results sitting on disk that this would break.
      • We could copy some of the logic in ButlerSubset into the data ID mangling code for forcedPhotCcd.py, having it call getKeys and queryMetadata directly to fill out the data ID. We could alternatively add a new Butler method to do this more flexibly that the forcedPhotCcd.py code could delegate to.
      • We could modify ButlerSubset or queryMetdata to explicitly ignore skymap (tract and patch) data ID keys when querying the registry. I think this would be a simple fix and it would avoid painful workarounds of the sort mentioned above when reading forced_src data.

      Kian-Tat Lim, Nate Pease, the last of the above solutions is my preferred one, but I wanted to check with you to make sure you weren't bothered by the special-casing of "tract" and "patch" data ID keys in the butler. I'm happy to do the work myself (this is to support some DESC work that uncovered the bug, and I'm at DESC hack week).

        Attachments

          Container Issues

            Activity

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Nate Pease, Paul Price
                Watchers:
                Jim Bosch, Kian-Tat Lim, Nate Pease, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel