Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-37584

Rewrite DMTN-172 as a Great Calibration Refactor plan

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      DMTN-172 was originally an attempt to crowd-source input on how to rework the initial stage of the DRP pipelines, but now it's really just

      • a simplified description of the "current" pipelines that's actually out of date in a few key ways
      • a vision of one possible future pipeline that's also slightly out of date, since some of its recommendations have already been implemented with some differences from it envisioned them.

      At the same time, the real pipeline has gotten much closer to what we want in the end, and a few key questions now have answers:

      • We're definitely going to run Piff in a full-focal-plane task, regardless of whether it makes use of more than one detector at once, and that task (finalizeCharacterization) exists.
      • We're going to user per-visit ExposureCatalog datasets to take consolidate the various Exposure components at two or three points within the pipeline.  The last of these is what could be a data product that is retained long term for use in recreating PVIs, though using components attached to compressed PVIs is another option.
      • While it hasn't been fully demonstrated yet, we can probably start assuming we'll use compensated aperture fluxes to feed FGCM and hence don't need to do our most sophisticated backgrounds before FGCM.
      • While it's unclear whether we'll be able to get away with not running PSFEx (or something even simpler) to do PSF modeling at the very beginning, we can just assume we are now and not expect any big-picture changes from disabling it later.

      With that in mind, it's time to rewrite this technote to put together a concrete plan for ISR through PVIs and final Sources.  It may not have all of the bells and whistles considered at CalibPalooza, but it should be actionable.

      Three things it should definitely aim to include:

      • either doing initial astrometry in a full-focal-plane task or adding a full-focal-plane astrometry recovery task shortly afterwards;
      • a point at which we can re-do all pixel interpolation from the masks, and generate similarly interpolated noise images for coaddition at the same time;
      • consideration for masks derived from coaddition and diffim in the final PVIs and source tables.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            Parejkoj, I've responded to all crowd-sourced comments from way back when (mostly answering questions, occasionally changing text), and I think this is ready for a formal review to get it merged.

            Timeline is up to you, since I think you're the one who gains the most from getting it merged right now.

            PR is https://github.com/lsst-dm/dmtn-172/pull/2

            jbosch Jim Bosch added a comment - Parejkoj , I've responded to all crowd-sourced comments from way back when (mostly answering questions, occasionally changing text), and I think this is ready for a formal review to get it merged. Timeline is up to you, since I think you're the one who gains the most from getting it merged right now. PR is https://github.com/lsst-dm/dmtn-172/pull/2
            Parejkoj John Parejko added a comment -

            I put a number of comments on the PR. I saw in your responses to Nate that you didn't want it to get bogged down in naming discussions, but we should replace all the "bootstrapImage" with "calibrateImage" (as now exists) and "bootstrapVisit" with "consolidateVisit" or something like that.

            Parejkoj John Parejko added a comment - I put a number of comments on the PR. I saw in your responses to Nate that you didn't want it to get bogged down in naming discussions, but we should replace all the "bootstrapImage" with "calibrateImage" (as now exists) and "bootstrapVisit" with "consolidateVisit" or something like that.

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              John Parejko
              Eli Rykoff, Jim Bosch, John Parejko, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.