Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22078

Reconciling the DPDD with the required information for pipeline QA.

    XMLWordPrintable

    Details

    • Epic Name:
      sst-dpdd-qa-reconciliation
    • Story Points:
      20
    • Team:
      DM Science
    • Cycle:
      Spring 2020

      Description

      As discussed at DMLT F2F 2019-10-28, work is needed to reconcile the outputs of the pipeline processing with what is specified in the DPDD. Pipelines developers need to output many more quantities from the processing, primarily for QA purposes, than are specified in the DPDD. Not all of these quantities will necessarily be interesting for science.

      A possible approach is to add more information, not currently specified in the DPDD, to the tables, or, to put some of the QA quantities in a separate table that can be joined with. The DPDD is regarded as a minimum, but note that changes have impacts on the sizing model.

      This epic captures tickets for changes to the DPDD to reconcile the outputs of the pipeline processing with the DPDD

        Attachments

          Issue Links

            Activity

            Hide
            erykoff Eli Rykoff added a comment -

            Re: DPDD columns, the DPDD currently says this about the sky column in the Data Release Catalog Source Table: “Estimated background (sky) surface brightness at the position (centroid) of the source” with units nJy/arcsec^2. What does this mean? Right now, in https://github.com/lsst/obs_subaru/blob/master/policy/Source.yaml this is defined as base_LocalBackground_instFlux which is the local background in the calexp (which has been sky/background subtracted!) measured in a somewhat arbitrary annulus around the source. But I’m wondering if instead this is supposed to be an estimate of the full sky/background from the calexpBackground at the location of the source? As far as I can tell, this isn’t actually specified anywhere in the DPDD.

            Show
            erykoff Eli Rykoff added a comment - Re: DPDD columns, the DPDD currently says this about the sky column in the Data Release Catalog Source Table: “Estimated background (sky) surface brightness at the position (centroid) of the source” with units nJy/arcsec^2. What does this mean? Right now, in https://github.com/lsst/obs_subaru/blob/master/policy/Source.yaml this is defined as base_LocalBackground_instFlux which is the local background in the calexp (which has been sky/background subtracted!) measured in a somewhat arbitrary annulus around the source. But I’m wondering if instead this is supposed to be an estimate of the full sky/background from the calexpBackground at the location of the source? As far as I can tell, this isn’t actually specified anywhere in the DPDD.
            Hide
            ctslater Colin Slater added a comment -

            I would interpret it as an evaluation of the background model, not the measurement after background-subtraction.

            Show
            ctslater Colin Slater added a comment - I would interpret it as an evaluation of the background model, not the measurement after background-subtraction.

              People

              Assignee:
              lguy Leanne Guy
              Reporter:
              lguy Leanne Guy
              Watchers:
              Colin Slater, Eli Rykoff, Hsin-Fang Chiang, Jim Bosch, Lauren MacArthur, Leanne Guy, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Due:
                Created:
                Updated:
                Start date:
                End date:

                  Summary Panel