Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22078

Reconciling the DPDD with the required information for pipeline QA.

    Details

    • Epic Name:
      sst-dpdd-qa-reconciliation
    • Story Points:
      20
    • Team:
      DM Science
    • Cycle:
      Spring 2020

      Description

      As discussed at DMLT F2F 2019-10-28, work is needed to reconcile the outputs of the pipeline processing with what is specified in the DPDD. Pipelines developers need to output many more quantities from the processing, primarily for QA purposes, than are specified in the DPDD. Not all of these quantities will necessarily be interesting for science.

      A possible approach is to add more information, not currently specified in the DPDD, to the tables, or, to put some of the QA quantities in a separate table that can be joined with. The DPDD is regarded as a minimum, but note that changes have impacts on the sizing model.

      This epic captures tickets for changes to the DPDD to reconcile the outputs of the pipeline processing with the DPDD

        Attachments

          Issue Links

            Activity

            Hide
            erykoff Eli Rykoff added a comment -

            Re: DPDD columns, the DPDD currently says this about the sky column in the Data Release Catalog Source Table: “Estimated background (sky) surface brightness at the position (centroid) of the source” with units nJy/arcsec^2. What does this mean? Right now, in https://github.com/lsst/obs_subaru/blob/master/policy/Source.yaml this is defined as base_LocalBackground_instFlux which is the local background in the calexp (which has been sky/background subtracted!) measured in a somewhat arbitrary annulus around the source. But I’m wondering if instead this is supposed to be an estimate of the full sky/background from the calexpBackground at the location of the source? As far as I can tell, this isn’t actually specified anywhere in the DPDD.

            Show
            erykoff Eli Rykoff added a comment - Re: DPDD columns, the DPDD currently says this about the sky column in the Data Release Catalog Source Table: “Estimated background (sky) surface brightness at the position (centroid) of the source” with units nJy/arcsec^2. What does this mean? Right now, in https://github.com/lsst/obs_subaru/blob/master/policy/Source.yaml this is defined as base_LocalBackground_instFlux which is the local background in the calexp (which has been sky/background subtracted!) measured in a somewhat arbitrary annulus around the source. But I’m wondering if instead this is supposed to be an estimate of the full sky/background from the calexpBackground at the location of the source? As far as I can tell, this isn’t actually specified anywhere in the DPDD.
            Hide
            ctslater Colin Slater added a comment -

            I would interpret it as an evaluation of the background model, not the measurement after background-subtraction.

            Show
            ctslater Colin Slater added a comment - I would interpret it as an evaluation of the background model, not the measurement after background-subtraction.

              People

              • Assignee:
                lguy Leanne Guy
                Reporter:
                lguy Leanne Guy
                Watchers:
                Colin Slater, Eli Rykoff, Jim Bosch, Lauren MacArthur, Leanne Guy, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Start date:
                  End date:

                  Summary Panel