Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12584

clarify the role of fake source implantation in Level 1 detection efficiencies

    Details

      Description

      The purpose of this ticket is to resolve whether or not the implantation of fake sources will be done as part of the Level 1 detection efficiency characterization, or if an alternate method will be sufficient for the science needs. K.-T. says the computational system is sized to accommodate it. If this issue remains unclear, perhaps a study that assess the options, their computational costs, and their scientific payoff should be done so that DM can better assess the need for fake sources and allocating staffing hours as necessary. We note that this is not a question limited to Special Programs, but it is possible that this capability is a feature that a user-proposed program might expect in the processing pipeline.

      Section 4.9 of DMTN-065 summarizes the existing requirements on DM to produce detection efficiencies, copied here for convenience:

      • The DPDD (LSE-163) does not have any specific data product related to detection efficiencies, but Section 3.2 ”Image Characterization Data” does specify that ”Each processed image .. will record information on the pixel variance ... as well as the per-pixel masks ... These will allow the users to determine the validity and usefullness of each pixel in estimating the flux density recorded in that area of the sky”.

      • The DMSR (LSE-61), Section 1.2.11 ”Level 1 Data Quality Report Definition” (ID: DMS-REQ0097): ”The DMS shall produce a Level 1 Data Quality Report that contains indicators of data quality that result from running the DMS pipelines, including at least ... detection efficiency for point sources vs. mag for each utilized filter.” However, this is a nightly data quality assessment and not a per-image product.

      • The DMAD (LDM-151), Section 5.6.3 ”MakeSelectionMaps”, states that this calibration step ”is responsible for producing multi-scale maps that describe LSST’s depth and efficiency at detecting different classes of object. The details of what metrics will be mapped, the format and scale of the maps (e.g. hierarchical pixelizations vs. polygons), and the way the metrics will be computed are all unknown”. It also states that this must be extendable to Level 3, but that ”the details of what DM will provide still needs to be clarified to the community”, and notes that the reprocessing time for fake plants could be prohibitive. (Section 3 ”Alert Production” also specifies that in LDM-151 ”we do not address estimation of the selection function for alert generation through the injection of simulated sources ... Source detection thresholds can be estimated through the use of sky sources”.)

      Additional context for this ticket can be found in Section 4.9 of the DMTN study on DM and Special Programs, available at https://dmtn-065.lsst.io/

        Attachments

          Issue Links

            Activity

            mgraham Melissa Graham created issue -
            ebellm Eric Bellm made changes -
            Field Original Value New Value
            Summary clarify the roll of fake source implantation in Level 1 detection efficiencies clarify the role of fake source implantation in Level 1 detection efficiencies
            Hide
            ebellm Eric Bellm added a comment - - edited

            Fake injection is not without its own challenges — it's hard to be sure we're truly spanning the space of real transient signals, and there's a bit of circular reasoning involved in sorting real-but-not-injected sources from boguses — but I don't see any other way to systematically show we're meeting our requirements other than fake injection, at least before operations.

            Show
            ebellm Eric Bellm added a comment - - edited Fake injection is not without its own challenges — it's hard to be sure we're truly spanning the space of real transient signals, and there's a bit of circular reasoning involved in sorting real-but-not-injected sources from boguses — but I don't see any other way to systematically show we're meeting our requirements other than fake injection, at least before operations.
            Hide
            mjuric Mario Juric added a comment -

            What Melissa Graham and I had in mind was to try to clarify in one of our docs that determining point-source detection efficiency (as a function of image position and magnitude) will be done in real-time (Level 1), while that estimates of our sensitivity to specific types of objects (e.g., could we identify a Type Ia in a particular pointing given the observations we've taken) is left to the community (Level 3).

            Show
            mjuric Mario Juric added a comment - What Melissa Graham and I had in mind was to try to clarify in one of our docs that determining point-source detection efficiency (as a function of image position and magnitude) will be done in real-time (Level 1), while that estimates of our sensitivity to specific types of objects (e.g., could we identify a Type Ia in a particular pointing given the observations we've taken) is left to the community (Level 3).
            swinbank John Swinbank made changes -
            Team DM Science [ 12218 ]
            mgraham Melissa Graham made changes -
            Description The purpose of this ticket is to resolve whether or not the implantation of fake sources will be done as part of the Level 1 detection efficiency characterization, or if an alternate method will be sufficient for the science needs. K.-T. says the computational system is sized to accommodate it. If this issue remains unclear, perhaps a study that assess the options, their computational costs, and their scientific payoff should be done so that DM can better assess the need for fake sources and allocating staffing hours as necessary. We note that this is not a question limited to Special Programs, but it is possible that this capability is a feature that a user-proposed program might expect in the processing pipeline.

            The nominal deadline has been set to 1.5 months prior to the expected call for Special Programs white paper proposals, but need not be resolved by then. Additional context for this ticket can be found in Section 4.9 of the DMTN study on DM and Special Programs, available at https://github.com/lsst-dmsst/DM_SP_study/blob/master/ms.pdf
            The purpose of this ticket is to resolve whether or not the implantation of fake sources will be done as part of the Level 1 detection efficiency characterization, or if an alternate method will be sufficient for the science needs. K.-T. says the computational system is sized to accommodate it. If this issue remains unclear, perhaps a study that assess the options, their computational costs, and their scientific payoff should be done so that DM can better assess the need for fake sources and allocating staffing hours as necessary. We note that this is not a question limited to Special Programs, but it is possible that this capability is a feature that a user-proposed program might expect in the processing pipeline.

            The nominal deadline has been set to 1.5 months prior to the expected call for Special Programs white paper proposals, but need not be resolved by then. Additional context for this ticket can be found in Section 4.9 of the DMTN study on DM and Special Programs, available at https://dmtn-065.lsst.io/
            Hide
            ebellm Eric Bellm added a comment -

            I think this is the right approach.  Similar to the one taken here: http://adsabs.harvard.edu/abs/2017ApJS..230....4F  One challenge is that efficiency also depends on position on host galaxies, so it's not as simple as just spraying fakes over the image at random.

            Show
            ebellm Eric Bellm added a comment - I think this is the right approach.  Similar to the one taken here: http://adsabs.harvard.edu/abs/2017ApJS..230....4F   One challenge is that efficiency also depends on position on host galaxies, so it's not as simple as just spraying fakes over the image at random.
            Hide
            mgraham Melissa Graham added a comment -

            I'm still open to working this towards a resolution, e.g., compiling a study of the scientific needs for detection efficiencies for LSST and the DM implementation options for meeting those needs, if that would be helpful. Detection efficiencies via fake implantation is something I've done to derive SN rates from an imaging survey, and of course for single-image detection limits.

            Show
            mgraham Melissa Graham added a comment - I'm still open to working this towards a resolution, e.g., compiling a study of the scientific needs for detection efficiencies for LSST and the DM implementation options for meeting those needs, if that would be helpful. Detection efficiencies via fake implantation is something I've done to derive SN rates from an imaging survey, and of course for single-image detection limits.
            mgraham Melissa Graham made changes -
            Assignee Mario Juric [ mjuric ] Eric Bellm [ ebellm ]
            mgraham Melissa Graham made changes -
            Due Date 15/Feb/18 31/Dec/18
            Hide
            mgraham Melissa Graham added a comment - - edited

            Reassigned to Eric but Melissa plans to move forward on this study.

            Note the existence of this relevant repo: https://github.com/lsst/synpipe

            Show
            mgraham Melissa Graham added a comment - - edited Reassigned to Eric but Melissa plans to move forward on this study. Note the existence of this relevant repo: https://github.com/lsst/synpipe
            lguy Leanne Guy made changes -
            Labels dm-sst SpecialPrograms dm-sst
            mgraham Melissa Graham made changes -
            Description The purpose of this ticket is to resolve whether or not the implantation of fake sources will be done as part of the Level 1 detection efficiency characterization, or if an alternate method will be sufficient for the science needs. K.-T. says the computational system is sized to accommodate it. If this issue remains unclear, perhaps a study that assess the options, their computational costs, and their scientific payoff should be done so that DM can better assess the need for fake sources and allocating staffing hours as necessary. We note that this is not a question limited to Special Programs, but it is possible that this capability is a feature that a user-proposed program might expect in the processing pipeline.

            The nominal deadline has been set to 1.5 months prior to the expected call for Special Programs white paper proposals, but need not be resolved by then. Additional context for this ticket can be found in Section 4.9 of the DMTN study on DM and Special Programs, available at https://dmtn-065.lsst.io/
            The purpose of this ticket is to resolve whether or not the implantation of fake sources will be done as part of the Level 1 detection efficiency characterization, or if an alternate method will be sufficient for the science needs. K.-T. says the computational system is sized to accommodate it. If this issue remains unclear, perhaps a study that assess the options, their computational costs, and their scientific payoff should be done so that DM can better assess the need for fake sources and allocating staffing hours as necessary. We note that this is not a question limited to Special Programs, but it is possible that this capability is a feature that a user-proposed program might expect in the processing pipeline.

            Section 4.9 of DMTN-065 summarizes the existing requirements on DM to produce detection efficiencies, copied here for convenience:

            • The DPDD (LSE-163) does not have any specific data product related to detection efficiencies, but Section 3.2 ”Image Characterization Data” does specify that ”Each processed image .. will record information on the pixel variance ... as well as the per-pixel masks ... These will allow the users to determine the validity and usefullness of each pixel in estimating the flux density recorded in that area of the sky”.

            • The DMSR (LSE-61), Section 1.2.11 ”Level 1 Data Quality Report Definition” (ID: DMS-REQ0097): ”The DMS shall produce a Level 1 Data Quality Report that contains indicators of data quality that result from running the DMS pipelines, including at least ... detection efficiency for point sources vs. mag for each utilized filter.” However, this is a nightly data quality assessment and not a per-image product.

            • The DMAD (LDM-151), Section 5.6.3 ”MakeSelectionMaps”, states that this calibration step ”is responsible for producing multi-scale maps that describe LSST’s depth and efficiency at detecting different classes of object. The details of what metrics will be mapped, the format and scale of the maps (e.g. hierarchical pixelizations vs. polygons), and the way the metrics will be computed are all unknown”. It also states that this must be extendable to Level 3, but that ”the details of what DM will provide still needs to be clarified to the community”, and notes that the reprocessing time for fake plants could be prohibitive. (Section 3 ”Alert Production” also specifies that in LDM-151 ”we do not address estimation of the selection function for alert generation through the injection of simulated sources ... Source detection thresholds can be estimated through the use of sky sources”.)

            Additional context for this ticket can be found in Section 4.9 of the DMTN study on DM and Special Programs, available at [https://dmtn-065.lsst.io/]
            mgraham Melissa Graham made changes -
            Risk Score 0
            mgraham Melissa Graham made changes -
            Link This issue is parent task of DM-19308 [ DM-19308 ]
            Hide
            mgraham Melissa Graham added a comment -

            Child task DM-19308 is now "in progress" with work on the draft DMTN proceeding at https://github.com/MelissaGraham/dmtn-tbd-deteffs

            Contributions fully welcome.

            Show
            mgraham Melissa Graham added a comment - Child task DM-19308 is now "in progress" with work on the draft DMTN proceeding at  https://github.com/MelissaGraham/dmtn-tbd-deteffs Contributions fully welcome.
            lguy Leanne Guy made changes -
            Epic Link DM-20827 [ 373834 ]

              People

              • Assignee:
                ebellm Eric Bellm
                Reporter:
                mgraham Melissa Graham
                Watchers:
                Eric Bellm, John Swinbank, Krzysztof Findeisen, Melissa Graham
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:

                  Summary Panel