Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23631

Create simple python script to create and insert fakes for the HiTS2015 AP processing.

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe, ap_verify
    • Labels:
      None
    • Story Points:
      8
    • Epic Link:
    • Sprint:
      AP S20-4 (March)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Create a simple script to insert fakes for AP using the HiTS2015 dataset. Likely this will be on an un-merged ticket of ap_verify.

      This script will be fairly hacky as it will copy data around in the ap_verify_hists2015 dataset and edit the DecamMapper on the fly.

        Attachments

          Issue Links

            Activity

            Hide
            cmorrison Chris Morrison added a comment - - edited

            Eric Bellm I've completed an initial run of fake injection into the full ap_verify_hits2015 dataset.

            You can find the resulting Apdb here:

            /project/morriscb/src/ap_verify/fakesRepo/output/rerun/testFakes/fakesAssociation.db

            The fakes that were injected are cataloged here:

            calexp: /project/morriscb/src/ap_verify/fakesRepo/calexpFakesTract0.csv
            coadd: /project/morriscb/src/ap_verify/fakesRepo/coaddFakesTract0.csv
            

            The calexp and coadd files share half of their sources, the rest are unique to the calexp or coadd injection. 180k PSF like sources were created over the tract in total. The shared objects are the last half of the calexp file and the first half of the coadd. The positions are uniformly random over the tract with the magnitudes being uniform random from 19-24 in g band magnitude. The calexp file has an additional 2% uniform scatter added to the magnitude so an object in the calexp should have a magnitude within 2% of the one in the coadd.

            I haven't looking in the Apdb yet, but I am about to attempt to confirm that the sources were added to both the calexp and coadd.

            Show
            cmorrison Chris Morrison added a comment - - edited Eric Bellm I've completed an initial run of fake injection into the full ap_verify_hits2015 dataset. You can find the resulting Apdb here: /project/morriscb/src/ap_verify/fakesRepo/output/rerun/testFakes/fakesAssociation.db The fakes that were injected are cataloged here: calexp: /project/morriscb/src/ap_verify/fakesRepo/calexpFakesTract0.csv coadd: /project/morriscb/src/ap_verify/fakesRepo/coaddFakesTract0.csv The calexp and coadd files share half of their sources, the rest are unique to the calexp or coadd injection. 180k PSF like sources were created over the tract in total. The shared objects are the last half of the calexp file and the first half of the coadd. The positions are uniformly random over the tract with the magnitudes being uniform random from 19-24 in g band magnitude. The calexp file has an additional 2% uniform scatter added to the magnitude so an object in the calexp should have a magnitude within 2% of the one in the coadd. I haven't looking in the Apdb yet, but I am about to attempt to confirm that the sources were added to both the calexp and coadd.
            Hide
            cmorrison Chris Morrison added a comment -

            Created a ap_verify/pipe script with configurable command line options to inject fakes into the the ap_verify_hits2015 or ap_verify_ci_hits2015 dataset.

            To use the script: clone and setup the branches associated with this ticket. Clone and setup the ap_verify_hits2015 dataset. It is recommended to submit the job to the lsst-dev verification cluster on one node with 24 jobs requested in the script.

            To use the CI dataset, one must currently edit the script. However, I could add this as an additional configurable if requested.

            To be clear, this ticket will not be merged to master as it is a hacky work around to inject fakes into AP that is reliant on Gen2 butler. The real and final script will be created on another ticket once AP is fully Gen3 compatible.

            Show
            cmorrison Chris Morrison added a comment - Created a ap_verify/pipe script with configurable command line options to inject fakes into the the ap_verify_hits2015 or ap_verify_ci_hits2015 dataset. To use the script: clone and setup the branches associated with this ticket. Clone and setup the ap_verify_hits2015 dataset. It is recommended to submit the job to the lsst-dev verification cluster on one node with 24 jobs requested in the script. To use the CI dataset, one must currently edit the script. However, I could add this as an additional configurable if requested. To be clear, this ticket will not be merged to master as it is a hacky work around to inject fakes into AP that is reliant on Gen2 butler. The real and final script will be created on another ticket once AP is fully Gen3 compatible.
            Hide
            cmorrison Chris Morrison added a comment -

            Currently with the ap_verify_hits2015 dataset, some patches have fakes inserted with NaN values. This is caused by the computeApertureFlux method of the PSF returning NaN for certain patches in the coadd. One specific example is dataId "tract=0 patch=10,9 filter=g". Fixing or investigating this is beyond the scope of this ticket.

            Show
            cmorrison Chris Morrison added a comment - Currently with the ap_verify_hits2015 dataset, some patches have fakes inserted with NaN values. This is caused by the computeApertureFlux method of the PSF returning NaN for certain patches in the coadd. One specific example is dataId "tract=0 patch=10,9 filter=g". Fixing or investigating this is beyond the scope of this ticket.
            Hide
            ebellm Eric Bellm added a comment -

            Thanks Chris Morrison! I created https://jira.lsstcorp.org/browse/DM-23914 for the followup Gen 3 version.

            Show
            ebellm Eric Bellm added a comment - Thanks Chris Morrison ! I created https://jira.lsstcorp.org/browse/DM-23914 for the followup Gen 3 version.

              People

              • Assignee:
                cmorrison Chris Morrison
                Reporter:
                cmorrison Chris Morrison
                Reviewers:
                Eric Bellm
                Watchers:
                Chris Morrison, Eric Bellm, John Swinbank
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel