Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19525

Demonstrate image differencing w/ decorrelation using HiTS instcals

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Repeat DM-19524 using HiTS data.

      Specifically,

      • Identify some HiTS data in the [ap_verify_ci_hits2015https://github.com/lsst/ap_verify_ci_hits2015] repository;
      • Fetch instcals corresponding to that data from the DECam archive;
      • Using those instcals as both template and science image;
      • Using A&L differencing, with decorrelation

      demonstrate that you can recover raw DIASource counts comparable to those reported in DMTN-021.

        Attachments

          Issue Links

            Activity

            Hide
            gkovacs Gabor Kovacs added a comment -

            Notes from the June false-positives spint trello board:

            We might want to consider the followings:

            • Shall we create a ticket to improve ap_pipe to support instcals? Currently, the lack of support originates from the dataset specification hard-wired as raw. If the repository does not contain raw type data, runDataRef in the task won't be called whatever its functionality may be. Preliminary decision was no.
            • Shall we develop a cmd line wrapper for AssociationTask in pipe_tasks ? The current wrapper script sets up a mock configuration similar enough to set up the same schema that ap_pipe produces, I am unsure that everything is OK here.

            Meredith Rawls:

            You did successfully make a ppdb, but you only ran 3 visits through so the largest number of DIASources for any object is 3. And there are some mysterious CCDs which appear to have either not processed properly or have zero sources in them for some other reason. Is it possible to run this with the full HiTS 2015 dataset instead of just a few visits? (It will take a while... and should probably be done via slurm...)

            Scripts and command logs are located in lsst-dm/ap_pipe-notebooks, currently in the branch u/gkovacs/false_positives but will be merged before the completion of this ticket.

            files: manual_procCcd_imageDiff_apAssoc.md ; run_assoc_2019-06-21.py

            Show
            gkovacs Gabor Kovacs added a comment - Notes from the June false-positives spint trello board: We might want to consider the followings: Shall we create a ticket to improve ap_pipe to support instcals ? Currently, the lack of support originates from the dataset specification hard-wired as raw . If the repository does not contain raw type data, runDataRef in the task won't be called whatever its functionality may be. Preliminary decision was no. Shall we develop a cmd line wrapper for AssociationTask in pipe_tasks ? The current wrapper script sets up a mock configuration similar enough to set up the same schema that ap_pipe produces, I am unsure that everything is OK here. Meredith Rawls : You did successfully make a ppdb, but you only ran 3 visits through so the largest number of DIASources for any object is 3. And there are some mysterious CCDs which appear to have either not processed properly or have zero sources in them for some other reason. Is it possible to run this with the full HiTS 2015 dataset instead of just a few visits? (It will take a while... and should probably be done via slurm...) Scripts and command logs are located in lsst-dm/ap_pipe-notebooks , currently in the branch u/gkovacs/false_positives but will be merged before the completion of this ticket. files: manual_procCcd_imageDiff_apAssoc.md ; run_assoc_2019-06-21.py
            Hide
            mrawls Meredith Rawls added a comment -

            Thanks for keeping clear notes about what you had to do to effectively "run ap_pipe" on instcals!

            I'll just note that the last part of this ticket, "demonstrate that you can recover raw DIASource counts comparable to those reported in DMTN-021," wound up being out of scope. I'll be taking a look at that shortly myself.

            Show
            mrawls Meredith Rawls added a comment - Thanks for keeping clear notes about what you had to do to effectively "run ap_pipe" on instcals! I'll just note that the last part of this ticket, "demonstrate that you can recover raw DIASource counts comparable to those reported in DMTN-021," wound up being out of scope. I'll be taking a look at that shortly myself.
            Hide
            swinbank John Swinbank added a comment -

            By “out of scope”, we mean “will be done on DM-20534”.

            Show
            swinbank John Swinbank added a comment - By “out of scope”, we mean “will be done on DM-20534 ”.

              People

              • Assignee:
                gkovacs Gabor Kovacs
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Meredith Rawls
                Watchers:
                Eric Bellm, Gabor Kovacs, John Swinbank, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel