Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7376

Failure to identify location of clear supernova in a subtraction

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_diffim
    • Labels:
      None

      Description

      I've run a subtraction of two images, one with a clear supernova (SNR~100), the other taken a year later. The supernova shows up very clearly, but does not get properly identified by the object detection and measurement.

      If run with config.doMerge=False then I get the first attachment, which shows the identification of a footprint in which the SN is found, but the X, Y position are set to be the presumably the first object that went in to the footprint. It's not re-centered at the SN.

      If I run with config.doMerge=True, then I get the third attachment, in which I believe the SN has been subsumed into the larger object of the dipole at the galaxy center.

      Some notes
      1. There is significant galaxy background (that's what I wanted to do the subtraction). This changes the noise properties and, from my understanding of a discussion on Hipchat, breaks assumptions made by the de-correlation step.
      2. These images are from a NIR detector, so they are created from a sequence of raw images in a grid dither pattern, which have been processed and combined to make these final images. This means that the noise varies significantly across the image, from the center which is covered by all raw images, to the corners which are each just covered by one.

      To recreate on lsst-dev.ncsa.illinois.edu:

      Set up the LSST stack + some custom branches in pipe_tasks, ip_diffim, obs_file:

      . ~wmwv/.bashrc.lsst
      

      WIYN=/lsst8/wmwv/WIYN
      imageDifference.py ${WIYN}/test_dr1 --id fileroot=SN2011gy_A_H_20111115  --templateId fileroot=SN2011gy_A_H_20121028 --output ${WIYN}/test_dr1 --configfile ~wmwv/wiyn/diffimconfig.py --logdest wiyn_imageDifference.log --clobber-config --clobber-versions
      

      If you're testing for yourself, you may wish to change the output repo to your own cusomt repo so we don't accidentally confuse each other with various runs.

      The active parts of diffimconfig.py are

      config.doUseRegister=False
      config.convolveTemplate=True
      config.doWriteMatchedExp=True
      config.doDecorrelation=True
      config.doMerge=True
      config.subtract.kernel.name='AL'
      from lsst.ip.diffim.getTemplate import GetCalexpAsTemplateTask
      config.getTemplate.retarget(GetCalexpAsTemplateTask)
      

      For a quick start, the following will display the images:

      ds9 /lsst8/wmwv/WIYN/test_dr1/calexp/SN2011gy_A_H_{20111115,20121028}.fits /lsst8/wmwv/WIYN/test_dr1/diff/SN2011gy_A_H_20111115/diffexp.fits
      

        Attachments

          Issue Links

            Activity

            Hide
            reiss David Reiss added a comment -

            I agree with Colin Slater's assessment. This needs much further discussion. I should note that for this image, the case of doMerge=True is similar to Colin Slater's example case, but because the centroid is (now) weighted by flux, the resulting centroid is close (this is helped by the fact that the SN is very bright) – (1354.9, 1345.9) for doMerge=False, vs. (1353.0, 1347.0) for doMerge=True. This is not the correct answer, but it is still an improvement.

            Michael Wood-Vasey I believe that it would not be trivial to effectively capture this issue in a test. Perhaps worthy of a separate ticket.

            Show
            reiss David Reiss added a comment - I agree with Colin Slater 's assessment. This needs much further discussion. I should note that for this image, the case of doMerge=True is similar to Colin Slater 's example case, but because the centroid is (now) weighted by flux, the resulting centroid is close (this is helped by the fact that the SN is very bright) – (1354.9, 1345.9) for doMerge=False, vs. (1353.0, 1347.0) for doMerge=True. This is not the correct answer, but it is still an improvement. Michael Wood-Vasey I believe that it would not be trivial to effectively capture this issue in a test. Perhaps worthy of a separate ticket.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            I just created DM-7395 as a general ticket to create a test suite.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - I just created DM-7395 as a general ticket to create a test suite.
            Hide
            reiss David Reiss added a comment -

            Would you mind doing a review of this small fix to a bug in ip_diffim? Take a look at the ticket page to see what the specific issue was and let me know if you have any questions. Thanks.

            Show
            reiss David Reiss added a comment - Would you mind doing a review of this small fix to a bug in ip_diffim? Take a look at the ticket page to see what the specific issue was and let me know if you have any questions. Thanks.
            Hide
            vpk24 Vishal Kasliwal [X] (Inactive) added a comment -

            While I'm not very familiar with ip_diffim, the work done on the ticket looks reasonable given the scope of the ticket. Can you clean up the commented out code and rebase off've master? Otherwise it looks good!

            Show
            vpk24 Vishal Kasliwal [X] (Inactive) added a comment - While I'm not very familiar with ip_diffim, the work done on the ticket looks reasonable given the scope of the ticket. Can you clean up the commented out code and rebase off've master? Otherwise it looks good!
            Hide
            reiss David Reiss added a comment -

            Thanks a lot for the review, Vishal Kasliwal [X]. I've made the requested changes and tested the build on Jenkins:
            https://ci.lsst.codes/job/stack-os-matrix/15183/label=centos-6,python=py2/console

            Merged.

            Show
            reiss David Reiss added a comment - Thanks a lot for the review, Vishal Kasliwal [X] . I've made the requested changes and tested the build on Jenkins: https://ci.lsst.codes/job/stack-os-matrix/15183/label=centos-6,python=py2/console Merged.

              People

              Assignee:
              reiss David Reiss
              Reporter:
              wmwood-vasey Michael Wood-Vasey
              Reviewers:
              Vishal Kasliwal [X] (Inactive)
              Watchers:
              Colin Slater, David Reiss, Michael Wood-Vasey, Simon Krughoff, Vishal Kasliwal [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.