Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ip_diffim
-
Labels:None
-
Story Points:8
-
Epic Link:
-
Sprint:Alert Production F16 - 8
-
Team:Alert Production
Description
I've run a subtraction of two images, one with a clear supernova (SNR~100), the other taken a year later. The supernova shows up very clearly, but does not get properly identified by the object detection and measurement.
If run with config.doMerge=False then I get the first attachment, which shows the identification of a footprint in which the SN is found, but the X, Y position are set to be the presumably the first object that went in to the footprint. It's not re-centered at the SN.
If I run with config.doMerge=True, then I get the third attachment, in which I believe the SN has been subsumed into the larger object of the dipole at the galaxy center.
Some notes
1. There is significant galaxy background (that's what I wanted to do the subtraction). This changes the noise properties and, from my understanding of a discussion on Hipchat, breaks assumptions made by the de-correlation step.
2. These images are from a NIR detector, so they are created from a sequence of raw images in a grid dither pattern, which have been processed and combined to make these final images. This means that the noise varies significantly across the image, from the center which is covered by all raw images, to the corners which are each just covered by one.
To recreate on lsst-dev.ncsa.illinois.edu:
Set up the LSST stack + some custom branches in pipe_tasks, ip_diffim, obs_file:
. ~wmwv/.bashrc.lsst
|
WIYN=/lsst8/wmwv/WIYN
|
imageDifference.py ${WIYN}/test_dr1 --id fileroot=SN2011gy_A_H_20111115 --templateId fileroot=SN2011gy_A_H_20121028 --output ${WIYN}/test_dr1 --configfile ~wmwv/wiyn/diffimconfig.py --logdest wiyn_imageDifference.log --clobber-config --clobber-versions
|
If you're testing for yourself, you may wish to change the output repo to your own cusomt repo so we don't accidentally confuse each other with various runs.
The active parts of diffimconfig.py are
config.doUseRegister=False
|
config.convolveTemplate=True
|
config.doWriteMatchedExp=True
|
config.doDecorrelation=True
|
config.doMerge=True
|
config.subtract.kernel.name='AL'
|
from lsst.ip.diffim.getTemplate import GetCalexpAsTemplateTask
|
config.getTemplate.retarget(GetCalexpAsTemplateTask)
|
For a quick start, the following will display the images:
ds9 /lsst8/wmwv/WIYN/test_dr1/calexp/SN2011gy_A_H_{20111115,20121028}.fits /lsst8/wmwv/WIYN/test_dr1/diff/SN2011gy_A_H_20111115/diffexp.fits
|
Attachments
Issue Links
Activity
I have updated the NaiveDipoleCentroid code to fix a number of logic errors. This now returns the correct centroid for the SN. The value is consistent (although not identical) whether doMerge is set to True or False.
This does not fix the issue with over-detection, which is related to the complex noise structure of the images (leading to poor background estimation and less-than-exact decorrelation). These will be addressed in other tickets.
I will test these fixes on DECam images to make sure I haven't broken anything, before submitting for code review.
Can you expand the test coverage to catch this issue to prevent future regressions?
Michael Wood-Vasey
added a comment - Can you expand the test coverage to catch this issue to prevent future regressions? The fix for NaiveDipoleCentroid is certainly better, but I also don't think it can be the final solution. For example, if that noise peak far from the SN had been negative, then this would report some flux weighted mean centroid, which I think is still the wrong answer. I think the bottom line is we might need the deblender (or some simplified version thereof), and testing that would be non-trivial work. That's definitely another ticket though, and this fix should be a good improvement in the interim.
Colin Slater
added a comment - The fix for NaiveDipoleCentroid is certainly better, but I also don't think it can be the final solution. For example, if that noise peak far from the SN had been negative, then this would report some flux weighted mean centroid, which I think is still the wrong answer. I think the bottom line is we might need the deblender (or some simplified version thereof), and testing that would be non-trivial work. That's definitely another ticket though, and this fix should be a good improvement in the interim. I agree with Colin Slater's assessment. This needs much further discussion. I should note that for this image, the case of doMerge=True is similar to Colin Slater's example case, but because the centroid is (now) weighted by flux, the resulting centroid is close (this is helped by the fact that the SN is very bright) – (1354.9, 1345.9) for doMerge=False, vs. (1353.0, 1347.0) for doMerge=True. This is not the correct answer, but it is still an improvement.
Michael Wood-Vasey I believe that it would not be trivial to effectively capture this issue in a test. Perhaps worthy of a separate ticket.
I just created DM-7395 as a general ticket to create a test suite.
Michael Wood-Vasey
added a comment - I just created DM-7395 as a general ticket to create a test suite. Would you mind doing a review of this small fix to a bug in ip_diffim? Take a look at the ticket page to see what the specific issue was and let me know if you have any questions. Thanks.
While I'm not very familiar with ip_diffim, the work done on the ticket looks reasonable given the scope of the ticket. Can you clean up the commented out code and rebase off've master? Otherwise it looks good!
Thanks a lot for the review, Vishal Kasliwal [X]. I've made the requested changes and tested the build on Jenkins:
https://ci.lsst.codes/job/stack-os-matrix/15183/label=centos-6,python=py2/console
Merged.
This ticket has two main issues – one regarding detection/measurement and the other regarding decorrelation. I will work on the measurement issue first, and for the decorrelation issue, this data set is a good example which can be used for the analysis in
DM-6162.