Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ip_diffim
-
Labels:
-
Story Points:8
-
Epic Link:
-
Sprint:AP F19-1, AP F19-4 (2H September), AP F19-5 (October), AP F19-6 (November)
-
Team:Alert Production
Description
Yusra AlSayyad identified in the June '19 False Positive Sprint that the variances were wrong by 10-15% (
https://nbviewer.jupyter.org/github/yalsayyad/dm_notebooks/blob/master/examples/DiaSourceCensusCcdVisitNight-HSC-RC2.ipynb). This is likely due to pixel covariances introduced by coaddition and warping. This ticket is to investigate rescaling the variances appropriately with ScaleVariance.
Attachments
Issue Links
Activity
Paul Price: no--running it is the purpose of this ticket.
Adding this to the AP sprint, but we should coordinate with Yusra AlSayyad to see if & when she is planning to work on something similar before we go ahead with it.
The code in this ticket adds doScaleTemplateVariance config option to ImageDifferenceTask. If enabled, ScaleVarianceTask is invoked in ImageDifferenceTask to rescale the template before PSF matching and subtraction. The resulting rescaling factor is saved as metadata entry scaleTemplateVarianceFactor.
Also, full module API document generation is now enabled for ScaleVarianceTask.
Note, this implementation adjusts the template variance at the ImageDifferenceTask level, after the template is created as an Exposure from the coadd patches, covering the same sky area as the science image (plus some margin). This is before the Warper resamples the template onto exactly the same pixel grid as the science Exposure. The resampling may involve scale change. If we want the variance correction after this step, then this configuration option should be introduced independently into both the AL and ZOGY subtasks.
There are logs attached of a HiTS2015 test run (visit=412060, ccd=25), which actually down scales the template variance and increases the number of detections by about 1% here.
Yusra AlSayyad Would you please review this ticket?
Looks good! Will you say a little something about why the template is scaled before matching vs. the diffim scaled before detection? (I'm not saying it's wrong; I can see an argument for either). Would it help if I reran 
?
Also, I noticed its False by default. You planning on doing more testing before setting default to True?
Will you say a little something about why the template is scaled before matching vs. the diffim scaled before detection?
I understood that this was primarily a correction for the template, an empirical workaround for the low variance introduced by coaddition. The template variance may (slightly) affect the image PSF matching itself.
Considering the variance propagation through image differencing, I think we have two issues:
- if we convolve the template (default), then any correlation in the template noise has negligible contribution to the difference image noise as the science image noise is 1-2 orders of magnitude higher and this will dominate the difference image noise
- if we convolve the science image, then the convolution correlates and also reduces the noise level indeed that propagates into the difference image. We have issues to correct this by the decorrelation afterburner at the moment (see
DM-21702), so it may make sense to do empirical correction. I think we can consider a doDiffimVarianceScale option once the Gen3 image differencing has taken shape.
You planning on doing more testing before setting default to True?
I think we can return to this question once the Gen3 image differencing has taken shape.
Is ScaleVarianceTask being run?