Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20558

Investigate rescaling the coadd variances for difference imaging templates

    Details

    • Story Points:
      8
    • Sprint:
      AP F19-1, AP F19-4 (2H September), AP F19-5 (October), AP F19-6 (November)
    • Team:
      Alert Production

      Description

      Yusra AlSayyad identified in the June '19 False Positive Sprint that the variances were wrong by 10-15% (
      https://nbviewer.jupyter.org/github/yalsayyad/dm_notebooks/blob/master/examples/DiaSourceCensusCcdVisitNight-HSC-RC2.ipynb). This is likely due to pixel covariances introduced by coaddition and warping. This ticket is to investigate rescaling the variances appropriately with ScaleVariance.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Is ScaleVarianceTask being run?

            Show
            price Paul Price added a comment - Is ScaleVarianceTask being run?
            Hide
            ebellm Eric Bellm added a comment - - edited

            Paul Price: no--running it is the purpose of this ticket.

            Show
            ebellm Eric Bellm added a comment - - edited Paul Price : no--running it is the purpose of this ticket.
            Hide
            swinbank John Swinbank added a comment -

            Adding this to the AP sprint, but we should coordinate with Yusra AlSayyad to see if & when she is planning to work on something similar before we go ahead with it.

            Show
            swinbank John Swinbank added a comment - Adding this to the AP sprint, but we should coordinate with Yusra AlSayyad to see if & when she is planning to work on something similar before we go ahead with it.
            Hide
            gkovacs Gabor Kovacs added a comment - - edited

            The code in this ticket adds doScaleTemplateVariance config option to ImageDifferenceTask. If enabled, ScaleVarianceTask is invoked in ImageDifferenceTask to rescale the template before PSF matching and subtraction. The resulting rescaling factor is saved as metadata entry scaleTemplateVarianceFactor.

            Also, full module API document generation is now enabled for ScaleVarianceTask.

            Note,  this implementation adjusts the template variance at the ImageDifferenceTask level, after the template is created as an Exposure from the coadd patches, covering the same sky area as the science image (plus some margin). This is before the Warper resamples the template onto exactly the same pixel grid as the science Exposure. The resampling may involve scale change. If we want the variance correction after this step, then this configuration option should be introduced independently into both the AL and ZOGY subtasks.


            There are logs attached of a HiTS2015  test run (visit=412060, ccd=25), which actually down scales the template variance and increases the number of detections by about 1% here.

            Show
            gkovacs Gabor Kovacs added a comment - - edited The code in this ticket adds doScaleTemplateVariance config option to ImageDifferenceTask . If enabled, ScaleVarianceTask is invoked in ImageDifferenceTask to rescale the template before PSF matching and subtraction. The resulting rescaling factor is saved as metadata entry scaleTemplateVarianceFactor . Also, full module API document generation is now enabled for ScaleVarianceTask. Note,  this implementation adjusts the template variance at the ImageDifferenceTask level , after the template is created as an Exposure from the coadd patches, covering the same sky area as the science image (plus some margin). This is before the Warper resamples the template onto exactly the same pixel grid as the science Exposure. The resampling may involve scale change. If we want the variance correction after this step, then this configuration option should be introduced independently into both the AL and ZOGY subtasks. There are logs attached of a HiTS2015  test run (visit=412060, ccd=25), which actually down scales the template variance and increases the number of detections by about 1% here.
            Hide
            gkovacs Gabor Kovacs added a comment - - edited

            Yusra AlSayyad Would you please review this ticket?

            Show
            gkovacs Gabor Kovacs added a comment - - edited Yusra AlSayyad Would you please review this ticket?
            Hide
            yusra Yusra AlSayyad added a comment -

            Looks good! Will you say a little something about why the template is scaled before matching vs. the diffim scaled before detection? (I'm not saying it's wrong; I can see an argument for either). Would it help if I reran ?

            Also, I noticed its False by default. You planning on doing more testing before setting default to True?

            Show
            yusra Yusra AlSayyad added a comment - Looks good! Will you say a little something about why the template is scaled before matching vs. the diffim scaled before detection? (I'm not saying it's wrong; I can see an argument for either). Would it help if I reran ? Also, I noticed its False by default. You planning on doing more testing before setting default to True?
            Hide
            gkovacs Gabor Kovacs added a comment - - edited

            Will you say a little something about why the template is scaled before matching vs. the diffim scaled before detection?

             

            I understood that this was primarily a correction for the template, an empirical workaround for the low variance introduced  by coaddition. The template variance may (slightly) affect the image PSF matching itself.

            Considering the variance propagation through image differencing, I think we have two issues:

            • if we convolve the template (default), then any correlation in the template noise has negligible contribution to the difference image noise as the science image noise is 1-2 orders of magnitude higher and this will dominate the difference image noise
            • if we convolve the science image, then the convolution correlates and also reduces the noise level indeed that propagates into the difference image. We have issues to correct this by the decorrelation afterburner at the moment  (see DM-21702), so it may make sense to do empirical correction. I think we can consider a doDiffimVarianceScale option once the Gen3 image differencing has taken shape.

            You planning on doing more testing before setting default to True?

            I think we can return to this question once the Gen3 image differencing has taken shape.

            Show
            gkovacs Gabor Kovacs added a comment - - edited Will you say a little something about why the template is scaled before matching vs. the diffim scaled before detection?   I understood that this was primarily a correction for the template, an empirical workaround for the low variance introduced  by coaddition. The template variance may (slightly) affect the image PSF matching itself. Considering the variance propagation through image differencing, I think we have two issues: if we convolve the template (default), then any correlation in the template noise has negligible contribution to the difference image noise as the science image noise is 1-2 orders of magnitude higher and this will dominate the difference image noise if we convolve the science image, then the convolution correlates and also reduces the noise level indeed that propagates into the difference image. We have issues to correct this by the decorrelation afterburner at the moment  (see DM-21702 ), so it may make sense to do empirical correction. I think we can consider a doDiffimVarianceScale option once the Gen3 image differencing has taken shape. You planning on doing more testing before setting default to True? I think we can return to this question once the Gen3 image differencing has taken shape.

              People

              • Assignee:
                gkovacs Gabor Kovacs
                Reporter:
                ebellm Eric Bellm
                Reviewers:
                Yusra AlSayyad
                Watchers:
                Eric Bellm, Gabor Kovacs, John Swinbank, Paul Price, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: