Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21809

Check the variance plane operations and the decorrelation afterburner operations in the science convolution case

    Details

    • Story Points:
      4
    • Sprint:
      AP F19-5 (October), AP F19-6 (November)
    • Team:
      Alert Production

      Description

      In connection to DM-21702, it seems that the main reason of the exploding number of sources in the science convolution case is a lower variance level in the difference images.

      • Check the operations that result this lower value.
      • Check what operations are performed in the decorrelation step.

        Attachments

          Issue Links

            Activity

            Hide
            gkovacs Gabor Kovacs added a comment -

            Variance plane operations.

            The relevant definition are in include/lsst/afw/image/Pixel.h:375 and Pixel.h:391 in variance_multiplies and variance_plus template structures. For multiplication of two pixels the variance is cross-weighted: x2 * var_y + y2 * var_x
            See also DM-16544.

            The convolution kernel does not have any variance, but pixel variances are weighted by kernel^2^ values. In effect the variance plane is convolved by the square of the AL kernel solution. Assuming independent noise in the pixels, this is what expected. This means that convolution by a 10x10 pixel 1-sum kernel can modify the variance by a factor of 0.01 in the most extreme case.

            The decorrelation afterburner

            While swapping the two images in image differencing seems straightforward, regarding the variance plane of the difference image, the cases are asymmetric. If we convolve the negligible noise template, then the difference image inherits all the noise from the science image variance plane. If we convolve the science image, the variance in the science image decreases significantly due to the convolution. The difference image will have a lower level (and correlated) noise. It is then the task of the decorrelation afterburner to deconvolve and scale back the variance of the difference image. This functionality does not perform well at the moment.

            • The decorrelation afterburner calculates a weighted inverse of the Fourier components of the convolution kernel to deconvolve the noise in an optimal way. In the limiting case of noiseless template, this reduces to the simple reciprocal Fourier components.
            • In the decorrelation afterburner, the role of the two images are not symmetric, it assumes that the second image is matched to the first and subtracted from the first. This is addressed in this ticket branch. Swapping the input images breaks the science image preconvolution support in the decorrelator task, we disallow this configuration combination.

            Conclusions

            We note that applying the fix in this ticket, the decorrelated images are still bad quality:

            • The variance plane is overcorrected by orders of magnitude. See the variance plane scale in the screenshot attached.
            • Difference image is full of ringing artefacts. The implementation and the theoretical assumptions need further investigation.

            For follow up investigation, DM-21868 is created.

            Show
            gkovacs Gabor Kovacs added a comment - Variance plane operations. The relevant definition are in include/lsst/afw/image/Pixel.h:375 and Pixel.h:391 in variance_multiplies and variance_plus template structures. For multiplication of two pixels the variance is cross-weighted: x2 * var_y + y2 * var_x See also DM-16544 . The convolution kernel does not have any variance, but pixel variances are weighted by kernel^2^ values. In effect the variance plane is convolved by the square of the AL kernel solution. Assuming independent noise in the pixels, this is what expected. This means that convolution by a 10x10 pixel 1-sum kernel can modify the variance by a factor of 0.01 in the most extreme case. The decorrelation afterburner While swapping the two images in image differencing seems straightforward, regarding the variance plane of the difference image, the cases are asymmetric. If we convolve the negligible noise template, then the difference image inherits all the noise from the science image variance plane. If we convolve the science image, the variance in the science image decreases significantly due to the convolution. The difference image will have a lower level (and correlated) noise. It is then the task of the decorrelation afterburner to deconvolve and scale back the variance of the difference image. This functionality does not perform well at the moment. The decorrelation afterburner calculates a weighted inverse of the Fourier components of the convolution kernel to deconvolve the noise in an optimal way. In the limiting case of noiseless template, this reduces to the simple reciprocal Fourier components. In the decorrelation afterburner, the role of the two images are not symmetric, it assumes that the second image is matched to the first and subtracted from the first. This is addressed in this ticket branch. Swapping the input images breaks the science image preconvolution support in the decorrelator task, we disallow this configuration combination. Conclusions We note that applying the fix in this ticket, the decorrelated images are still bad quality: The variance plane is overcorrected by orders of magnitude. See the variance plane scale in the screenshot attached. Difference image is full of ringing artefacts. The implementation and the theoretical assumptions need further investigation. For follow up investigation, DM-21868 is created.
            Hide
            gkovacs Gabor Kovacs added a comment -

            Could you please have a look at this ticket ? Beside the comments, there is a code for fixing the usage of the decorrelation afterburner; though it does not really rectify anything at the moment due to DM-21868, the decorrelation afterburner itself needs fixing. Shall we merge?

            Show
            gkovacs Gabor Kovacs added a comment - Could you please have a look at this ticket ? Beside the comments, there is a code for fixing the usage of the decorrelation afterburner; though it does not really rectify anything at the moment due to DM-21868 , the decorrelation afterburner itself needs fixing. Shall we merge?
            Hide
            ebellm Eric Bellm added a comment -

            Looks good.

            Show
            ebellm Eric Bellm added a comment - Looks good.

              People

              • Assignee:
                gkovacs Gabor Kovacs
                Reporter:
                gkovacs Gabor Kovacs
                Reviewers:
                Eric Bellm
                Watchers:
                Eric Bellm, Gabor Kovacs
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel