Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17825

Examine bad subtractions

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_diffim
    • Labels:
      None
    • Story Points:
      15
    • Epic Link:
    • Sprint:
      AP S19-3, AP S19-4, AP S19-5, AP S19-6, AP F19-1, AP F19-2
    • Team:
      Alert Production

      Description

      Meredith Rawls has identified some bad subtractions while running ap_pipe. She will provide a short list (a few DIASources) on this ticket together with instructions to reproduce. For each one:

      • Reproduce the problem;
      • Dive into the ip_diffim code and see if you can identify what caused the problem;
      • If it's fixable, file a ticket to fix it.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Per our meeting of 2019-03-26, we agreed that it's appropriate for Gabor Kovacs to spent another couple of weeks of investigation on this ticket. At that point — if the work hasn't converged —, he should summarize his results here and solicit inputs from elsewhere in the project (likely primarily from Robert Lupton).

            Show
            swinbank John Swinbank added a comment - Per our meeting of 2019-03-26, we agreed that it's appropriate for Gabor Kovacs to spent another couple of weeks of investigation on this ticket. At that point — if the work hasn't converged —, he should summarize his results here and solicit inputs from elsewhere in the project (likely primarily from Robert Lupton).
            Hide
            swinbank John Swinbank added a comment -

            Gabor Kovacs — it has now been a lot longer since the “couple of weeks” we mentioned on 26 March, and I know you have taken a lot of action (including talking to Robert).

            Could you please update this ticket with a note of what actions you've taken, what you've discovered, and what the plan is for making progress. Then I think we can close out this ticket as mission accomplished (certainly, these bad subtractions have been examined in detail!!).

            Show
            swinbank John Swinbank added a comment - Gabor Kovacs — it has now been a lot longer since the “couple of weeks” we mentioned on 26 March, and I know you have taken a lot of action (including talking to Robert). Could you please update this ticket with a note of what actions you've taken, what you've discovered, and what the plan is for making progress. Then I think we can close out this ticket as mission accomplished (certainly, these bad subtractions have been examined in detail!!).
            Hide
            gkovacs Gabor Kovacs added a comment - - edited

            Fingerprinting pattern empirical characteristics

            Fingerprinting solutions occur on the DECam images when both the
            science and template images are sharp, there is little PSF width
            difference between the science and the template and the science image
            have higher number of sources (several hundreds compared to ~100). As
            these effects appear together, we did not separate which effect is
            the dominant. This could be better understood if we could generate
            images with simulated sources.

            The fingerprinting solution is characterized by order of magnitude
            higher coefficient values in the higher order basis functions
            compared to the 1st basis function (that scales the flux). The stamp
            image of the kernel shows oscillations (rings) between positive and
            negative values. DM-19443 provides debug plots to see the AL solution
            coeffiecients at different parts of the image and their
            histograms. Unfortunately, example plots of earlier test runs were
            not preserved.

            The spatially varying solution also inherits the oscillating nature
            everywhere in the image. Also the distribution of the higher order
            coefficients are wider and have a high mean across the local
            solutions. ip_diffim has debug plots via afwDisplay that
            can be interactively visualized ds9.

            In the visually good cases, when the fingerprinting pattern is away,
            the spatial kernel solution is smooth, Gaussian-like and do not have
            outlying high coefficients of higher order basis functions. Test runs
            of ap_pipe and lightcurve/flux plotting suggests however that the
            flux values may be inaccurate. Fluxes are often around half of their
            original values which may simply be a subtraction artefact, not
            physical variablity.

            Notes

            Due to the zero normalization of the basis functions (except for the
            first), these components just redistribute flux. Also their
            normalization make these basis functions to have a few highly
            outlying pixel values (sharp hole or peak in the middle) while their
            median pixel values are low. This actually makes it plausible to get
            an oscillating solution and high coefficient values.

            There was no clear cut algorithm modification in DM-19820 that removed
            all the visual artefacts at once on its own.

            The deconvolution case use an approximation for the inverse of
            Gaussian convolution by a linear combination of 3 Gaussians. We note
            that the zero normalization of the basis functions may not be
            compatible with the inverse approximation but this needs further
            confirmation.

            The exact PSF match case is not well handled in the code as in this
            case the config file external sigma values are taken for the basis
            functions. The close to match PSF sigmas case is limited by the
            minimum sigma config values.

            Future work ideas

            In the light of DM-19660 and the false positives sprint in June 2019,
            it may be useful to re-run ap_pipe with DM-19820 development ip_diffim
            branches and to generate the current metrics of quality beyond the
            occasional visual inspection of the images. Properly document, screenshot
            debug displays to preserve the result of these test runs.

            Study the spatial distribution of the first coefficient that carries
            the flux scaling. Are there any specific pattern around the sources
            that are bright but otherwise look normal? This is to examine the visually
            good, but questionable difference flux cases.

            Show
            gkovacs Gabor Kovacs added a comment - - edited Fingerprinting pattern empirical characteristics Fingerprinting solutions occur on the DECam images when both the science and template images are sharp, there is little PSF width difference between the science and the template and the science image have higher number of sources (several hundreds compared to ~100). As these effects appear together, we did not separate which effect is the dominant. This could be better understood if we could generate images with simulated sources. The fingerprinting solution is characterized by order of magnitude higher coefficient values in the higher order basis functions compared to the 1st basis function (that scales the flux). The stamp image of the kernel shows oscillations (rings) between positive and negative values. DM-19443 provides debug plots to see the AL solution coeffiecients at different parts of the image and their histograms. Unfortunately, example plots of earlier test runs were not preserved. The spatially varying solution also inherits the oscillating nature everywhere in the image. Also the distribution of the higher order coefficients are wider and have a high mean across the local solutions. ip_diffim has debug plots via afwDisplay that can be interactively visualized ds9. In the visually good cases, when the fingerprinting pattern is away, the spatial kernel solution is smooth, Gaussian-like and do not have outlying high coefficients of higher order basis functions. Test runs of ap_pipe and lightcurve/flux plotting suggests however that the flux values may be inaccurate. Fluxes are often around half of their original values which may simply be a subtraction artefact, not physical variablity. Notes Due to the zero normalization of the basis functions (except for the first), these components just redistribute flux. Also their normalization make these basis functions to have a few highly outlying pixel values (sharp hole or peak in the middle) while their median pixel values are low. This actually makes it plausible to get an oscillating solution and high coefficient values. There was no clear cut algorithm modification in DM-19820 that removed all the visual artefacts at once on its own. The deconvolution case use an approximation for the inverse of Gaussian convolution by a linear combination of 3 Gaussians. We note that the zero normalization of the basis functions may not be compatible with the inverse approximation but this needs further confirmation. The exact PSF match case is not well handled in the code as in this case the config file external sigma values are taken for the basis functions. The close to match PSF sigmas case is limited by the minimum sigma config values. Future work ideas In the light of DM-19660 and the false positives sprint in June 2019, it may be useful to re-run ap_pipe with DM-19820 development ip_diffim branches and to generate the current metrics of quality beyond the occasional visual inspection of the images. Properly document, screenshot debug displays to preserve the result of these test runs. Study the spatial distribution of the first coefficient that carries the flux scaling. Are there any specific pattern around the sources that are bright but otherwise look normal? This is to examine the visually good, but questionable difference flux cases.
            Hide
            gkovacs Gabor Kovacs added a comment -

            This ticket only has closing comments.

            Show
            gkovacs Gabor Kovacs added a comment - This ticket only has closing comments.
            Hide
            swinbank John Swinbank added a comment -

            Just to be clear on the terminology — you use the term “fingerprinting solution” to describe a kernel which oscillates, because it looks something like rings on a fingerprint. Is that correct?

            Also — I don't think it is clear from the above, but I believe you see these fingerprints in both the deconvolution and non-deconvolution case. Is that right?

            Finally, in DM-20730 you wrote a short report in an attempt to get some feedback from Robert Lupton. Did that get anywhere? I don't think we've captured his response (if any).

            Otherwise, looks great — thank you!

            Show
            swinbank John Swinbank added a comment - Just to be clear on the terminology — you use the term “fingerprinting solution” to describe a kernel which oscillates, because it looks something like rings on a fingerprint. Is that correct? Also — I don't think it is clear from the above, but I believe you see these fingerprints in both the deconvolution and non-deconvolution case. Is that right? Finally, in DM-20730 you wrote a short report in an attempt to get some feedback from Robert Lupton . Did that get anywhere? I don't think we've captured his response (if any). Otherwise, looks great — thank you!
            Hide
            gkovacs Gabor Kovacs added a comment -

            Just to be clear on the terminology — you use the term “fingerprinting solution” to describe a kernel which oscillates, because it looks something like rings on a fingerprint. Is that correct?

            "Fingerprinting" originally comes from the visual pattern in the image differences themselves. And yes, on decam images, we observed this fingerprinting in the image differences in coincidence with fingerprinting (oscillating) kernel solutions.

            It is true that in first approximation, we expect that a Gaussian like PSF should be convolved by a Gaussian like smooth kernel to match the PSFs. I don't think however, that all "good" kernel solutions must be necessarily non-oscillating. As far as I recall, in one of the test debug cases with simulated and slightly shifted Gaussian sources, the kernel solutions can have some rings while the image difference still preserves the Gaussian PSF shape. Also, if we consider the deconvolution case: in Fourier space, the inversion of the image space Gaussian convolution is division by a Gaussian like image in frequency space that will contain close to zero high
            frequency values (and may not be practically carried out if values are zeroes – this is the much blurring case: if the Gaussian is wide in image space and sharp in frequency space).

            Also — I don't think it is clear from the above, but I believe you see these fingerprints in both the deconvolution and non-deconvolution case. Is that right?

            Yes. While we just briefly focused on the deconvolution case, the patterns were around in all the images seen in case of deconvolution.

            Finally, in DM-20730 you wrote a short report in an attempt to get some feedback from Robert Lupton. Did that get anywhere? I don't think we've captured his response (if any).

            I'm afraid there was no follow up discussion on the report.

            Show
            gkovacs Gabor Kovacs added a comment - Just to be clear on the terminology — you use the term “fingerprinting solution” to describe a kernel which oscillates, because it looks something like rings on a fingerprint. Is that correct? "Fingerprinting" originally comes from the visual pattern in the image differences themselves. And yes, on decam images, we observed this fingerprinting in the image differences in coincidence with fingerprinting (oscillating) kernel solutions. It is true that in first approximation, we expect that a Gaussian like PSF should be convolved by a Gaussian like smooth kernel to match the PSFs. I don't think however, that all "good" kernel solutions must be necessarily non-oscillating. As far as I recall, in one of the test debug cases with simulated and slightly shifted Gaussian sources, the kernel solutions can have some rings while the image difference still preserves the Gaussian PSF shape. Also, if we consider the deconvolution case: in Fourier space, the inversion of the image space Gaussian convolution is division by a Gaussian like image in frequency space that will contain close to zero high frequency values (and may not be practically carried out if values are zeroes – this is the much blurring case: if the Gaussian is wide in image space and sharp in frequency space). Also — I don't think it is clear from the above, but I believe you see these fingerprints in both the deconvolution and non-deconvolution case. Is that right? Yes. While we just briefly focused on the deconvolution case, the patterns were around in all the images seen in case of deconvolution. Finally, in DM-20730 you wrote a short report in an attempt to get some feedback from Robert Lupton. Did that get anywhere? I don't think we've captured his response (if any). I'm afraid there was no follow up discussion on the report.

              People

              • Assignee:
                gkovacs Gabor Kovacs
                Reporter:
                swinbank John Swinbank
                Reviewers:
                John Swinbank
                Watchers:
                Gabor Kovacs, John Swinbank
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel