Fingerprinting pattern empirical characteristics
Fingerprinting solutions occur on the DECam images when both the
science and template images are sharp, there is little PSF width
difference between the science and the template and the science image
have higher number of sources (several hundreds compared to ~100). As
these effects appear together, we did not separate which effect is
the dominant. This could be better understood if we could generate
images with simulated sources.
The fingerprinting solution is characterized by order of magnitude
higher coefficient values in the higher order basis functions
compared to the 1st basis function (that scales the flux). The stamp
image of the kernel shows oscillations (rings) between positive and
negative values. DM-19443 provides debug plots to see the AL solution
coeffiecients at different parts of the image and their
histograms. Unfortunately, example plots of earlier test runs were
not preserved.
The spatially varying solution also inherits the oscillating nature
everywhere in the image. Also the distribution of the higher order
coefficients are wider and have a high mean across the local
solutions. ip_diffim has debug plots via afwDisplay that
can be interactively visualized ds9.
In the visually good cases, when the fingerprinting pattern is away,
the spatial kernel solution is smooth, Gaussian-like and do not have
outlying high coefficients of higher order basis functions. Test runs
of ap_pipe and lightcurve/flux plotting suggests however that the
flux values may be inaccurate. Fluxes are often around half of their
original values which may simply be a subtraction artefact, not
physical variablity.
Notes
Due to the zero normalization of the basis functions (except for the
first), these components just redistribute flux. Also their
normalization make these basis functions to have a few highly
outlying pixel values (sharp hole or peak in the middle) while their
median pixel values are low. This actually makes it plausible to get
an oscillating solution and high coefficient values.
There was no clear cut algorithm modification in DM-19820 that removed
all the visual artefacts at once on its own.
The deconvolution case use an approximation for the inverse of
Gaussian convolution by a linear combination of 3 Gaussians. We note
that the zero normalization of the basis functions may not be
compatible with the inverse approximation but this needs further
confirmation.
The exact PSF match case is not well handled in the code as in this
case the config file external sigma values are taken for the basis
functions. The close to match PSF sigmas case is limited by the
minimum sigma config values.
Future work ideas
In the light of DM-19660 and the false positives sprint in June 2019,
it may be useful to re-run ap_pipe with DM-19820 development ip_diffim
branches and to generate the current metrics of quality beyond the
occasional visual inspection of the images. Properly document, screenshot
debug displays to preserve the result of these test runs.
Study the spatial distribution of the first coefficient that carries
the flux scaling. Are there any specific pattern around the sources
that are bright but otherwise look normal? This is to examine the visually
good, but questionable difference flux cases.
Per our meeting of 2019-03-26, we agreed that it's appropriate for Gabor Kovacs to spent another couple of weeks of investigation on this ticket. At that point — if the work hasn't converged —, he should summarize his results here and solicit inputs from elsewhere in the project (likely primarily from Robert Lupton).