I generated nine sets of eight simulated images of the same field, each with the same sources and airmass range but a different range of seeing. The range of seeing in the images used to construct the DcrModel started at zero (constant seeing), and gradually increased to nearly a factor of two. The seeing range was drawn from the same OpSim simulated observations, but the seeing values were scaled to force the maximum seeing to be the given percent greater than the minimum. The DcrModel and compareWarp Coadd were both constructed using all eight of the images, with no additional cuts on seeing.

To test the DcrModel and compareWarp templates, an additional set of 24 observations were simulated for the same sources, with airmass and seeing taken directly from the OpSim simulated observations. The same 24 "science" images were used for image differencing for each of the nine "year 1" simulations using the DCR-matched template and the compareWarp Coadd as the template.

In the attached plots, the arrow starts at the number of dipoles detected in the residual image using the compareWarp template, and ends at the number of dipoles detected using the DCR-matched template. Each arrow is color-coded by the seeing of the "science" image, and there is no obvious correlation between the number of dipoles and the seeing of the science image, demonstrating that PSF matching in image differencing is working well.

Below is the average number of dipoles detected over the 24 alert images, for each range of seeing allowed in the template images.

Seeing range deep dcr
0%: 16.96 2.57
10%: 17.43 2.61
21%: 17.17 3.30
33%: 19.04 4.52
46%: 19.13 4.22
61%: 19.35 4.17
77%: 20.48 4.13
95%: 23.17 4.52

Please take a look at the plots and comment posted above. I think this addresses the question posed in this ticket, but I could see repeating the analysis for different simulated fields (and a different range of airmasses contributing to the template), or with fewer observations of the field in year 1.

https://github.com/lsst-dm/experimental_DCR/blob/tickets/DM-18709/DM-18709%20image%20difference%20source%20detection.ipynb

Hi Ian Sullivan, a few small requests that I think should be pretty straightforward:

Bookkeeping:

• In your table, can you also report the number of detected sources in the difference (in addition to the number of dipoles), for both CompareWarp and dcr differences?
• Can you report how many sources would be expected from Gaussian noise in these simulations? (see https://dmtn-006.lsst.io/#noise-in-difference-images)

A few plot things:

• Can you swap the x and y axes in your plots? I think of airmass as the independent variable.
• I'd also be interested in a more compact version of the plot where you could overlay several different cases--I think maybe taking the fractional change in the number of dipoles per image (rather than drawing an arrow) would allow you to make a plot with day 0%, 33%, 77%, and 95% seeing variations all overlayed, which would make a useful presentation figure.

 
 
The mean number of false positives due to noise should be ~0.1 in each of these simulations, following the calculation in DMTN-006. Below I've switched to reporting the median number of dipoles or DIA sources instead of the mean to reject the cluster of science images with very good seeing that have a large number of dipoles in all of the plots. These images have narrower PSFs than the coadd PSF, and the number of dipoles and DIA sources for these images drop to the levels in the other images if I convolve the science image with the PSF-matching kernel instead of the template.

Median number of dipoles detected in the image differences

Median number of DIA sources detected in the image differences

Unfortunately, the PSF of the DCR-based template has complex structure when it is built using a wide range of seeing values, and this is not well captured by the PSF matching kernel. As a result there are bright oscillating residuals at the location of many sources, and these are not detected. Based on the sharp increase in the number of DIA sources above 46% seeing range, I would suggest that be a reasonable upper limit for either algorithm currently.

Because I found the number of dipoles or the number of DIA sources to be unreliable for templates built with a wide variation in seeing, I attempted to come up with a better metric to capture the quality of the subtraction. I took a single science image mask as a reference, and evaluated the mean of the absolute value of the pixels identified as detected in the reference mask for every difference image. My hope was that this would clearly show the residuals getting worse as the seeing range increased and the template got worse. The median of that residual across all of the difference images for each seeing range is tabulated below, but unfortunately this metric appears to get better despite the residuals clearly getting worse by eye. 

Median residual in the difference images

Calculated using the absolute value of the residual within the footprints of sources detected in the science image.

Agreed per meeting of 2019-04-26 that Ian Sullivan will make the figure that Eric Bellm requested above, then close this ticket as “done”. Future investigations will go on new tickets.

Here is the figure showing the fractional reduction in the number of dipoles for several of the seeing ranges. Note that some of the constant seeing observations ("0%") show a fractional reduction of 0. This is because those observations had only 0 or 1 dipole in the standard image difference, and no fewer in the DCR image difference.

To my eye this is still a very busy plot, and I find it hard to pick out trends. This may be mostly due to the issue described above, where the metric (in this case, the number of dipoles) is not accurately describing the quality of the image difference. I welcome suggestions on how to improve this figure!

Agreed it's a busy plot, but I still think informative. Thanks!