At our meeting (Eric Bellm, Meredith Rawls, John Swinbank) of 2018-10-30 we agreed that it would be fun to try excluding pixels within N pixels of the edge of a chip, based on the DIASource x, y. We reckon that's probably easy and worth spending ~an hour on in the context of this ticket. If it turns out not to be easy, make a new ticket for it.

Gabor Kovacs, would you be willing to review this?

The ap_pipe rerun is located on lsst-dev in /project/mrawls/hits2015/rerun/newtemplate1. To run ap_pipe, I used slurm scripts created by an updated version of prep_ap_pipe.sh (pushed to ap_pipe for you to review).

The main thing to look at, however, is the new notebook in ap_pipe-notebooks. There are some histograms of the objects and sources detected along with some plots of objects/sources on the sky. Among other things, it shows that, as demonstrated in DM-15080, the new templates are at least not worse than before, and are arguably better.

It's worth noting there are PRs both here and here which Jira is being slow to pick up on due to its now-daily-5pm-crash habit.

As I reported during standup today (11/1), it is non-trivial to exclude plotting sources near the edge of chips as John Swinbank requested above because the xy pixel information is not yet in the association database. Therefore, I've skipped that for now.

Ok, I'll do the review.

If we think that DM-14762-New-Template-Reprocessing.ipynb will be re-used in the future, I'd suggest adding a short summary about what the pipeline feature/configuration differences actually were between the "old" and "new" template coadds; just copy-pasting a few relevant lines from DM-15080. (As I understand, the image selection and psf matching options were different.)

In the figure of the last Section (Define a "mini region" with a few thousand objects for future investigations), the black (low nDiaSources) patterns look quite different on the top right corner chips. Also the horizontal "black lines" are more pronounced in case of the new templates not just at the edges. Do we understand or care about these?

Thank you!

To briefly address your questions: the main difference between the old and new templates is whatever has changed in the stack since the old ones were made (nearly a year ago now). For example, this includes crosstalk correction being implemented in obs_decam. The image selection is slightly different, but both used a best-seeing subset of all the HiTS 2014 visits, and both are psf-matched coadds.

The horizontal black lines are edges and bad columns in the CCDs. These sources are almost certainly in masked pixel regions, but we do not yet have the ability to tell if a source or object falls in a masked region. These plots will look much cleaner when we visualize objects with sources that fall in unmasked/unflagged areas only. It's not entirely clear why the new templates result in more sources per object in these bad regions than the old ones did, but it ultimately shouldn't matter.