As before, I processed 30+ full-field (103 CCD) visits in each of HSC-G, HSC-R2, HSC-I2, HSC-Z, and HSC-Y.
Last time, there were lots of database lock errors and background noise sources. This time, I used PostgreSQL instead of sqlite and implemented difference image variance scaling. The results were generally very similar, but with no database lock errors. The difference image variance scaling likely removed a handful of background noise sources but the difference is not stark.
The new notebook
DM-25990-HSC-ap_pipe-take2.ipynb explores the APDB from this rerun. Some outstanding issues are listed in the notebook and also written here for easy reference.
- There is a new APDB issue on DM-26700. It thinks there are duplicate DIA Objects or DIA Forced Sources being written, and fails.
- As before, there are many failures to retrieve a template which register as "No coadd PhotoCalib found!" This is being tracked on DM-25988, along with some ancillary issues below.
- As before, some PSFs are erroneously large. However, now that the database lock error is fixed, this is more prevalent, to the point that slurm log messages are too big and crash the whole job. This is likely what is causing fewer visits to be processed than my input visit lists suggest.
- Valid HSC visit numbers are even, so an easy way to get rid of some of the wonky ones (especially prevalent in g and y) should be to delete the odd visit numbers. However, I revisited my input visit lists, and they don't have any odd numbers, so something odd (pun intended) is happening behind the scenes.
- The difference image variance scaling didn't make a big difference at a glance. The plethora of sources is at least partly due to the fact that HSC images are deeper than DECam images, both in terms of telescope/optics/detector as well as exposure times. I am also not at all convinced that my templates are particularly great, though, and a bunch of ~0 `totFlux` measurements is a bit of a red flag.