Fix Version/s: None
Team:Data Release Production
For many, many years we have needed a way to consistently select stars for consistent reserving of psf stars for overlapping observations (and possible cross-band as well). At some point it was thought that the output from fgcmcal would be suitable for this, as it does the global star matching. However, the order of operations has made that difficult to implement in a useful way for psf star selection.
This ticket is to add a task to match all the (relatively) isolated stars with observations above a configurable s/n ratio and output in a useful format that can be ingested by both the new psf builder (which will be run after calibrate and matching), and by the fgcmcal. At its core this will be a port of the matching done within fgcmcal, but done in a more parallelizable, memory-aware, and butler-compatible way.
After some discussion with Jim Bosch I have decided that the task will:
- Take sourceTable_visit catalogs as input. Some quick benchmarking shows that if you have at least ~5 detectors overlapping a region it is faster to read in the full sourceTable_visit (with 100 detectors of data!) than it is to read in 5 individual sourceTable files.
- At first this task will run per-tract to be compatible with existing repos. The ability to run per-healpix (with different resolutions!) should be easy to add when the middleware supports that.
- The task will run on all bands for a consistent set of matched stars.
- Will select stars above a configurable s/n limit in each observation.
- The output will consist of 3 files per quantum.
- There will be a list of unique ids, ra/dec positions, and pointers to the index table (location and number of observations).
- The index table will have lists of indices to the observation table.
- The observation table will have ra_obs, dec_obs, visit, detector, and several other configurable rows from the source tables, including the sourceId (default will be those rows necessary for input to fgcmcal and jointcal). The fraction of star observations with sufficient s/n at high latitude is ~5% of the total of a catalog, and this is a small subset of rows, so the storage overhead should be minimal but the convenience for the known consumers of these catalogs will be great.
TBD is performance in low latitude high stellar density regions.