Details
-
Type:
Story
-
Status: Invalid
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: meas_algorithms, meas_photocal, obs_cfht, obs_sdss, obs_subaru, obs_test, pipe_tasks
-
Labels:None
-
Team:Data Access and Database
Description
There are some configuration issues associated with reference catalogs, including:
1) Determining the source of the reference catalog. This information is used in two ways:
- mapping the filter used to take an exposure to a filter in a reference catalog, for matching sources to reference objects (e.g. for astrometry and photometry)
- picking the correct set of colorterms based on the filter and type of CCD used for the science image and filters in the reference catalog
At present this information is encoded in the eups version name of the astrometry_net_data package used. This requires the use of eups and assumes astrometry.net index files. This is the most serious immediate issue.
2) Colorterm correction data is saved in obs_* packages as config files. This makes it harder to include a temporal component.
This ticket is a request for a cleaner solution.
The plan has been to wait for "the new butler", e.g. DM-2404 (taking this use case into account in its design) before designing the solution. However, if we need something sooner, one possibility for fixing the first problem is to put the source of the reference catalog into metadata attached to the reference catalog.
Attachments
Issue Links
Activity
While RFC-624 doesn't solve this problem, we should make sure it is compatible with potential future solutions to this problem (which in turn should likely be the subject of Community and/or RFC discussion).
Can someone tell me whether this ticket is still relevant and also whether it needs new butler functionality as implied in the description.
The issue identifies a still-existing problem - the colorterms system is fragile - but it's a low-priority one to me because I think what we have now is mostly relevant for precursor datasets; for LSST DRP, where FGCM will have the final say, and for LSST AP, where I expect us to use a DRP-generated reference catalog, the role of colorterms in the future is unclear, and certainly diminished.
I also don't think I see any middleware work to do here anymore - I think the main limitation was the difficulty of adding new dataset types in the Gen2. I think in Gen3 it'd be quite natural to put the colorterms data in butler datasets with instrument+physical_filter dimension and a dataset type name based on the corresponding reference catalog name. There would still be fragility in making sure the reference catalog and colorterms were loaded consistently, but that's something that pipeline contracts would help with.
Closing this since it seems from the discussion that it's all under control.
I agree that we need to think a bit harder about what is being requested.
My guess is that what the pipelines want to say is
Somewhere on the pipeline side could resolve "colorterms for using PanSTARRS photometry version 2.3 with the LSST camera" to a filename, but then we'd need to add mapper entries for N catalogues to each obs_camera. That would be a possibility. Another would be for the pipelines to define the template for the N*M possibilities, then pass that to the butler with appropriate dataId entries to resolve to a particular file (modulo date – I think we all agree that this is a butler issue).