If I convert a DECam Gen 2 repository containing defects, the conversion succeeds, but running ProcessCcd on the result gives an error:
The immediate cause is that one copy of the defects is from the Gen 2 repository, the other copy was ingested during the conversion.
Discussion on #dm-middleware suggests that this is due to conflicting responsibilities:
- obs.base.Instrument has some hard-coded definitions of curated files, which are processed any time ConvertRepoConfig.doWriteCuratedCalibrations is set. Defects are included.
- ConvertRepoConfig also has a curatedCalibrations field, which presumably allows other datasets to be handled using writeCuratedCalibrations. This list does not include defects by default.
- HSC adds defects to ConvertRepoConfig.curatedCalibrations. The bug described above does not occur.
- DECam makes no changes to curatedCalibrations, and its config override file makes no mention of defects. The bug does occur.
The above behavior implies that DECam repositories with defects cannot be converted using the default configs, which goes against our general aim to provide working defaults for all Tasks.
- Do away with ConvertRepoConfig.curatedCalibrations, as proposed by Tim Jenness. However, this does raise the question of how to handle observatory-specific curated calibrations.
- Have DECam explicitly register defects in curatedCalibrations, as HSC does. May be hard to scale to other instruments.
- Duplicate the dataset types hardcoded into Instrument in the default ConvertRepoConfig.curatedCalibrations. Won't benefit configs that overwrite this field instead of appending to it.