Now that DM-13451 is merged, repository issues require changes to ap_verify, not ap_pipe.

Meredith Rawls found a lot of extra info in the registry that seems to relate different datasets to each other. A good regression test for this issue would try to query this relationship info in addition to filters.

The bug appears to be the result of too-simplistic handling of the data ID in ApPipeTask and ap_verify, rather than anything to do with the registry. That explains why it doesn't appear when running ImageDifferenceTask directly from the command line (at least, on HSC data). Many thanks to Yusra AlSayyad for her patience with my questions!

Further investigation shows that, in fact, ImageDifferenceTask cannot handle DECam data if the user doesn't provide a filter. Since it's not clear how or even whether to fix this behavior in ImageDifferenceTask and/or obs_decam (see below), and I can't think of a workaround in ap_verify that wouldn't require special-case code, I'm marking this issue as Won't Fix and opening DM-14359 to address the data ID handling problems I found in the meantime.

For the record, the process by which image differencing runs without an explicit filter fail on DECam is the following:

• ImageDifferenceTask's argument parser interprets data IDs as references to calexps.
• obs_decam gives calexps a URI that depends only on the visit and ccd.
• Based on parsing the mapping information, obs_base considers the "keys" for calexp data IDs to be visit and ccd.
• The Butler defines complete dataIds as those that have values for all their keys.
• Therefore, data IDs for DECam calexps never have a filter added, even though this information can be extracted from the repository. (Note that HSC's mapper does include the filter in a calexp's URI, so by the preceding logic a complete HSC calexp dataref has an explicit filter.)
• ip_diffim tries to look up templates (in my case, a deepCoaddPsfMatched) based on the calexp dataref, supplemented with tract and patch IDs.
• obs_base identifies coadds using tract, patch, and filter. The Butler tries to look up the missing filter information (based on all available information, i.e. visit, ccd, patch, and tract), but there are no tables listed in which to do the lookup. The information is provided in raw, but coadds are not, themselves, raw data, and not all lookups for coadds are guaranteed to have a visit in their dataId.

This is a contract mismatch problem: each step makes sense in isolation, but the pieces don't work together. Since there's no single piece that's obviously wrong, and changing any of the behaviors I listed above will almost certainly have side effects, I defer the question of what, if anything, can be done about it to people more familiar with the framework than I.

FWIW, I believe this is the same as or related to DM-9148.

I'm not convinced it's the same (my stack trace does not contain CameraMapper._setFilter, and the registry does have all necessary info), but it's similar enough to note.

The intent of the Gen2 Butler design was to eventually be able to determine that filter can be derived from visit in the dataId using (e.g.) the raw table.  I'm not sure that the current policy is sufficient to accomplish this for coadds; it might involve a combination of adding "tables: raw" and something for the apparently-undocumented "columns:" if it is indeed possible.