Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Labels:
-
Story Points:6
-
Epic Link:
-
Team:Data Release Production
Description
A data ID with (camera, visit, sensor) keys may be enough to uniquely identify a Dataset, but we should expand it to include physical_filter (implied by visit) and abstract_filter (implied by physical_filter), both so we can consistently populate the Dataset table and so we can use the expanded keys in templates.
Note that we will also need to expand data IDs before transferring them (e.g. via a QuantumGraph after preflight) to a limited Registry, as the limited Registry doesn't have the necessary information to expand them itself. That implies that the expansion should include attaching the region, because PipelineTask execution may want that region and limited Registry can't attach it either.
Attachments
Issue Links
Activity
Nate Lust, as we discussed this is not urgent, so continue to enjoy your holidays, but it should fix the bug with the filename templates that frequently caused outputs to be written to the same file. It's also, I'm afraid, a rather substantial change. I'd recommend reviewing commit-by-commit, to separate the shallow+broad API change commits from the deep+narrow implementation change commits.
Main PR is for daf_butler, with a totally trivial doc update in obs_base.
Minor things on pr. I mostly followed the logic, but there were a few places I trusted you not knowing the context of the code. All the new code looked sound in its syntax.
Nate Lust, I've just added a ci_hsc PR with a unit test to check that data IDs are now expanded in gen2convert: https://github.com/lsst/ci_hsc/pull/67.
DM-15034includes a new DataId object that has a place for these extra metadata fields and a Registry API for populating them; this ticket now just involves a Butler-level and Datastore-level logic to find out what a template needs and request it.