Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15675

Make sure data IDs are expanded when adding Datasets and filling templates

    Details

      Description

      A data ID with (camera, visit, sensor) keys may be enough to uniquely identify a Dataset, but we should expand it to include physical_filter (implied by visit) and abstract_filter (implied by physical_filter), both so we can consistently populate the Dataset table and so we can use the expanded keys in templates.

      Note that we will also need to expand data IDs before transferring them (e.g. via a QuantumGraph after preflight) to a limited Registry, as the limited Registry doesn't have the necessary information to expand them itself.  That implies that the expansion should include attaching the region, because PipelineTask execution may want that region and limited Registry can't attach it either.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            DM-15034 includes a new DataId object that has a place for these extra metadata fields and a Registry API for populating them; this ticket now just involves a Butler-level and Datastore-level logic to find out what a template needs and request it.

            Show
            jbosch Jim Bosch added a comment - DM-15034 includes a new DataId object that has a place for these extra metadata fields and a Registry API for populating them; this ticket now just involves a Butler-level and Datastore-level logic to find out what a template needs and request it.
            Hide
            jbosch Jim Bosch added a comment -

            Nate Lust, as we discussed this is not urgent, so continue to enjoy your holidays, but it should fix the bug with the filename templates that frequently caused outputs to be written to the same file. It's also, I'm afraid, a rather substantial change. I'd recommend reviewing commit-by-commit, to separate the shallow+broad API change commits from the deep+narrow implementation change commits.

            Main PR is for daf_butler, with a totally trivial doc update in obs_base.

            Show
            jbosch Jim Bosch added a comment - Nate Lust , as we discussed this is not urgent, so continue to enjoy your holidays, but it should fix the bug with the filename templates that frequently caused outputs to be written to the same file. It's also, I'm afraid, a rather substantial change. I'd recommend reviewing commit-by-commit, to separate the shallow+broad API change commits from the deep+narrow implementation change commits. Main PR is for daf_butler , with a totally trivial doc update in obs_base .
            Hide
            nlust Nate Lust added a comment -

            Minor things on pr. I mostly followed the logic, but there were a few places I trusted you not knowing the context of the code. All the new code looked sound in its syntax.

            Show
            nlust Nate Lust added a comment - Minor things on pr. I mostly followed the logic, but there were a few places I trusted you not knowing the context of the code. All the new code looked sound in its syntax.
            Hide
            jbosch Jim Bosch added a comment -

            Nate Lust, I've just added a ci_hsc PR with a unit test to check that data IDs are now expanded in gen2convert: https://github.com/lsst/ci_hsc/pull/67.

            Show
            jbosch Jim Bosch added a comment - Nate Lust , I've just added a ci_hsc PR with a unit test to check that data IDs are now expanded in gen2convert: https://github.com/lsst/ci_hsc/pull/67 .
            Hide
            nlust Nate Lust added a comment -

            looks fine to me

            Show
            nlust Nate Lust added a comment - looks fine to me

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Nate Lust
                Watchers:
                Jim Bosch, Nate Lust
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: