Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15675

Make sure data IDs are expanded when adding Datasets and filling templates

    XMLWordPrintable

Details

    Description

      A data ID with (camera, visit, sensor) keys may be enough to uniquely identify a Dataset, but we should expand it to include physical_filter (implied by visit) and abstract_filter (implied by physical_filter), both so we can consistently populate the Dataset table and so we can use the expanded keys in templates.

      Note that we will also need to expand data IDs before transferring them (e.g. via a QuantumGraph after preflight) to a limited Registry, as the limited Registry doesn't have the necessary information to expand them itself.  That implies that the expansion should include attaching the region, because PipelineTask execution may want that region and limited Registry can't attach it either.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            DM-15034 includes a new DataId object that has a place for these extra metadata fields and a Registry API for populating them; this ticket now just involves a Butler-level and Datastore-level logic to find out what a template needs and request it.

            jbosch Jim Bosch added a comment - DM-15034 includes a new DataId object that has a place for these extra metadata fields and a Registry API for populating them; this ticket now just involves a Butler-level and Datastore-level logic to find out what a template needs and request it.
            jbosch Jim Bosch added a comment -

            nlust, as we discussed this is not urgent, so continue to enjoy your holidays, but it should fix the bug with the filename templates that frequently caused outputs to be written to the same file. It's also, I'm afraid, a rather substantial change. I'd recommend reviewing commit-by-commit, to separate the shallow+broad API change commits from the deep+narrow implementation change commits.

            Main PR is for daf_butler, with a totally trivial doc update in obs_base.

            jbosch Jim Bosch added a comment - nlust , as we discussed this is not urgent, so continue to enjoy your holidays, but it should fix the bug with the filename templates that frequently caused outputs to be written to the same file. It's also, I'm afraid, a rather substantial change. I'd recommend reviewing commit-by-commit, to separate the shallow+broad API change commits from the deep+narrow implementation change commits. Main PR is for daf_butler , with a totally trivial doc update in obs_base .
            nlust Nate Lust added a comment -

            Minor things on pr. I mostly followed the logic, but there were a few places I trusted you not knowing the context of the code. All the new code looked sound in its syntax.

            nlust Nate Lust added a comment - Minor things on pr. I mostly followed the logic, but there were a few places I trusted you not knowing the context of the code. All the new code looked sound in its syntax.
            jbosch Jim Bosch added a comment -

            nlust, I've just added a ci_hsc PR with a unit test to check that data IDs are now expanded in gen2convert: https://github.com/lsst/ci_hsc/pull/67.

            jbosch Jim Bosch added a comment - nlust , I've just added a ci_hsc PR with a unit test to check that data IDs are now expanded in gen2convert: https://github.com/lsst/ci_hsc/pull/67 .
            nlust Nate Lust added a comment -

            looks fine to me

            nlust Nate Lust added a comment - looks fine to me

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Nate Lust
              Jim Bosch, Nate Lust
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.