Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14821

Provide packed integer versions of Gen3 data IDs

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      We frequently need to encode at least (camera, visit, sensor), (skymap, tract, patch), and (skymap, tract, patch, abstract_filter) data IDS into unique reversible integer IDs.  Gen3 needs to provide methods for translating both from dict to int and from int to dict.  Because we'd like to be able to further pack these with Source IDs, we also need to be able to report how many bits are utilized by the data IDs.  This is tricky because the number of bits occupied by any particular data ID key-value pair depends on the particular Camera or SkyMap with which it is associated.

      Ideally, at least the dict->int conversion would be handled via a custom, dict-like DataId class that makes the packed integer ID available as a property or method; we'd like to have something we can pass to Task.run methods that shouldn't require a butler but may want to able to report data ID information in either form for diagnostic purposes.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            As promised, here's the support for packing data IDs into integers.

            To use it from a PipelineTask, you should be able to something like:

            packer = dataId.packers["TractPatch"]
            packedIntegerId = packer.pack(dataId)
            maxBits = packer.maxBits
            

            You can see the names that can be used (at present) in the packers dict at the bottom of config/registry.yaml in daf_butler.

            As for the review, most changes are the infrastructure in daf_butler, which also includes the implementation for visit/exposure+detector packing. When looking at that please note the message on 77e7103dbe7e1321efac59c1b5fad575658203fe; the logic in data ID construction is already sufficiently hard to follow that I really want to fix it soon, but I do nevertheless want to defer that to another ticket.

            There are also much smaller changes to:

            • obs_subaru: just adds some now-necessary metadata to Instrument registration
            • skymap: new metadata in SkyMap registration and packer implementations for SkyMap-based data IDs.
            • ci_hsc: a new test.
            Show
            jbosch Jim Bosch added a comment - As promised, here's the support for packing data IDs into integers. To use it from a PipelineTask, you should be able to something like: packer = dataId.packers["TractPatch"] packedIntegerId = packer.pack(dataId) maxBits = packer.maxBits You can see the names that can be used (at present) in the packers dict at the bottom of config/registry.yaml in daf_butler. As for the review, most changes are the infrastructure in daf_butler , which also includes the implementation for visit/exposure+detector packing. When looking at that please note the message on 77e7103dbe7e1321efac59c1b5fad575658203fe ; the logic in data ID construction is already sufficiently hard to follow that I really want to fix it soon, but I do nevertheless want to defer that to another ticket. There are also much smaller changes to: obs_subaru : just adds some now-necessary metadata to Instrument registration skymap : new metadata in SkyMap registration and packer implementations for SkyMap-based data IDs. ci_hsc : a new test.
            Hide
            nlust Nate Lust added a comment -

            Some comments up, and some changes we talked about to the registry would be desireable

            Show
            nlust Nate Lust added a comment - Some comments up, and some changes we talked about to the registry would be desireable

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Nate Lust
              Watchers:
              Jim Bosch, Nate Lust
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.