Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14821

Provide packed integer versions of Gen3 data IDs

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      We frequently need to encode at least (camera, visit, sensor), (skymap, tract, patch), and (skymap, tract, patch, abstract_filter) data IDS into unique reversible integer IDs.  Gen3 needs to provide methods for translating both from dict to int and from int to dict.  Because we'd like to be able to further pack these with Source IDs, we also need to be able to report how many bits are utilized by the data IDs.  This is tricky because the number of bits occupied by any particular data ID key-value pair depends on the particular Camera or SkyMap with which it is associated.

      Ideally, at least the dict->int conversion would be handled via a custom, dict-like DataId class that makes the packed integer ID available as a property or method; we'd like to have something we can pass to Task.run methods that shouldn't require a butler but may want to able to report data ID information in either form for diagnostic purposes.

        Attachments

          Issue Links

            Activity

            Hide
            nlust Nate Lust added a comment -

            Some comments up, and some changes we talked about to the registry would be desireable

            Show
            nlust Nate Lust added a comment - Some comments up, and some changes we talked about to the registry would be desireable
            Hide
            jbosch Jim Bosch added a comment -

            As promised, here's the support for packing data IDs into integers.

            To use it from a PipelineTask, you should be able to something like:

            packer = dataId.packers["TractPatch"]
            packedIntegerId = packer.pack(dataId)
            maxBits = packer.maxBits
            

            You can see the names that can be used (at present) in the packers dict at the bottom of config/registry.yaml in daf_butler.

            As for the review, most changes are the infrastructure in daf_butler, which also includes the implementation for visit/exposure+detector packing. When looking at that please note the message on 77e7103dbe7e1321efac59c1b5fad575658203fe; the logic in data ID construction is already sufficiently hard to follow that I really want to fix it soon, but I do nevertheless want to defer that to another ticket.

            There are also much smaller changes to:

            • obs_subaru: just adds some now-necessary metadata to Instrument registration
            • skymap: new metadata in SkyMap registration and packer implementations for SkyMap-based data IDs.
            • ci_hsc: a new test.
            Show
            jbosch Jim Bosch added a comment - As promised, here's the support for packing data IDs into integers. To use it from a PipelineTask, you should be able to something like: packer = dataId.packers["TractPatch"] packedIntegerId = packer.pack(dataId) maxBits = packer.maxBits You can see the names that can be used (at present) in the packers dict at the bottom of config/registry.yaml in daf_butler. As for the review, most changes are the infrastructure in daf_butler , which also includes the implementation for visit/exposure+detector packing. When looking at that please note the message on 77e7103dbe7e1321efac59c1b5fad575658203fe ; the logic in data ID construction is already sufficiently hard to follow that I really want to fix it soon, but I do nevertheless want to defer that to another ticket. There are also much smaller changes to: obs_subaru : just adds some now-necessary metadata to Instrument registration skymap : new metadata in SkyMap registration and packer implementations for SkyMap-based data IDs. ci_hsc : a new test.

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Nate Lust
                Watchers:
                Jim Bosch, Nate Lust
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel