Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14821

Provide packed integer versions of Gen3 data IDs

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      We frequently need to encode at least (camera, visit, sensor), (skymap, tract, patch), and (skymap, tract, patch, abstract_filter) data IDS into unique reversible integer IDs.  Gen3 needs to provide methods for translating both from dict to int and from int to dict.  Because we'd like to be able to further pack these with Source IDs, we also need to be able to report how many bits are utilized by the data IDs.  This is tricky because the number of bits occupied by any particular data ID key-value pair depends on the particular Camera or SkyMap with which it is associated.

      Ideally, at least the dict->int conversion would be handled via a custom, dict-like DataId class that makes the packed integer ID available as a property or method; we'd like to have something we can pass to Task.run methods that shouldn't require a butler but may want to able to report data ID information in either form for diagnostic purposes.

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue blocks DM-14816 [ DM-14816 ]
            jbosch Jim Bosch made changes -
            Risk Score 0
            jbosch Jim Bosch made changes -
            Description We frequently need to encode at least (camera, visit, sensor), (skymap, tract, patch), and (skymap, tract, patch, abstract_filter) data IDS into unique _reversible_ integer IDs.  Gen3 Butlers need to provide methods for translating both from dict to int and from int to dict.  Because we'd like to be able to further pack these with Source IDs, we also need to be able to report how many bits are utilized by the data IDs.  This is tricky because the number of bits occupied by any particular data ID key-value pair depends on the particular Camera or SkyMap with which it is associated. We frequently need to encode at least (camera, visit, sensor), (skymap, tract, patch), and (skymap, tract, patch, abstract_filter) data IDS into unique _reversible_ integer IDs.  Gen3 needs to provide methods for translating both from dict to int and from int to dict.  Because we'd like to be able to further pack these with Source IDs, we also need to be able to report how many bits are utilized by the data IDs.  This is tricky because the number of bits occupied by any particular data ID key-value pair depends on the particular Camera or SkyMap with which it is associated.

            Ideally, at least the dict->int conversion would be handled via a custom, dict-like DataId class that makes the packed integer ID available as a property or method; we'd like to have something we can pass to {{Task.run}} methods that shouldn't require a butler but may want to able to report data ID information in either form for diagnostic purposes.
            swinbank John Swinbank made changes -
            Team Data Release Production [ 10301 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-15034 [ DM-15034 ]
            jbosch Jim Bosch made changes -
            Link This issue blocks DM-16227 [ DM-16227 ]
            jbosch Jim Bosch made changes -
            Assignee Jim Bosch [ jbosch ]
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            jbosch Jim Bosch added a comment -

            As promised, here's the support for packing data IDs into integers.

            To use it from a PipelineTask, you should be able to something like:

            packer = dataId.packers["TractPatch"]
            packedIntegerId = packer.pack(dataId)
            maxBits = packer.maxBits
            

            You can see the names that can be used (at present) in the packers dict at the bottom of config/registry.yaml in daf_butler.

            As for the review, most changes are the infrastructure in daf_butler, which also includes the implementation for visit/exposure+detector packing. When looking at that please note the message on 77e7103dbe7e1321efac59c1b5fad575658203fe; the logic in data ID construction is already sufficiently hard to follow that I really want to fix it soon, but I do nevertheless want to defer that to another ticket.

            There are also much smaller changes to:

            • obs_subaru: just adds some now-necessary metadata to Instrument registration
            • skymap: new metadata in SkyMap registration and packer implementations for SkyMap-based data IDs.
            • ci_hsc: a new test.
            Show
            jbosch Jim Bosch added a comment - As promised, here's the support for packing data IDs into integers. To use it from a PipelineTask, you should be able to something like: packer = dataId.packers["TractPatch"] packedIntegerId = packer.pack(dataId) maxBits = packer.maxBits You can see the names that can be used (at present) in the packers dict at the bottom of config/registry.yaml in daf_butler. As for the review, most changes are the infrastructure in daf_butler , which also includes the implementation for visit/exposure+detector packing. When looking at that please note the message on 77e7103dbe7e1321efac59c1b5fad575658203fe ; the logic in data ID construction is already sufficiently hard to follow that I really want to fix it soon, but I do nevertheless want to defer that to another ticket. There are also much smaller changes to: obs_subaru : just adds some now-necessary metadata to Instrument registration skymap : new metadata in SkyMap registration and packer implementations for SkyMap-based data IDs. ci_hsc : a new test.
            jbosch Jim Bosch made changes -
            Reviewers Nate Lust [ nlust ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            nlust Nate Lust added a comment -

            Some comments up, and some changes we talked about to the registry would be desireable

            Show
            nlust Nate Lust added a comment - Some comments up, and some changes we talked about to the registry would be desireable
            nlust Nate Lust made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Epic Link DM-16771 [ 235477 ]
            swinbank John Swinbank made changes -
            Link This issue relates to DM-6913 [ DM-6913 ]

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Nate Lust
                Watchers:
                Jim Bosch, Nate Lust
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel