Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:6
-
Epic Link:
-
Team:Data Release Production
Description
We frequently need to encode at least (camera, visit, sensor), (skymap, tract, patch), and (skymap, tract, patch, abstract_filter) data IDS into unique reversible integer IDs. Gen3 needs to provide methods for translating both from dict to int and from int to dict. Because we'd like to be able to further pack these with Source IDs, we also need to be able to report how many bits are utilized by the data IDs. This is tricky because the number of bits occupied by any particular data ID key-value pair depends on the particular Camera or SkyMap with which it is associated.
Ideally, at least the dict->int conversion would be handled via a custom, dict-like DataId class that makes the packed integer ID available as a property or method; we'd like to have something we can pass to Task.run methods that shouldn't require a butler but may want to able to report data ID information in either form for diagnostic purposes.
As promised, here's the support for packing data IDs into integers.
To use it from a PipelineTask, you should be able to something like:
packer = dataId.packers["TractPatch"]packedIntegerId = packer.pack(dataId)maxBits = packer.maxBitsYou can see the names that can be used (at present) in the packers dict at the bottom of config/registry.yaml in daf_butler.
As for the review, most changes are the infrastructure in daf_butler, which also includes the implementation for visit/exposure+detector packing. When looking at that please note the message on 77e7103dbe7e1321efac59c1b5fad575658203fe; the logic in data ID construction is already sufficiently hard to follow that I really want to fix it soon, but I do nevertheless want to defer that to another ticket.
There are also much smaller changes to: