Details
-
Type:
Story
-
Status: To Do
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: daf_butler
-
Labels:
-
Team:Data Release Production
-
Urgent?:No
Description
Use-cases for the healpix dimensions include creating HiPS images and healsparse property maps, which would at least ideally run via quantum with healpix dimensions and produce outputs with healpix dimensions.
Basic healpix support on DM-33946 won't be enough, because that just brings healpix to the level of support of our current HTM dimensions, and we can't really use those for anything other than prerequisite inputs, either.
We also have many more use cases where we are using tract or patch in contexts where we really want a tessellation with no overlaps, and any skypix dimension would be preferable if they were fully usable.
The fundamental problem here is that Registry data ID queries can't yield rows with skypix dimension columns (except, in special cases, the "common" htm7 dimension) because the database doesn't have any tables with those dimensions besides the ones that hold existing datasets. As a result, we can query for existing datasets with skypix dimensions - inefficiently, by using sphgeom to project other-dimension regions to skypix IDs and then querying for those IDs, one at a time. But the query system needs major upgrades to do anything else.
We have tables in the schema already that could solve this that are largely going unused, by storing explicitly-materialized overlaps between specific skypix dimensions and other spatial dimensions. I'll be working on including those in the query system on DM-31725.
Attachments
Issue Links
Activity
| Field | Original Value | New Value |
|---|---|---|
| Labels | gen3-middleware |
| Team | Data Release Production [ 10301 ] | |
| Assignee | Jim Bosch [ jbosch ] |
| Labels | gen3-middleware | gen3-middleware gen3-registry-incompatibility |
| Labels | gen3-middleware gen3-registry-incompatibility | gen3-middleware |
| Summary | Add healpix dimension support to butler | Make skypix dimensions usable in regular input, output, and quantum data IDs |
| Description |
After DM-13483 is done it will be possible to add a healpix dimension to the butler. I understand this will require a config change and a migration.
One of the known use-cases for the heakpix dimension is to be able to determine which patches overlap a given healpix id. For example, when creating HiPS images, which are on healpix cells, we need to know which patches are overlapping. This will require repos to be able to support both htm7 and healpix geometries. Another example are the healsparse property maps which need to aggregate tracts to healpix pixels. Note that these use cases require different healpix nside. Fortunately, using nest ordering it is possible to go from one resolution to another with a simple bit shift. |
Use-cases for the healpix dimensions include creating HiPS images and healsparse property maps, which would at least ideally run via quantum with healpix dimensions and produce outputs with healpix dimensions.
Basic healpix support on We also have many more use cases where we are using tract or patch in contexts where we really want a tessellation with no overlaps, and any skypix dimension would be preferable if they were fully usable. The fundamental problem here is that Registry data ID queries can't yield rows with skypix dimension columns (except, in special cases, the "common" htm7 dimension) because the database doesn't have any tables with those dimensions besides the ones that hold existing datasets. As a result, we can query for existing datasets with skypix dimensions - inefficiently, by using sphgeom to project other-dimension regions to skypix IDs and then querying for those IDs, one at a time. But the query system needs major upgrades to do anything else. We have tables in the schema already that could solve this that are largely going unused, by storing explicitly-materialized overlaps between specific skypix dimensions and other spatial dimensions. I'll be working on including those in the query system on |