Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31705

Make skypix dimensions usable in regular input, output, and quantum data IDs

    XMLWordPrintable

    Details

    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Use-cases for the healpix dimensions include creating HiPS images and healsparse property maps, which would at least ideally run via quantum with healpix dimensions and produce outputs with healpix dimensions.

      Basic healpix support on DM-33946 won't be enough, because that just brings healpix to the level of support of our current HTM dimensions, and we can't really use those for anything other than prerequisite inputs, either.

      We also have many more use cases where we are using tract or patch in contexts where we really want a tessellation with no overlaps, and any skypix dimension would be preferable if they were fully usable.

      The fundamental problem here is that Registry data ID queries can't yield rows with skypix dimension columns (except, in special cases, the "common" htm7 dimension) because the database doesn't have any tables with those dimensions besides the ones that hold existing datasets.  As a result, we can query for existing datasets with skypix dimensions - inefficiently, by using sphgeom to project other-dimension regions to skypix IDs and then querying for those IDs, one at a time.  But the query system needs major upgrades to do anything else.

      We have tables in the schema already that could solve this that are largely going unused, by storing explicitly-materialized overlaps between specific skypix dimensions and other spatial dimensions.  I'll be working on including those in the query system on DM-31725.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31725 Rewrite butler registry.queries subpackage

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33946 Configure basic healpix support in butler data repositories

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Note to self and others:

            • For basic healpix support, we just need to add an entry to dimensions.yaml, but that won't do many of the things Eli Rykoff  needs.
            • If we replaced the htm7 "common skypix" definition with a healpix level, we could make a repo that could do everything Eli needs (but could not read our HTM-sharded refcats).  This might be useful as a stopgap for testing.
            • Real support requires deep query-system work for materializing specific spatial overlap relationships and using those materializations in queries.  The schema-side groundwork for this was done back in Nov. 2020, but the interfaces to populate the table and use them never got done.
            Show
            jbosch Jim Bosch added a comment - Note to self and others: For basic healpix support, we just need to add an entry to dimensions.yaml , but that won't do many of the things Eli Rykoff   needs. If we replaced the htm7 "common skypix" definition with a healpix level, we could make a repo that could do everything Eli needs (but could not read our HTM-sharded refcats).  This might be useful as a stopgap for testing. Real support requires deep query-system work for materializing specific spatial overlap relationships and using those materializations in queries.  The schema-side groundwork for this was done back in Nov. 2020, but the interfaces to populate the table and use them never got done.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Adding a block on DM-31725, as that's where I'll at least start that deep query system work.

            DM-13483 is I think now just a "nice to have"; the healpy-based implentation Eli provided should be fine for this ticket.

            Show
            jbosch Jim Bosch added a comment - Adding a block on DM-31725 , as that's where I'll at least start that deep query system work. DM-13483 is I think now just a "nice to have"; the healpy-based implentation Eli provided should be fine for this ticket.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Tim Jenness, this ticket involves two things, and one of them is a schema change that I think we'll want to tackle up-front with the next batch of migrations. The rest should not involve a schema change and is the bit blocked on DM-31725.

            Show
            jbosch Jim Bosch added a comment - Tim Jenness , this ticket involves two things, and one of them is a schema change that I think we'll want to tackle up-front with the next batch of migrations. The rest should not involve a schema change and is the bit blocked on DM-31725 .
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Jim Bosch Thanks. Can you put in the main description for this ticket exactly what the schema change is? (or create a new ticket that is just the schema change).

            Show
            tjenness Tim Jenness added a comment - Jim Bosch Thanks. Can you put in the main description for this ticket exactly what the schema change is? (or create a new ticket that is just the schema change).
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            I've made DM-33946 for the dimensions configuration change mentioned here, dropped the schema-incompatibility label and link to the DM-33942 schema-change umbrella on this ticket, and added the label and link to that new one.

            That leaves this ticket responsible for making not just HEALPix but all skypix dimensions usable in the same ways as regular dimensions, and I've updated the title accordingly.  The beginning of that work will still be on DM-31725, and it's not clear how much will be left for this ticket to do after that's done, but it's good to keep around regardless to capture the feature.

            Show
            jbosch Jim Bosch added a comment - I've made DM-33946 for the dimensions configuration change mentioned here, dropped the schema-incompatibility label and link to the DM-33942 schema-change umbrella on this ticket, and added the label and link to that new one. That leaves this ticket responsible for making not just HEALPix but all skypix dimensions usable in the same ways as regular dimensions, and I've updated the title accordingly.  The beginning of that work will still be on DM-31725 , and it's not clear how much will be left for this ticket to do after that's done, but it's good to keep around regardless to capture the feature.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              erykoff Eli Rykoff
              Watchers:
              Eli Rykoff, Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.