Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14428

Add skypix and spatial relationships to Gen2->Gen3 conversion

    Details

    • Story Points:
      4
    • Sprint:
      BG3_S18_05, BG3_F18_06
    • Team:
      Data Release Production

      Description

      In the absence of DM-13990, I'll use HTM at depth 7 for everything, because that's what the reference catalog in ci_hsc uses.

      This will include updating daf_butler to work with the master version of skymap, which has slightly different APIs for sphgeom interoperability than the gen3-middleware branch.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Andy Salnikov, this is not quite as complete as I'd like it to be before review, but as I'm traveling for the next few weeks with limited time to work on it, I thought I'd send it to you now.  I think it does most of what you'll need to start testing preflight with many-to-many join views and tables.

            There are changes here for daf_butler, skymap, and ci_hsc; with those branches, the gen3-middleware branch of obs_subaru, and the w_2018_22 tag of all other ci_hsc dependencies, the ci_hsc SCons scripts will now construct a Gen3 repository with tables and views for all spatial relationships.  The HTM scale I've chosen (to match the scale used for reference catalogs, for now) is on the coarse side (larger than a sensor or patch, but smaller than a visit or tract), so there will need to be some postprocessing on any spatial join results to filter out spurious overlaps.

            I've also added an AbstractFilter view, as we discussed previously.

            I'd ultimately like to add some unit tests to daf_butler itself for this functionality, but I'd also be happy to merge this sooner and add those on another ticket if that would help you out (you're also just welcome to use this branch, of course).

            Show
            jbosch Jim Bosch added a comment - Andy Salnikov , this is not quite as complete as I'd like it to be before review, but as I'm traveling for the next few weeks with limited time to work on it, I thought I'd send it to you now.  I think it does most of what you'll need to start testing preflight with many-to-many join views and tables. There are changes here for daf_butler, skymap, and ci_hsc; with those branches, the gen3-middleware branch of obs_subaru, and the w_2018_22 tag of all other ci_hsc dependencies, the ci_hsc SCons scripts will now construct a Gen3 repository with tables and views for all spatial relationships.  The HTM scale I've chosen (to match the scale used for reference catalogs, for now) is on the coarse side (larger than a sensor or patch, but smaller than a visit or tract), so there will need to be some postprocessing on any spatial join results to filter out spurious overlaps. I've also added an AbstractFilter view, as we discussed previously. I'd ultimately like to add some unit tests to daf_butler itself for this functionality, but I'd also be happy to merge this sooner and add those on another ticket if that would help you out (you're also just welcome to use this branch, of course).

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Andy Salnikov
                Watchers:
                Andy Salnikov, Jim Bosch
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel