Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30030

Make calibration source selection consistent between gen2 and gen3 middleware - part deux

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Epic Link:
    • Sprint:
      DRP S21b
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Analysis on DM-29819 revealed some residual issues related to loading of reference catalogs. Specifically, because of an inconsistent setting of the config parameter:
      calibrate.astromRefObjLoader.pixelMargin
      and the
      computeVisitRegions["single-raw-wcs"].padding
      of defineVisits.py
      there are cases where slightly different reference catalogs are getting loaded in gen2 vs. gen3, I think the logic should be that pixelMargin <= computeVisitRegions["single-raw-wcs"].padding. However, as it stands, the default for pixelMargin is 300 and, in cases we were unlucky not to encounter in ci_hsc, this can lead to a smaller loaded region in gen3 because of the smaller visit padded definition (i.e. if a shard edge lies close to the padded visit edge). See DM-29819 for an example.

      While I'm not sure it will be possible to check the above condition in a config "validation", I think we should at least ensure it is satisfied for the stack defaults as well as any overrides in individual obs_ packages.

        Attachments

        1. allRefCats_19662_1_newPad.png
          allRefCats_19662_1_newPad.png
          359 kB
        2. allRefCats_19662_1_origPad.png
          allRefCats_19662_1_origPad.png
          370 kB
        3. allRefCats_257768_0_newPad.png
          allRefCats_257768_0_newPad.png
          411 kB
        4. allRefCats_257768_0_origPad.png
          allRefCats_257768_0_origPad.png
          417 kB
        5. allRefCats_289408_1_newPad.png
          allRefCats_289408_1_newPad.png
          414 kB
        6. allRefCats_289408_1_origPad.png
          allRefCats_289408_1_origPad.png
          424 kB
        7. fixedGen3_19662_1.png
          fixedGen3_19662_1.png
          482 kB
        8. gen2gen3trimmedRefCats.png
          gen2gen3trimmedRefCats.png
          837 kB
        9. gen2gen3trimmedRefCatsFix.png
          gen2gen3trimmedRefCatsFix.png
          738 kB
        10. gen3RefLoadingIssue.png
          gen3RefLoadingIssue.png
          655 kB
        11. trimmedInput.png
          trimmedInput.png
          152 kB
        12. wcsRoundTrip.png
          wcsRoundTrip.png
          180 kB
        13. wcsRoundTripZoom.png
          wcsRoundTripZoom.png
          393 kB

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment - - edited

            Looks good, but would you mind removing the obs_subaru defineVisits.py config override as well, now that it's unneccessary?  (https://github.com/lsst/obs_subaru/blob/master/config/defineVisits.py)

            Show
            jbosch Jim Bosch added a comment - - edited Looks good, but would you mind removing the obs_subaru defineVisits.py config override as well, now that it's unneccessary?  ( https://github.com/lsst/obs_subaru/blob/master/config/defineVisits.py )
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Thanks for the review, Jim. I made the changes you requested (removed above noted obs_subaru config, so there's a new PR here, and added a comment to the test override in ap_association). I also did a bit more testing with other datasets just to make sure things look ok (thanks to Meredith Rawls for getting my going with DECam processing in gen3!)

            The following plots show:

            • full reference catalog (i.e. all of the shards associated with the visit/detector) as blue circles
            • new "pre-filtered" catalog added in this ticket as orange crosses
            • sources actually used in the SFM astrometry fit (astrom_used) as purple stars
            • several WCS/bboxes: black = raw WCS, dotted pink = black + original 300 padding, dash-dot lime = dotted pink + 100 pixel "extra" outerRegion padding, dashed teal = calexp WCS.

            In the following series of plots for the three datasets, the first plot shows the above for the original padding of 300 and the second is for the new 250 pixel padding (the pink and lime bboxes still reflect the 300, so you can see the pre-filtered cat doesn't extend all the way out to them).

            HSC:
            300 pixel padding

            250 pixel padding

            DECam:
            300 pixel padding

            250 pixel padding

            DC2:
            300 pixel padding

            250 pixel padding

            They all look like the ended up with identical astrometry fits...except for DC2 example, where there seems to be a dearth of astrom_used points in the upper right corner of the original 300 pixel padding, but a bunch do show up in the 250 pixel padding. I actually looked at ~twenty different detectors for this DC2 visit and this was the only one showing any difference. Given that the calexp WCSs look identical and that the 250 padding seems to be "better", I'm not concerned about this difference.

            I had to do some rebasing and there's the new obs_subaru PR, so I kicked off another Jenkins (but seeing the recent traffic on #dm-build-problems, I won't be surprised if the MacOS build fails...)

            Show
            lauren Lauren MacArthur added a comment - - edited Thanks for the review, Jim. I made the changes you requested (removed above noted obs_subaru config, so there's a new PR here, and added a comment to the test override in ap_association ). I also did a bit more testing with other datasets just to make sure things look ok (thanks to Meredith Rawls for getting my going with DECam processing in gen3!) The following plots show: full reference catalog (i.e. all of the shards associated with the visit/detector) as blue circles new "pre-filtered" catalog added in this ticket as orange crosses sources actually used in the SFM astrometry fit (astrom_used) as purple stars several WCS/bboxes: black = raw WCS, dotted pink = black + original 300 padding, dash-dot lime = dotted pink + 100 pixel "extra" outerRegion padding, dashed teal = calexp WCS. In the following series of plots for the three datasets, the first plot shows the above for the original padding of 300 and the second is for the new 250 pixel padding (the pink and lime bboxes still reflect the 300, so you can see the pre-filtered cat doesn't extend all the way out to them). HSC: 300 pixel padding 250 pixel padding DECam: 300 pixel padding 250 pixel padding DC2: 300 pixel padding 250 pixel padding They all look like the ended up with identical astrometry fits...except for DC2 example, where there seems to be a dearth of astrom_used  points in the upper right corner of the original 300 pixel padding, but a bunch do show up in the 250 pixel padding. I actually looked at ~twenty different detectors for this DC2 visit and this was the only one showing any difference. Given that the calexp  WCSs look identical and that the 250 padding seems to be "better", I'm not concerned about this difference. I had to do some rebasing and there's the new obs_subaru PR, so I kicked off another Jenkins (but seeing the recent traffic on #dm-build-problems, I won't be surprised if the MacOS build fails...)
            Hide
            lauren Lauren MacArthur added a comment -

            Allrighty...we finally got a green Jenkins.  Merged and done!  Community post announcing the config default changes and potential edge cases with current gen3 repo visit definitions to follow soon.

            Show
            lauren Lauren MacArthur added a comment - Allrighty...we finally got a green Jenkins .  Merged and done!  Community post announcing the config default changes and potential edge cases with current gen3 repo visit definitions to follow soon.
            Hide
            ktl Kian-Tat Lim added a comment -

            Unfortunately, this ticket broke ap_verify as it did not remove the reference to defineVisits.py in https://github.com/lsst/obs_subaru/blob/master/config/datasetIngest-gen3.py#L7

            Show
            ktl Kian-Tat Lim added a comment - Unfortunately, this ticket broke ap_verify as it did not remove the reference to defineVisits.py in https://github.com/lsst/obs_subaru/blob/master/config/datasetIngest-gen3.py#L7
            Hide
            lauren Lauren MacArthur added a comment -

            Aargh...sorry about that. I can get in a fix this afternoon.

            Show
            lauren Lauren MacArthur added a comment - Aargh...sorry about that. I can get in a fix this afternoon.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Jim Bosch
              Watchers:
              Eli Rykoff, Jim Bosch, John Parejko, Kian-Tat Lim, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.