Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29514

Test resultant AM metrics on FitAffineWcsTask on HSC full plane distortion model using validate_drp.

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      6
    • Sprint:
      AP S21-5 (April), AP F21-1 (June)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Once we have checked that the full focal plane distortion model exists and works, test it and compare the metrics created with the current package for computing DRP metrics to the default with the distortion model turned off.

        Attachments

        1. affine_Hsc_output_HSC-I_check_astrometry.png
          affine_Hsc_output_HSC-I_check_astrometry.png
          130 kB
        2. affine_Hsc_output_HSC-I_check_photometry.png
          affine_Hsc_output_HSC-I_check_photometry.png
          295 kB
        3. affine_Hsc_output_HSC-I_PA1.png
          affine_Hsc_output_HSC-I_PA1.png
          187 kB
        4. affine_Hsc_output_HSC-I_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          affine_Hsc_output_HSC-I_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          274 kB
        5. affine_Hsc_output_HSC-I_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          affine_Hsc_output_HSC-I_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          273 kB
        6. affine_Hsc_output_HSC-I_validate_drp.TE1_D_1_arcmin.png
          affine_Hsc_output_HSC-I_validate_drp.TE1_D_1_arcmin.png
          188 kB
        7. affine_Hsc_output_HSC-I_validate_drp.TE2_D_5_arcmin.png
          affine_Hsc_output_HSC-I_validate_drp.TE2_D_5_arcmin.png
          184 kB
        8. affine_Hsc_output_HSC-R_check_astrometry.png
          affine_Hsc_output_HSC-R_check_astrometry.png
          122 kB
        9. affine_Hsc_output_HSC-R_check_photometry.png
          affine_Hsc_output_HSC-R_check_photometry.png
          298 kB
        10. affine_Hsc_output_HSC-R_PA1.png
          affine_Hsc_output_HSC-R_PA1.png
          197 kB
        11. affine_Hsc_output_HSC-R_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          affine_Hsc_output_HSC-R_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          276 kB
        12. affine_Hsc_output_HSC-R_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          affine_Hsc_output_HSC-R_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          272 kB
        13. affine_Hsc_output_HSC-R_validate_drp.TE1_D_1_arcmin.png
          affine_Hsc_output_HSC-R_validate_drp.TE1_D_1_arcmin.png
          196 kB
        14. affine_Hsc_output_HSC-R_validate_drp.TE2_D_5_arcmin.png
          affine_Hsc_output_HSC-R_validate_drp.TE2_D_5_arcmin.png
          203 kB
        15. affine_Hsc_output_HSC-Y_check_astrometry.png
          affine_Hsc_output_HSC-Y_check_astrometry.png
          129 kB
        16. affine_Hsc_output_HSC-Y_check_photometry.png
          affine_Hsc_output_HSC-Y_check_photometry.png
          302 kB
        17. affine_Hsc_output_HSC-Y_PA1.png
          affine_Hsc_output_HSC-Y_PA1.png
          162 kB
        18. affine_Hsc_output_HSC-Y_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          affine_Hsc_output_HSC-Y_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          271 kB
        19. affine_Hsc_output_HSC-Y_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          affine_Hsc_output_HSC-Y_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          281 kB
        20. affine_Hsc_output_HSC-Y_validate_drp.TE1_D_1_arcmin.png
          affine_Hsc_output_HSC-Y_validate_drp.TE1_D_1_arcmin.png
          203 kB
        21. affine_Hsc_output_HSC-Y_validate_drp.TE2_D_5_arcmin.png
          affine_Hsc_output_HSC-Y_validate_drp.TE2_D_5_arcmin.png
          207 kB
        22. hscTestAffine.out
          242 kB
        23. hscTestStack.out
          243 kB
        24. tanSip_Hsc_output_HSC-I_check_astrometry.png
          tanSip_Hsc_output_HSC-I_check_astrometry.png
          134 kB
        25. tanSip_Hsc_output_HSC-I_check_photometry.png
          tanSip_Hsc_output_HSC-I_check_photometry.png
          297 kB
        26. tanSip_Hsc_output_HSC-I_PA1.png
          tanSip_Hsc_output_HSC-I_PA1.png
          188 kB
        27. tanSip_Hsc_output_HSC-I_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          tanSip_Hsc_output_HSC-I_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          276 kB
        28. tanSip_Hsc_output_HSC-I_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          tanSip_Hsc_output_HSC-I_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          273 kB
        29. tanSip_Hsc_output_HSC-I_validate_drp.TE1_D_1_arcmin.png
          tanSip_Hsc_output_HSC-I_validate_drp.TE1_D_1_arcmin.png
          193 kB
        30. tanSip_Hsc_output_HSC-I_validate_drp.TE2_D_5_arcmin.png
          tanSip_Hsc_output_HSC-I_validate_drp.TE2_D_5_arcmin.png
          190 kB
        31. tanSip_Hsc_output_HSC-R_check_astrometry.png
          tanSip_Hsc_output_HSC-R_check_astrometry.png
          122 kB
        32. tanSip_Hsc_output_HSC-R_check_photometry.png
          tanSip_Hsc_output_HSC-R_check_photometry.png
          299 kB
        33. tanSip_Hsc_output_HSC-R_PA1.png
          tanSip_Hsc_output_HSC-R_PA1.png
          198 kB
        34. tanSip_Hsc_output_HSC-R_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          tanSip_Hsc_output_HSC-R_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          278 kB
        35. tanSip_Hsc_output_HSC-R_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          tanSip_Hsc_output_HSC-R_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          272 kB
        36. tanSip_Hsc_output_HSC-R_validate_drp.TE1_D_1_arcmin.png
          tanSip_Hsc_output_HSC-R_validate_drp.TE1_D_1_arcmin.png
          193 kB
        37. tanSip_Hsc_output_HSC-R_validate_drp.TE2_D_5_arcmin.png
          tanSip_Hsc_output_HSC-R_validate_drp.TE2_D_5_arcmin.png
          199 kB
        38. tanSip_Hsc_output_HSC-Y_check_astrometry.png
          tanSip_Hsc_output_HSC-Y_check_astrometry.png
          126 kB
        39. tanSip_Hsc_output_HSC-Y_check_photometry.png
          tanSip_Hsc_output_HSC-Y_check_photometry.png
          302 kB
        40. tanSip_Hsc_output_HSC-Y_PA1.png
          tanSip_Hsc_output_HSC-Y_PA1.png
          162 kB
        41. tanSip_Hsc_output_HSC-Y_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          tanSip_Hsc_output_HSC-Y_validate_drp.AM1_D_5_arcmin_17.0_21.5_mag.png
          264 kB
        42. tanSip_Hsc_output_HSC-Y_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          tanSip_Hsc_output_HSC-Y_validate_drp.AM2_D_20_arcmin_17.0_21.5_mag.png
          271 kB
        43. tanSip_Hsc_output_HSC-Y_validate_drp.TE1_D_1_arcmin.png
          tanSip_Hsc_output_HSC-Y_validate_drp.TE1_D_1_arcmin.png
          202 kB
        44. tanSip_Hsc_output_HSC-Y_validate_drp.TE2_D_5_arcmin.png
          tanSip_Hsc_output_HSC-Y_validate_drp.TE2_D_5_arcmin.png
          207 kB

          Issue Links

            Activity

            Hide
            lguy Leanne Guy added a comment -

            Chris Morrison [X] Ian Sullivan validate_drp is deprecated and replaced with faro. All AM metrics that were implemented in validate_drp have all been ported as-is to faro. Could this be done using faro?

            Show
            lguy Leanne Guy added a comment - Chris Morrison [X] Ian Sullivan validate_drp is deprecated and replaced with faro. All AM metrics that were implemented in validate_drp have all been ported as-is to faro. Could this be done using faro?
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Hey Leanne, sorry to drop your message here, I've been working on the push towards DP0. I just finished the analysis using validate_drp. If Ian Sullivan or Eric Bellm want I can run the data again using faro but that would be on another ticket at this point.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Hey Leanne, sorry to drop your message here, I've been working on the push towards DP0. I just finished the analysis using validate_drp. If Ian Sullivan or Eric Bellm want I can run the data again using faro but that would be on another ticket at this point.
            Hide
            lguy Leanne Guy added a comment -

            OK, If you have done it then no need to run again. Would be good to use faro in the future as we do not plan to continue to maintain validate_drp

            Show
            lguy Leanne Guy added a comment - OK, If you have done it then no need to run again. Would be good to use faro in the future as we do not plan to continue to maintain validate_drp
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited

            After running over the validate_drp data for HSC that has a distortion model, the simple FitAffineWcsTask performs fairly similar to the default FitTanSipWcsTask fitter.

            For design level metrics in validate_drp, the default Wcs fitter fails 13 metrics vs FitAffineWcsTask failing 16. I've attached the full output from both runs to compare metrics directly but overall the FitAffineWcsTask is within roughly 10% of the TanSip fitter. The additional design level failures come from the Y band data which is only barely for the passing for AM1/2 for the TansSip fitter.

            My assessment here is that it is likely worth analyzing a larger dataset on HSC or another camera with a well modeled distortion.

            I've attached the check plots for both runs as well as the raw output for both runs to this ticket. Full outputs are attached at hscTestAffine.out and hscTestStack.out for the affine and tanSip fitter respectively.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited After running over the validate_drp data for HSC that has a distortion model, the simple FitAffineWcsTask performs fairly similar to the default FitTanSipWcsTask fitter. For design level metrics in validate_drp, the default Wcs fitter fails 13 metrics vs FitAffineWcsTask failing 16. I've attached the full output from both runs to compare metrics directly but overall the FitAffineWcsTask is within roughly 10% of the TanSip fitter. The additional design level failures come from the Y band data which is only barely for the passing for AM1/2 for the TansSip fitter. My assessment here is that it is likely worth analyzing a larger dataset on HSC or another camera with a well modeled distortion. I've attached the check plots for both runs as well as the raw output for both runs to this ticket. Full outputs are attached at hscTestAffine.out and hscTestStack.out for the affine and tanSip fitter respectively.
            Hide
            jbosch Jim Bosch added a comment -

            Looks fine. I don't think repeating exactly this exercise with faro is worthwhile, because I think the astrometry metrics implemented by both validate_drp and faro aren't really relevant for this task, at least not the way we're running it; they're really designed to test the rigidness of the astrometric solution for many visits, which is something we can only get from jointcal-style astrometry in DRP.

            Instead, we need this task to get us:

            • super robust (but not necessarily precise) initial astrometry in DRP, so we're basically just looking at the failure rate on a large dataset, while making sure catastrophically bad fits are never declare successes;
            • super robust, low-residual, and ideally unbiased fits to a reference catalog in AP (with that reference catalog being something like a DRP object catalog filtered down to just point sources, and hence much deeper and denser than external reference catalogs we have now), so we'd want to test that primarily by looking at plots that just compare measured positions to reference catalog WCS-projected positions.

            Of course, it's possible metrics that capture those sorts of things are being added to faro already, and if not adding them to faro as a part of doing that follow-up analysis (with associated plots in and/or based on analysis_drp) would be great.

            And, finally, I should mention a big caveat: the metrics used here are relevant for this task in the way we currently run ImSim DC2 data, because we don't follow the initial astrometry there up with jointcal (because we don't think we have to). So that's a sort of artificial situation that I'd prefer to not have to spend on time on, but given the importance of ImSim DC2 data for a lot of our plans, it may be worth looking into using this task there as well. But I think it's also true that processing DC2 data (even in DRP mode) is pretty close to the real-world AP astrometry context: we have a much deeper reference catalog that we can assume is truth, and hence we can probably learn everything we need to learn about the astrometry by comparisons to it.

            Show
            jbosch Jim Bosch added a comment - Looks fine. I don't think repeating exactly this exercise with faro is worthwhile, because I think the astrometry metrics implemented by both validate_drp and faro aren't really relevant for this task, at least not the way we're running it; they're really designed to test the rigidness of the astrometric solution for many visits, which is something we can only get from jointcal-style astrometry in DRP. Instead, we need this task to get us: super robust (but not necessarily precise) initial astrometry in DRP, so we're basically just looking at the failure rate on a large dataset, while making sure catastrophically bad fits are never declare successes; super robust, low-residual, and ideally unbiased fits to a reference catalog in AP (with that reference catalog being something like a DRP object catalog filtered down to just point sources, and hence much deeper and denser than external reference catalogs we have now), so we'd want to test that primarily by looking at plots that just compare measured positions to reference catalog WCS-projected positions. Of course, it's possible metrics that capture those sorts of things are being added to faro already, and if not adding them to faro as a part of doing that follow-up analysis (with associated plots in and/or based on analysis_drp ) would be great. And, finally, I should mention a big caveat: the metrics used here are relevant for this task in the way we currently run ImSim DC2 data, because we don't follow the initial astrometry there up with jointcal (because we don't think we have to). So that's a sort of artificial situation that I'd prefer to not have to spend on time on, but given the importance of ImSim DC2 data for a lot of our plans, it may be worth looking into using this task there as well. But I think it's also true that processing DC2 data (even in DRP mode) is pretty close to the real-world AP astrometry context: we have a much deeper reference catalog that we can assume is truth, and hence we can probably learn everything we need to learn about the astrometry by comparisons to it.
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            The task is currently in the stack and can be used as a drop in replacement for the default fitter. The tests you are discussing can be run at any time.

            Previously when run on no-distortion model (in the cast of DECam) and one that wasn't properly used at the time (HSC), the fitter return values around the ~30 marasec scale. Finally running on the HSC data with a distortion model and returning similar results to the poly fitter, means that the code is doing something reasonable and that, assuming the Wcs distortion model is good enough, could be used in the context of AP for simple Wcs tweak. That last point is the context in which the fitter was created and this test was done. Having it work beyond that and add "robustness" to the WCS is, as you suggest above, future work that will need to be scoped.

            I'll point out that you should take a look at how AstrometryTask is constructed currently before expecting any more robustness to be gained as it is basically designed around the outputs of the fitter and may have assumptions baked in on how the fitted matches are processed and clipped on scatter.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - The task is currently in the stack and can be used as a drop in replacement for the default fitter. The tests you are discussing can be run at any time. Previously when run on no-distortion model (in the cast of DECam) and one that wasn't properly used at the time (HSC), the fitter return values around the ~30 marasec scale. Finally running on the HSC data with a distortion model and returning similar results to the poly fitter, means that the code is doing something reasonable and that, assuming the Wcs distortion model is good enough, could be used in the context of AP for simple Wcs tweak. That last point is the context in which the fitter was created and this test was done. Having it work beyond that and add "robustness" to the WCS is, as you suggest above, future work that will need to be scoped. I'll point out that you should take a look at how AstrometryTask is constructed currently before expecting any more robustness to be gained as it is basically designed around the outputs of the fitter and may have assumptions baked in on how the fitted matches are processed and clipped on scatter.

              People

              Assignee:
              cmorrison Chris Morrison [X] (Inactive)
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Jim Bosch
              Watchers:
              Chris Morrison [X] (Inactive), Ian Sullivan, Jim Bosch, Leanne Guy
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.