Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19328

Investigate cause of small differences in Uber calibrations from w_2019_[06/10/14]

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      3
    • Epic Link:
    • Sprint:
      DRP S19-5
    • Team:
      Data Release Production

      Description

      While validating the fix in DM-19265, it was noted that there are differences in the uber calibration results from both jointcal and meas_mosaic between recent weekly runs of the RC2 dataset. As the plots on DM-19265 show, there have been some subtle changes from w_2019_06-to-w_2019_10-to-w_2019_14. The differences are small (stdev of difference is at the sub mmag level), but it is still important that we understand where they originated. Since both algorithms are affected, the route cause must be something that effects both. A first guess would be something about how object selection is being done.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            I've compared the flux and astrom calibrations between weekly runs. The differences first show up when comparing w_2019_02 vs w_2019_06 (i.e. there is zero difference when comparing w_2019_02 vs. w_2018_50). Differences also exist between w_2019_06 vs w_2019_10 and between w_2019_10 vs w_2019_14, indicating that subtle changes are happening between each of the reprocessings.

            I think I've narrowed this down to (subtle) changes in the astrometrey matcher which leads to slightly different matches for astrometry --> slightly different WCS solution --> slightly differenct source selection for PhotoCal --> slightly different zeropoints from SFM --> slightly different UberCal solutions.

            First, there is the switch of the default to PessimisticB in w04, explaining the differences between w02 and w06:

            New in w.2019.04
            DM-14857: Switch the default matcher to PessimisticB. [meas_astrom, obs_cfht, obs_subaru]
            

            Difference in SFM zp between w02 and w06:

            Difference in meas_mosaic calibration between w02 and w06 (jointcal still lacked color term corrections in w02 so looks much worse):

            Next, there were some tweeks in w07 along the lines of a "final verify step, using the first round of verified matches to refit the shift/rotation matrix" and a "distance tolerance softening", explaining the differences between w06 and w10:

            New in w.2019.07
            DM-17552: Check SQuaSH metric regressions using Match PessimisticB [meas_astrom, validate_drp]
            DM-17843: A new set of processCcd failure in HSC-RC2 reprocessing [meas_astrom]
            

            Difference in SFM zp between w06 and w10:

            Difference in meas_mosaic calibration between w06 and w10:

            Difference in jointcal calibration:

            Finally, another tweek in w14, explaining the differences between w10 and w14:

            New in w.2019.14
            DM-18400: Number of matches is smaller than request [meas_astrom]
            

            Difference in SFM zp between w10 and w14:

            Difference in meas_mosaic calibration between w10 and w14:

            Difference in jointcal calibration between w10 and w14:

            So, my conclusion is that these small differences in reprocessing-to-reprocessing calibrations are all understood by the above astrometry matcher churn, starting with the switch from "Optimistic" to "Pessimistic". My speculation is that assuming things have settled down in the matcher tweeking, we will not see any difference between the current w_2019_14 and the next reprocessing, presumably w_2019_18.

            Show
            lauren Lauren MacArthur added a comment - - edited I've compared the flux and astrom calibrations between weekly runs. The differences first show up when comparing w_2019_02 vs w_2019_06 (i.e. there is zero difference when comparing w_2019_02 vs. w_2018_50 ). Differences also exist between w_2019_06 vs w_2019_10 and between w_2019_10 vs w_2019_14 , indicating that subtle changes are happening between each of the reprocessings. I think I've narrowed this down to (subtle) changes in the astrometrey matcher which leads to slightly different matches for astrometry --> slightly different WCS solution --> slightly differenct source selection for PhotoCal --> slightly different zeropoints from SFM --> slightly different UberCal solutions. First, there is the switch of the default to PessimisticB in w04 , explaining the differences between w02 and w06 : New in w.2019.04 DM-14857: Switch the default matcher to PessimisticB. [meas_astrom, obs_cfht, obs_subaru] Difference in SFM zp between w02 and w06 : Difference in meas_mosaic calibration between w02 and w06 ( jointcal still lacked color term corrections in w02 so looks much worse): Next, there were some tweeks in w07 along the lines of a "final verify step, using the first round of verified matches to refit the shift/rotation matrix" and a "distance tolerance softening", explaining the differences between w06 and w10 : New in w.2019.07 DM-17552: Check SQuaSH metric regressions using Match PessimisticB [meas_astrom, validate_drp] DM-17843: A new set of processCcd failure in HSC-RC2 reprocessing [meas_astrom] Difference in SFM zp between w06 and w10 : Difference in meas_mosaic calibration between w06 and w10 : Difference in jointcal calibration: Finally, another tweek in w14 , explaining the differences between w10 and w14 : New in w.2019.14 DM-18400: Number of matches is smaller than request [meas_astrom] Difference in SFM zp between w10 and w14 : Difference in meas_mosaic calibration between w10 and w14 : Difference in jointcal calibration between w10 and w14 : So, my conclusion is that these small differences in reprocessing-to-reprocessing calibrations are all understood by the above astrometry matcher churn, starting with the switch from "Optimistic" to "Pessimistic". My speculation is that assuming things have settled down in the matcher tweeking , we will not see any difference between the current w_2019_14 and the next reprocessing, presumably w_2019_18 .
            Hide
            lauren Lauren MacArthur added a comment -

            Jim, might you have a moment to look this over and see if you agree with my findings?  In the case you do agree, I'm inclined to close this out and make another small ticket to double-check my speculation when the next RC2 reprocessing becomes available.

            In order to dig into this, I had to make a few updates to the pipe_analysis scripts, so there is a PR. The plots shown here are my validation of the changes. As they still live in lsst-dm, standards do not need to be strictly followed, but feel free to scrutinize at any level you feel warranted.

            Show
            lauren Lauren MacArthur added a comment - Jim, might you have a moment to look this over and see if you agree with my findings?  In the case you do agree, I'm inclined to close this out and make another small ticket to double-check my speculation when the next RC2 reprocessing becomes available. In order to dig into this, I had to make a few updates to the pipe_analysis scripts, so there is a PR. The plots shown here are my validation of the changes. As they still live in lsst-dm , standards do not need to be strictly followed, but feel free to scrutinize at any level you feel warranted.
            Hide
            jbosch Jim Bosch added a comment -

            Looks good!  Your conclusion seems entirely reasonable.

            Show
            jbosch Jim Bosch added a comment - Looks good!  Your conclusion seems entirely reasonable.
            Hide
            erykoff Eli Rykoff added a comment -

            There were some subtle differences in matcher parameters from DM-10800 and DM-19304. These should be very small for HSC, but there is definitely the possibility that a star or two will fall into or out of the selection causing small mmag-level shifts.
            The main change that I think could have an effect but probably won't for HSC is that now source selection and matching are done consistently on either aperture or psf magnitudes as configured, while this wasn't the case previously. However, due to happy accidents, for HSC this was done correctly while for CFHT it wasn't.
            But there are slight differences in the order in which things are run, so I wouldn't put my money on 100% the same.

            Show
            erykoff Eli Rykoff added a comment - There were some subtle differences in matcher parameters from DM-10800 and DM-19304 . These should be very small for HSC, but there is definitely the possibility that a star or two will fall into or out of the selection causing small mmag-level shifts. The main change that I think could have an effect but probably won't for HSC is that now source selection and matching are done consistently on either aperture or psf magnitudes as configured, while this wasn't the case previously. However, due to happy accidents, for HSC this was done correctly while for CFHT it wasn't. But there are slight differences in the order in which things are run, so I wouldn't put my money on 100% the same.
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks Eli!  I created DM-19477 as my follow-up ticket and made note of this there.

            Show
            lauren Lauren MacArthur added a comment - Thanks Eli!  I created DM-19477 as my follow-up ticket and made note of this there.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Jim Bosch
              Watchers:
              Eli Rykoff, Jim Bosch, John Parejko, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.