Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11090

Identify regression in HSC astrometric matching success rate between w_2017_17 and w_2017_25

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      6
    • Epic Link:
    • Sprint:
      DRP F17-2
    • Team:
      Data Release Production

      Description

      Hsin-Fang Chiang reports more failures in processing the HSC RC dataset with a recent version of the stack:

      Using the latest stack (w_2017_25) I'm reprocessing the RC dataset in DM-11020. I'm seeing more failures (81 ccds) in the single frame processing stage, compared to w_2017_17 (46 ccds).

      Out of the 81 failures:
      62 failed with: "Unable to match sources"
      4 failed with: "No objects passed our cuts for consideration as psf stars"
      4 failed with: "No sources remaining in match list after magnitude limit cuts"
      11 failed with: "PSF star selector found [123] candidates"

      I'm not sure if this is expected or not. Maybe somebody from the Science Pipeline would want to know? (edited)

      The difference seems to be those failed with "Unable to match sources". There were only 27 of those in w_2017_17. (All with the same RC dataset, 320 visits in total)

      We need to understand what went wrong here and fix it.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            Paul, would you mind having a look at this? Robert Lupton may also want to weigh in as this new setting (as opposed to a full revert) could influence HSC processing.

            If a full revert is deemed a better solution (which I don't see any need for), we would need to address my comment about the coupled nature of maxOffsetPix w.r.t. bbox growing for reference object search vs. max allowed frame translation above.

            Show
            lauren Lauren MacArthur added a comment - - edited Paul, would you mind having a look at this? Robert Lupton may also want to weigh in as this new setting (as opposed to a full revert) could influence HSC processing. If a full revert is deemed a better solution (which I don't see any need for), we would need to address my comment about the coupled nature of maxOffsetPix w.r.t. bbox growing for reference object search vs. max allowed frame translation above.
            Hide
            price Paul Price added a comment -

            Do we believe that the reason the failure curve turns up at large maxOffsetPix is because of shortcomings in the matcher (it settles on a bad fit in the larger solution space)?

            250 pixels seems reasonable to me — that's 42 arcsec in HSC, which should be well more than the typical pointing error.

            Do we want to adjust the default as well as the HSC setting?

            You've checked only the CCDs that failed with w_2017_25; can you check the ones that also succeeded, to be sure this doesn't have a bad impact on the rest of the population?

            Show
            price Paul Price added a comment - Do we believe that the reason the failure curve turns up at large maxOffsetPix is because of shortcomings in the matcher (it settles on a bad fit in the larger solution space)? 250 pixels seems reasonable to me — that's 42 arcsec in HSC, which should be well more than the typical pointing error. Do we want to adjust the default as well as the HSC setting? You've checked only the CCDs that failed with w_2017_25 ; can you check the ones that also succeeded, to be sure this doesn't have a bad impact on the rest of the population?
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Do we believe that the reason the failure curve turns up at large maxOffsetPix is because of shortcomings in the matcher (it settles on a bad fit in the larger solution space)?

            Yes, I believe this is indeed the current belief (and has been previously reported: see DM-10738)

            Do we want to adjust the default as well as the HSC setting?

            I'm going with "no" on this one. The default is 300, which is not all that different from 250 and I don't want to mess with what others may be using with no reported issues.

            can you check the ones that also succeeded, to be sure this doesn't have a bad impact on the rest of the population?

            Oh, yes, that was the point of running the analysis scripts and looking in particular at the astrometry results to make sure everything else looked equivalent. Here is an example:

            Pre-regression w_2017_17:

            Post-regression w_2017_26 + config fix:

            (sorry the plots are not quite the same...it appears the matplotlib versions on tiger vs. lsst-dev differ). I considered this a good enough way to check that all else was working as it should...let me know if you would like something more in-depth before I merge the config change.

            Show
            lauren Lauren MacArthur added a comment - - edited Do we believe that the reason the failure curve turns up at large maxOffsetPix is because of shortcomings in the matcher (it settles on a bad fit in the larger solution space)? Yes, I believe this is indeed the current belief (and has been previously reported: see DM-10738 ) Do we want to adjust the default as well as the HSC setting? I'm going with "no" on this one. The default is 300, which is not all that different from 250 and I don't want to mess with what others may be using with no reported issues. can you check the ones that also succeeded, to be sure this doesn't have a bad impact on the rest of the population? Oh, yes, that was the point of running the analysis scripts and looking in particular at the astrometry results to make sure everything else looked equivalent. Here is an example: Pre-regression w_2017_17 : Post-regression w_2017_26 + config fix : (sorry the plots are not quite the same...it appears the matplotlib versions on tiger vs. lsst-dev differ). I considered this a good enough way to check that all else was working as it should...let me know if you would like something more in-depth before I merge the config change.
            Hide
            price Paul Price added a comment -

            That's great, thanks!

            Show
            price Paul Price added a comment - That's great, thanks!
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks, Paul. Ran a Jenkins lsst_distrib + ci_hsc which passed and merged to master.

            Show
            lauren Lauren MacArthur added a comment - Thanks, Paul. Ran a Jenkins lsst_distrib + ci_hsc which passed and merged to master.

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Paul Price
                Watchers:
                Hsin-Fang Chiang, Jim Bosch, Lauren MacArthur, Paul Price, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel