Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16640

Raise when there are not enough matched stars in astrometric fitting

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: meas_astrom
    • Labels:
      None
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      There are occasional silent failures of the astrometry task in processCcd, that Dominique Boutigny has long complained about on Slack: e.g., https://lsstc.slack.com/archives/C978LTJGN/p1526484056000588 but that never got ticketed.

      The gist of the problem is that sometimes the astrometry fails badly (in particular in many DESC DC2 u-band ccds, but there are probably other examples) but the fitter thinks everything is fine.  This has led to Dominique Boutigny's script to check for these failures:  https://github.com/LSSTDESC/ImageProcessingPipelines/blob/master/python/util/checkCcdAstrometry.py (see DM-16301 for a ticket to add the logic from this script to processCcd).

      Recent investigation (see, e.g., https://lsstc.slack.com/archives/C978LTJGN/p1543335472430300 and following) makes it seem like the culprit for most (all?) of these silent failures is that the fitter is trying to fit more SIP terms than there are constraints from the stars.  When these stars are in only one section of the CCD, this leads to overfitting for the matched stars and wild extrapolations for other stars, leading to internal statistics from the fitter looking "good" but the overall fit being bad.

      (In general, if the default order of the SIP fit is reduced to 2 this seems to greatly increase the success rate on simulated u-band images).

      Specifically, for this ticket, I propose having the fitter raise an exception of there are not enough matched stars for the SIP order that was requested.  This would turn silent failures into loud failures, as well as alerting the user of where things were going wrong.

        Attachments

          Issue Links

            Activity

            erykoff Eli Rykoff created issue -
            rhl Robert Lupton made changes -
            Field Original Value New Value
            Watchers Dominique Boutigny, Eli Rykoff, John Parejko [ Dominique Boutigny, Eli Rykoff, John Parejko ] Dominique Boutigny, Eli Rykoff, John Parejko, John Swinbank [ Dominique Boutigny, Eli Rykoff, John Parejko, John Swinbank ]
            swinbank John Swinbank made changes -
            Description There are occasional silent failures of the astrometry task in processCcd, that [~boutigny] has long complained about on Slack: e.g., [https://lsstc.slack.com/archives/C978LTJGN/p1526484056000588] but that never got ticketed.

            The gist of the problem is that sometimes the astrometry fails badly (in particular in many DESC DC2 u-band ccds, but there are probably other examples) but the fitter things everything is fine.  This has led to [~boutigny]'s script to check for these failures:  [https://github.com/LSSTDESC/ImageProcessingPipelines/blob/master/python/util/checkCcdAstrometry.py] (see DM-16301 for a ticket to add the logic from this script to processCcd).

            Recent investigation (see, e.g., [https://lsstc.slack.com/archives/C978LTJGN/p1543335472430300] and following) makes it seem like the culprit for most (all?) of these silent failures is that the fitter is trying to fit more SIP terms than there are constraints from the stars.  When these stars are in only one section of the CCD, this leads to overfitting for the matched stars and wild extrapolations for other stars, leading to internal statistics from the fitter looking "good" but the overall fit being bad.

            (In general, if the default order of the SIP fit is reduced to 2 this seems to greatly increase the success rate on simulated u-band images).

            Specifically, for this ticket, I propose having the fitter raise an exception of there are not enough matched stars for the SIP order that was requested.  This would turn silent failures into loud failures, as well as alerting the user of where things were going wrong.
            There are occasional silent failures of the astrometry task in processCcd, that [~boutigny] has long complained about on Slack: e.g., [https://lsstc.slack.com/archives/C978LTJGN/p1526484056000588] but that never got ticketed.

            The gist of the problem is that sometimes the astrometry fails badly (in particular in many DESC DC2 u-band ccds, but there are probably other examples) but the fitter thinks everything is fine.  This has led to [~boutigny]'s script to check for these failures:  [https://github.com/LSSTDESC/ImageProcessingPipelines/blob/master/python/util/checkCcdAstrometry.py] (see DM-16301 for a ticket to add the logic from this script to processCcd).

            Recent investigation (see, e.g., [https://lsstc.slack.com/archives/C978LTJGN/p1543335472430300] and following) makes it seem like the culprit for most (all?) of these silent failures is that the fitter is trying to fit more SIP terms than there are constraints from the stars.  When these stars are in only one section of the CCD, this leads to overfitting for the matched stars and wild extrapolations for other stars, leading to internal statistics from the fitter looking "good" but the overall fit being bad.

            (In general, if the default order of the SIP fit is reduced to 2 this seems to greatly increase the success rate on simulated u-band images).

            Specifically, for this ticket, I propose having the fitter raise an exception of there are not enough matched stars for the SIP order that was requested.  This would turn silent failures into loud failures, as well as alerting the user of where things were going wrong.
            swinbank John Swinbank made changes -
            Summary There are occasional silent failures of the astrometry task in processCcd Raise when there are not enough matched stars in astrometric fitting
            swinbank John Swinbank made changes -
            Priority Undefined [ 10000 ] Major [ 3 ]
            swinbank John Swinbank made changes -
            Component/s meas_astrom [ 10745 ]
            Component/s Data Release Production [ 13901 ]
            swinbank John Swinbank made changes -
            Team Data Release Production [ 10301 ]
            lskelvin Lee Kelvin made changes -
            Assignee Lee Kelvin [ lskelvin ]
            Parejkoj John Parejko made changes -
            Watchers Dominique Boutigny, Eli Rykoff, John Parejko, John Swinbank, Lee Kelvin, Yusra AlSayyad [ Dominique Boutigny, Eli Rykoff, John Parejko, John Swinbank, Lee Kelvin, Yusra AlSayyad ] Chris Morrison, Dominique Boutigny, Eli Rykoff, John Parejko, John Swinbank, Lee Kelvin, Yusra AlSayyad [ Chris Morrison, Dominique Boutigny, Eli Rykoff, John Parejko, John Swinbank, Lee Kelvin, Yusra AlSayyad ]
            lskelvin Lee Kelvin made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-17737 [ DM-17737 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-16301 [ DM-16301 ]
            lskelvin Lee Kelvin made changes -
            Urgent? off
            lskelvin Lee Kelvin made changes -
            Status In Progress [ 3 ] To Do [ 10001 ]
            lskelvin Lee Kelvin made changes -
            Assignee Lee Kelvin [ lskelvin ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              erykoff Eli Rykoff
              Watchers:
              Chris Morrison [X] (Inactive), Dan Taranu, Dominique Boutigny, Eli Rykoff, John Parejko, Lee Kelvin, Yusra AlSayyad
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.