Data ManagementIcon indicating the project type

Data Management

Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12310

Figure out why most HSC datasets start with photometry chi2==NaN

    XMLWordPrintable

    Details

    • Story Points:
      4
    • Epic Link:
    • Sprint:
      Alert Production F17 - 10, Alert Production F17 - 11
    • Team:
      Alert Production

      Description

      Nearly all many-visit HSC datasets that I've tried to process fail to fit photometry because the chi2 starts out as NaN, resulting in all stars being rejected as outliers and (after a long time) Eigen blowing up. We can't proceed with the photometric part of the meas_mosaic comparison until I can produce a fit.

      Some steps to try:

      1. make jointcal fail immediately if chi2 is NaN (write a testcase for this)
      2. fit the SimplePhotometryModel instead, to see if that doesn't produce NaN
      3. print various input values to look for NaNs
      4. Is it worth writing the astrometry output before starting the photometry fit, so we at least have it in case of failure?

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11785 Run jointcal on acceptance data

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11931 Fix jointcal exit status and doRaise handling

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12384 exit/raise when data is less than parameters

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12473 Add getParallacticAngle() to visitInfo

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-8058 make jointcal except early when required values are NaN

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-10941 astrometrySourceSelector should remove sources with NaN

          • To Do

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Have to fix this to get jointcal's photometry to work for HSC.

            Show
            Parejkoj John Parejko added a comment - Have to fix this to get jointcal's photometry to work for HSC.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Some of the tracts even result in initial chi2=NaN for astrometry. I've bisected one of them down to the offending visits (either of 32504 and 32506 will individually trigger it as well):

            jointcal.py /datasets/hsc/repo --rerun=DM-10404/SFM:private/parejkoj/DM-11785 -C=jointcalConfig-hsc-astrometry.py --id ccd=0..8^10..103 filter=HSC-Y tract=15832 field=SSP_WIDE visit=32504^32506  --longlog --no-versions

            Show
            Parejkoj John Parejko added a comment - Some of the tracts even result in initial chi2=NaN for astrometry. I've bisected one of them down to the offending visits (either of 32504 and 32506 will individually trigger it as well): jointcal.py /datasets/hsc/repo --rerun=DM-10404/SFM:private/parejkoj/DM-11785 -C=jointcalConfig-hsc-astrometry.py --id ccd=0..8^10..103 filter=HSC-Y tract=15832 field=SSP_WIDE visit=32504^32506 --longlog --no-versions
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            I've at least narrowed down one of the photometry NaNs (triggerable via the same two visits above) to coming from the refcat. Where it's coming from in the refcat, I don't know.

            Show
            Parejkoj John Parejko added a comment - I've at least narrowed down one of the photometry NaNs (triggerable via the same two visits above) to coming from the refcat. Where it's coming from in the refcat, I don't know.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Fix found for photometry. Rerunning data to see if that caught all the cases.

            Now debugging astrometry: in the astrometry case listed above, it appears the NaN is sneaking in via the refraction vector, but which element of visitInfo that's coming from I don't yet know.

            Show
            Parejkoj John Parejko added a comment - Fix found for photometry. Rerunning data to see if that caught all the cases. Now debugging astrometry: in the astrometry case listed above, it appears the NaN is sneaking in via the refraction vector, but which element of visitInfo that's coming from I don't yet know.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Yusra AlSayyad, do you mind taking on this review of some python/C++ changes in jointcal?

            Note that there was a testdata_jointcal branch, but I've deleted it on github because I'm not actually going to make fake data for this.

            Show
            Parejkoj John Parejko added a comment - Yusra AlSayyad , do you mind taking on this review of some python/C++ changes in jointcal? Note that there was a testdata_jointcal branch, but I've deleted it on github because I'm not actually going to make fake data for this.
            Hide
            Permalink
            yusra Yusra AlSayyad added a comment -

            I made a minor comment and had a question on github.

            Is updating the counts in the unit test all you needed to do to ensure it failed without your new fix? If yes, perhaps the unit test commit should be merged with the commit it corresponds to so that the package is buildable with every commit. (The commits don't look independent.)

            Show
            yusra Yusra AlSayyad added a comment - I made a minor comment and had a question on github. Is updating the counts in the unit test all you needed to do to ensure it failed without your new fix? If yes, perhaps the unit test commit should be merged with the commit it corresponds to so that the package is buildable with every commit. (The commits don't look independent.)
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Thanks for your comments: they spurred me to cleanup the names of the files and methods related to saving the chi2 contributions. I also did some more commit squashing, per the above.

            Jenkins run (will merge once it's happy): https://ci.lsst.codes/job/stack-os-matrix/26968/

            Show
            Parejkoj John Parejko added a comment - Thanks for your comments: they spurred me to cleanup the names of the files and methods related to saving the chi2 contributions. I also did some more commit squashing, per the above. Jenkins run (will merge once it's happy): https://ci.lsst.codes/job/stack-os-matrix/26968/
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Jenkins happy. Merged and done.

            Show
            Parejkoj John Parejko added a comment - Jenkins happy. Merged and done.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Yusra AlSayyad
                Watchers:
                John Parejko, John Swinbank, Lauren MacArthur, Simon Krughoff, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel