XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
4
• Sprint:
• Team:

## Description

Nearly all many-visit HSC datasets that I've tried to process fail to fit photometry because the chi2 starts out as NaN, resulting in all stars being rejected as outliers and (after a long time) Eigen blowing up. We can't proceed with the photometric part of the meas_mosaic comparison until I can produce a fit.

Some steps to try:

1. make jointcal fail immediately if chi2 is NaN (write a testcase for this)
2. fit the SimplePhotometryModel instead, to see if that doesn't produce NaN
3. print various input values to look for NaNs
4. Is it worth writing the astrometry output before starting the photometry fit, so we at least have it in case of failure?

## Activity

Hide
John Parejko added a comment -

Have to fix this to get jointcal's photometry to work for HSC.

Show
John Parejko added a comment - Have to fix this to get jointcal's photometry to work for HSC.
Hide
John Parejko added a comment -

Some of the tracts even result in initial chi2=NaN for astrometry. I've bisected one of them down to the offending visits (either of 32504 and 32506 will individually trigger it as well):

 jointcal.py /datasets/hsc/repo --rerun=DM-10404/SFM:private/parejkoj/DM-11785 -C=jointcalConfig-hsc-astrometry.py --id ccd=0..8^10..103 filter=HSC-Y tract=15832 field=SSP_WIDE visit=32504^32506 --longlog --no-versions 

Show
John Parejko added a comment - Some of the tracts even result in initial chi2=NaN for astrometry. I've bisected one of them down to the offending visits (either of 32504 and 32506 will individually trigger it as well): jointcal.py /datasets/hsc/repo --rerun=DM-10404/SFM:private/parejkoj/DM-11785 -C=jointcalConfig-hsc-astrometry.py --id ccd=0..8^10..103 filter=HSC-Y tract=15832 field=SSP_WIDE visit=32504^32506 --longlog --no-versions
Hide
John Parejko added a comment -

I've at least narrowed down one of the photometry NaNs (triggerable via the same two visits above) to coming from the refcat. Where it's coming from in the refcat, I don't know.

Show
John Parejko added a comment - I've at least narrowed down one of the photometry NaNs (triggerable via the same two visits above) to coming from the refcat. Where it's coming from in the refcat, I don't know.
Hide
John Parejko added a comment -

Fix found for photometry. Rerunning data to see if that caught all the cases.

Now debugging astrometry: in the astrometry case listed above, it appears the NaN is sneaking in via the refraction vector, but which element of visitInfo that's coming from I don't yet know.

Show
John Parejko added a comment - Fix found for photometry. Rerunning data to see if that caught all the cases. Now debugging astrometry: in the astrometry case listed above, it appears the NaN is sneaking in via the refraction vector, but which element of visitInfo that's coming from I don't yet know.
Hide
John Parejko added a comment -

Yusra AlSayyad, do you mind taking on this review of some python/C++ changes in jointcal?

Note that there was a testdata_jointcal branch, but I've deleted it on github because I'm not actually going to make fake data for this.

Show
John Parejko added a comment - Yusra AlSayyad , do you mind taking on this review of some python/C++ changes in jointcal? Note that there was a testdata_jointcal branch, but I've deleted it on github because I'm not actually going to make fake data for this.
Hide

Is updating the counts in the unit test all you needed to do to ensure it failed without your new fix? If yes, perhaps the unit test commit should be merged with the commit it corresponds to so that the package is buildable with every commit. (The commits don't look independent.)

Show
Yusra AlSayyad added a comment - I made a minor comment and had a question on github. Is updating the counts in the unit test all you needed to do to ensure it failed without your new fix? If yes, perhaps the unit test commit should be merged with the commit it corresponds to so that the package is buildable with every commit. (The commits don't look independent.)
Hide
John Parejko added a comment -

Thanks for your comments: they spurred me to cleanup the names of the files and methods related to saving the chi2 contributions. I also did some more commit squashing, per the above.

Jenkins run (will merge once it's happy): https://ci.lsst.codes/job/stack-os-matrix/26968/

Show
John Parejko added a comment - Thanks for your comments: they spurred me to cleanup the names of the files and methods related to saving the chi2 contributions. I also did some more commit squashing, per the above. Jenkins run (will merge once it's happy): https://ci.lsst.codes/job/stack-os-matrix/26968/
Hide
John Parejko added a comment -

Jenkins happy. Merged and done.

Show
John Parejko added a comment - Jenkins happy. Merged and done.

## People

• Assignee:
John Parejko
Reporter:
John Parejko
Reviewers:
Watchers:
John Parejko, John Swinbank, Lauren MacArthur, Simon Krughoff, Yusra AlSayyad