# Explore numeric difference in Mac/Linux ConstrainedPhotometry fitting

XMLWordPrintable

#### Details

• Type: Story
• Status: Invalid
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
8
• Team:

#### Description

While working on DM-14155, I discovered that Linux and Mac produce different results in the DECAM ConstrainedPhotometryModel tests. This may be a hint about why that model is unstable and not producing good fits, or it may be due to something else. Either way, I should understand what's going on, which is going to require a bunch of debug printing, and maybe looking at the matrix elements themselves.

As an example, after the error scales are frozen, the mac chi2/ndof is 5073.41/2108, while on linux it is 5072.94/2108. Subsequently, different outliers are rejected and the final fits are measureably different.

#### Activity

Hide
John Parejko added a comment -

In addition, the "4sigma_outliers" decam astrometry test gives different results on centos6. When I dug into that (see DM-14552), there were differences in chi2 values that showed up before any outliers were rejected (different between macOS, centos6, and Ubuntu). Those differences went away with opt=0. There may be something pathological about the decam data in particular.

Show
John Parejko added a comment - In addition, the "4sigma_outliers" decam astrometry test gives different results on centos6. When I dug into that (see DM-14552 ), there were differences in chi2 values that showed up before any outliers were rejected (different between macOS, centos6, and Ubuntu). Those differences went away with opt=0 . There may be something pathological about the decam data in particular.
Hide
John Parejko added a comment -

After updating the refcats to the HTM indexed version (which have a higher source density), the default constrained astrometry test also gives different results on mac vs. linux (on macOS, the no_rank_update and with rank update versions have identical results, on Linux they are different). I'm leaving further investigation of this to DM-17597.

Show
John Parejko added a comment - After updating the refcats to the HTM indexed version (which have a higher source density), the default constrained astrometry test also gives different results on mac vs. linux (on macOS, the no_rank_update and with rank update versions have identical results, on Linux they are different). I'm leaving further investigation of this to DM-17597 .
Hide
Robert Lupton added a comment -

I'm nervous about leaving this sort of debt behind.  The change of chi^2 at constant dof suggests that we've added and removed an object, and as the 2108 suggests that we have a tiny catalogue this should be easy to sort out.

I'm also worried about the opt=0 comment.  In the bad old days this could be a register/memory difference in precision, but I thought that had gone away.  In general things that change with optimisation level have a reasonably high probability of being bugs.

Show
Robert Lupton added a comment - I'm nervous about leaving this sort of debt behind.  The change of chi^2 at constant dof suggests that we've added and removed an object, and as the 2108 suggests that we have a tiny catalogue this should be easy to sort out. I'm also worried about the opt=0 comment.  In the bad old days this could be a register/memory difference in precision, but I thought that had gone away.  In general things that change with optimisation level have a reasonably high probability of being bugs.
Hide
Jim Bosch added a comment -

I had an interesting conversation with Mike Jarvis a while ago on Slack about our optimization settings, and I recall coming away surprised at how much -O3 was allowed to make floating point math architecture/compiler-dependent.  If there is no problem with -O2, I'd personally be content dropping this ticket with no further investigation -  but if that turns out to be the case, we may want to open a new ticket to investigate relaxing our default optimization settings more globally.

Show
Jim Bosch added a comment - I had an interesting conversation with Mike Jarvis a while ago on Slack about our optimization settings, and I recall coming away surprised at how much -O3 was allowed to make floating point math architecture/compiler-dependent.  If there is no problem with -O2, I'd personally be content dropping this ticket with no further investigation -  but if that turns out to be the case, we may want to open a new ticket to investigate relaxing our default optimization settings more globally.
Hide
John Swinbank added a comment -

Whacking this into an F19 epic. I note the comments from various product owners above about how we shouldn't ignore issues like this, so I hope one of them will choose to prioritise it at a forthcoming planning meeting...!

Show
John Swinbank added a comment - Whacking this into an F19 epic. I note the comments from various product owners above about how we shouldn't ignore issues like this, so I hope one of them will choose to prioritise it at a forthcoming planning meeting...!
Hide
John Parejko added a comment - - edited

John Swinbank: should I mark this "Done" or "Invalid"? After exploration as part of DM-17597, I have entirely removed the offending tests, and added a notes about the quality of the testdata to both test_jointcal_decam.py and the testdata_jointcal/README.md. There is still one DECam test in jointcal, of a very simple model, so at least we can still demonstrate that we can use DECam input.

The input DECam astrometry is quite catastrophically bad (plotting the sources on sky, the CCDs are far from rectangular in one visit), even after I reprocessed it with a new stack and Gaia DR2, so the jointcal fitter cannot possibly achieve an acceptable fit. I think we could potentially use the data to explore how jointcal handles failed fits, but that would be for another ticket.

Show
John Parejko added a comment - - edited John Swinbank : should I mark this "Done" or "Invalid"? After exploration as part of DM-17597 , I have entirely removed the offending tests, and added a notes about the quality of the testdata to both test_jointcal_decam.py and the testdata_jointcal/README.md. There is still one DECam test in jointcal, of a very simple model, so at least we can still demonstrate that we can use DECam input. The input DECam astrometry is quite catastrophically bad (plotting the sources on sky, the CCDs are far from rectangular in one visit), even after I reprocessed it with a new stack and Gaia DR2, so the jointcal fitter cannot possibly achieve an acceptable fit. I think we could potentially use the data to explore how jointcal handles failed fits, but that would be for another ticket.

#### People

Assignee:
John Parejko
Reporter:
John Parejko
Watchers:
Dominique Boutigny, Eli Rykoff, Jim Bosch, John Parejko, John Swinbank, Pierre Astier, Robert Lupton