In addition, the "4sigma_outliers" decam astrometry test gives different results on centos6. When I dug into that (see DM-14552), there were differences in chi2 values that showed up before any outliers were rejected (different between macOS, centos6, and Ubuntu). Those differences went away with opt=0. There may be something pathological about the decam data in particular.

After updating the refcats to the HTM indexed version (which have a higher source density), the default constrained astrometry test also gives different results on mac vs. linux (on macOS, the no_rank_update and with rank update versions have identical results, on Linux they are different). I'm leaving further investigation of this to DM-17597.

I'm nervous about leaving this sort of debt behind.  The change of chi^2 at constant dof suggests that we've added and removed an object, and as the 2108 suggests that we have a tiny catalogue this should be easy to sort out.

I'm also worried about the opt=0 comment.  In the bad old days this could be a register/memory difference in precision, but I thought that had gone away.  In general things that change with optimisation level have a reasonably high probability of being bugs.

I had an interesting conversation with Mike Jarvis a while ago on Slack about our optimization settings, and I recall coming away surprised at how much -O3 was allowed to make floating point math architecture/compiler-dependent.  If there is no problem with -O2, I'd personally be content dropping this ticket with no further investigation -  but if that turns out to be the case, we may want to open a new ticket to investigate relaxing our default optimization settings more globally.

Whacking this into an F19 epic. I note the comments from various product owners above about how we shouldn't ignore issues like this, so I hope one of them will choose to prioritise it at a forthcoming planning meeting...!

John Swinbank: should I mark this "Done" or "Invalid"? After exploration as part of DM-17597, I have entirely removed the offending tests, and added a notes about the quality of the testdata to both `test_jointcal_decam.py` and the `testdata_jointcal/README.md`. There is still one DECam test in jointcal, of a very simple model, so at least we can still demonstrate that we can use DECam input.

The input DECam astrometry is quite catastrophically bad (plotting the sources on sky, the CCDs are far from rectangular in one visit), even after I reprocessed it with a new stack and Gaia DR2, so the jointcal fitter cannot possibly achieve an acceptable fit. I think we could potentially use the data to explore how jointcal handles failed fits, but that would be for another ticket.