Current status is:

• matplotlib 3.1 breaks fgcmcal.
• lsst_distrib builds fine with everything updated except matplotlib.
• lsst_sims does not build because sncosmo does not work with Astropy 4.

I'm not sure what the status of lsst_sims is in terms of support. Since it uses the same conda env as lsst_distrib I assume we aren't allowed to break lsst_sims builds.

John Swinbank the stack seems to build with these versions now (one outstanding ticket that will be merged today for validate_drp). I'll do a final jenkins run.

If everything works with these versions, I'm happy. (I've not checked myself, but I have absolute faith in your ability to run Jenkins!!)

I note that this is bumping us to pyarrow 0.15.1, which means we can presumably close DM-22125 when this goes in (that's linked to the RFC, but not to this implementation ticket so I mention it here to make sure it doesn't get missed). I note your comment on GitHub, but I don't think we really care about the provenance of pyarrow as long as the tests pass.

Well, Jenkins has thrown a spanner into the mix. It seems that pybind11 tries to build against boost but since it doesn't depend upon boost it picks up the conda boost not the eups boost and then fails. If I add eups boost as a dependency then it still finds the conda boost – confusing because with our current env it also has a (slightly older) conda boost available (conda boost comes from pyarrow). I assume the problem is the different compilers changing ABI but I'm confused about how this works at all at the moment.

Progress report: we had a small problem with pybind11/numpy with the new boost but fixes have been merged (Jim Bosch was very helpful for that and reviewed those changes). Everything now works with the updated stack except that we are getting a surprising failure in ci_hsc_gen3 where the queries are returning additional (incorrect) matches. Matching sqlalchemy versions does not help. I'm investigating but I'm not an expert in registry so it's not clear if we will be able to update the conda env this week.

DM-23489 is a sqlite bug. I am dropping the sqlite version to 3.30.1 which works. I'm doing a Jenkins run and then will merge and update lsstsw/newinstall with the new conda ref.

Jenkins now passes with the new conda env so we are ready to merge. I will make the switch on Monday.

It also seems that the act of me testing the new env on Jenkins broke Jenkins for everyone else because of caching so apologies for that (DM-23492).