Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22817

Update conda base environment

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Third Party Software
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      Arch 2019-12-16
    • Team:
      Architecture

      Description

      It's time to refresh the base conda environment to make use of Astropy 4 and the like. We generally do this once a year.

      • Do a test build with a bleeding edge conda env.
      • File an RFC for the updates.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Current status is:

            • matplotlib 3.1 breaks fgcmcal.
            • lsst_distrib builds fine with everything updated except matplotlib.
            • lsst_sims does not build because sncosmo does not work with Astropy 4.

            I'm not sure what the status of lsst_sims is in terms of support. Since it uses the same conda env as lsst_distrib I assume we aren't allowed to break lsst_sims builds.

            Show
            tjenness Tim Jenness added a comment - Current status is: matplotlib 3.1 breaks fgcmcal. lsst_distrib builds fine with everything updated except matplotlib. lsst_sims does not build because sncosmo does not work with Astropy 4. I'm not sure what the status of lsst_sims is in terms of support. Since it uses the same conda env as lsst_distrib I assume we aren't allowed to break lsst_sims builds.
            Hide
            tjenness Tim Jenness added a comment -

            John Swinbank the stack seems to build with these versions now (one outstanding ticket that will be merged today for validate_drp). I'll do a final jenkins run.

            Show
            tjenness Tim Jenness added a comment - John Swinbank the stack seems to build with these versions now (one outstanding ticket that will be merged today for validate_drp). I'll do a final jenkins run.
            Hide
            swinbank John Swinbank added a comment -

            If everything works with these versions, I'm happy. (I've not checked myself, but I have absolute faith in your ability to run Jenkins!!)

            I note that this is bumping us to pyarrow 0.15.1, which means we can presumably close DM-22125 when this goes in (that's linked to the RFC, but not to this implementation ticket so I mention it here to make sure it doesn't get missed). I note your comment on GitHub, but I don't think we really care about the provenance of pyarrow as long as the tests pass.

            Show
            swinbank John Swinbank added a comment - If everything works with these versions, I'm happy. (I've not checked myself, but I have absolute faith in your ability to run Jenkins!!) I note that this is bumping us to pyarrow 0.15.1, which means we can presumably close DM-22125 when this goes in (that's linked to the RFC, but not to this implementation ticket so I mention it here to make sure it doesn't get missed). I note your comment on GitHub, but I don't think we really care about the provenance of pyarrow as long as the tests pass.
            Hide
            tjenness Tim Jenness added a comment -

            Well, Jenkins has thrown a spanner into the mix. It seems that pybind11 tries to build against boost but since it doesn't depend upon boost it picks up the conda boost not the eups boost and then fails. If I add eups boost as a dependency then it still finds the conda boost – confusing because with our current env it also has a (slightly older) conda boost available (conda boost comes from pyarrow). I assume the problem is the different compilers changing ABI but I'm confused about how this works at all at the moment.

            Show
            tjenness Tim Jenness added a comment - Well, Jenkins has thrown a spanner into the mix. It seems that pybind11 tries to build against boost but since it doesn't depend upon boost it picks up the conda boost not the eups boost and then fails. If I add eups boost as a dependency then it still finds the conda boost – confusing because with our current env it also has a (slightly older) conda boost available (conda boost comes from pyarrow). I assume the problem is the different compilers changing ABI but I'm confused about how this works at all at the moment.
            Hide
            tjenness Tim Jenness added a comment -

            Progress report: we had a small problem with pybind11/numpy with the new boost but fixes have been merged (Jim Bosch was very helpful for that and reviewed those changes). Everything now works with the updated stack except that we are getting a surprising failure in ci_hsc_gen3 where the queries are returning additional (incorrect) matches. Matching sqlalchemy versions does not help. I'm investigating but I'm not an expert in registry so it's not clear if we will be able to update the conda env this week.

            Show
            tjenness Tim Jenness added a comment - Progress report: we had a small problem with pybind11/numpy with the new boost but fixes have been merged ( Jim Bosch was very helpful for that and reviewed those changes). Everything now works with the updated stack except that we are getting a surprising failure in ci_hsc_gen3 where the queries are returning additional (incorrect) matches. Matching sqlalchemy versions does not help. I'm investigating but I'm not an expert in registry so it's not clear if we will be able to update the conda env this week.
            Hide
            tjenness Tim Jenness added a comment -

            DM-23489 is a sqlite bug. I am dropping the sqlite version to 3.30.1 which works. I'm doing a Jenkins run and then will merge and update lsstsw/newinstall with the new conda ref.

            Show
            tjenness Tim Jenness added a comment - DM-23489 is a sqlite bug. I am dropping the sqlite version to 3.30.1 which works. I'm doing a Jenkins run and then will merge and update lsstsw/newinstall with the new conda ref.
            Hide
            tjenness Tim Jenness added a comment -

            Jenkins now passes with the new conda env so we are ready to merge. I will make the switch on Monday.

            It also seems that the act of me testing the new env on Jenkins broke Jenkins for everyone else because of caching so apologies for that (DM-23492).

            Show
            tjenness Tim Jenness added a comment - Jenkins now passes with the new conda env so we are ready to merge. I will make the switch on Monday. It also seems that the act of me testing the new env on Jenkins broke Jenkins for everyone else because of caching so apologies for that ( DM-23492 ).

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                tjenness Tim Jenness
                Reviewers:
                John Swinbank
                Watchers:
                Brian Van Klaveren, Gabriele Comoretto, John Swinbank, Kian-Tat Lim, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel