Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26657

Update python version to 3.8 in scipipe conda env

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: conda
    • Labels:
      None
    • Story Points:
      2
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Implement RFC-725 and update the base scipipe conda environment to python 3.8.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-25021 Upgrade to newer pybind11

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26675 Changes to validate_drp to support treecorr > 4

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27064 afw fails to build with pybind11 2.5 (and python 3.8)

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27364 ingestIndexManager does not set up global multiprocessing value before forking, breaking multiprocessing after python 3.8. on mac

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27377 fgcm multiprocessing strategy with shared memory manager doesn't work well with python 3.8

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27297 pybind11 errors out at runtime with afw because Background and BackgroundMI use different holders

          • Invalid
          is duplicated by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-25023 Upgrade to python 3.8

          • Invalid
          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-725 Update baseline python to version 3.8

          • Implemented
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26849 astshim incompatible with pybind11 2.5

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27005 Update scipipe_conda_env to distribute rubin-env

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27381 ctrl_mpexec fails (or may sometimes fail) on mac with python 3.8 in test_mpexec_faillure_failfast

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26870 errors when compiling afw with pybind11 for pixel types

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-25022 Upgrade to boost 1.72 or newer

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27436 Invetigate scarlet failures in python 3.8

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27631 use multiprocessing context with fork for ctrl_mpexec for python 3.8 support on mac

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            bvan Brian Van Klaveren added a comment -

            I think the following restrictions will need to be addressed to go to python 3.8:

             

            • pybind11 < 2.3

            • mpich = 3.2.1

            • treecorr < 4

             

             

            I will check on mpich because I think there was a bug but there's been new releases since then. pybind11 already has a ticket, but I don't think treecorr has a ticket right now.

            Show
            bvan Brian Van Klaveren added a comment - I think the following restrictions will need to be addressed to go to python 3.8:   • pybind11 < 2.3 • mpich = 3.2.1 • treecorr < 4     I will check on mpich because I think there was a bug but there's been new releases since then. pybind11 already has a ticket, but I don't think treecorr has a ticket right now.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            I'm sorry, I had not realized you already had a ticket for 3.8 – you didn't mention it on the RFC. Shall I close this one?

            Show
            tjenness Tim Jenness added a comment - I'm sorry, I had not realized you already had a ticket for 3.8 – you didn't mention it on the RFC. Shall I close this one?
            Hide
            Permalink
            tjenness Tim Jenness added a comment - - edited

            Is the treecorr issue an API change that breaks our usage or a python 3.8 issue?

            Show
            tjenness Tim Jenness added a comment - - edited Is the treecorr issue an API change that breaks our usage or a python 3.8 issue?
            Hide
            Permalink
            bvan Brian Van Klaveren added a comment -

            mpich may be blocked until 3.4 unless this issue is no longer relevant:

            https://jira.lsstcorp.org/browse/DM-24746

             

            (we don't have a unit test for that, it is something we discovered at runtime)

             

            I don't have any data on treecorr, I think it was an API change that broke things. I will test a build in jenkins with treecorr 4 right now

            Show
            bvan Brian Van Klaveren added a comment - mpich may be blocked until 3.4 unless this issue is no longer relevant: https://jira.lsstcorp.org/browse/DM-24746   (we don't have a unit test for that, it is something we discovered at runtime)   I don't have any data on treecorr, I think it was an API change that broke things. I will test a build in jenkins with treecorr 4 right now
            Hide
            Permalink
            bvan Brian Van Klaveren added a comment -

            A ticket for the treecorr migration has been created - there is a small API change between 3 and 4 for ggcorrelation, which validate_drp uses.

            Show
            bvan Brian Van Klaveren added a comment - A ticket for the treecorr migration has been created - there is a small API change between 3 and 4 for ggcorrelation, which validate_drp uses.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Did treecorr get fixed?

            Show
            tjenness Tim Jenness added a comment - Did treecorr get fixed?
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Has this ticket been made irrelevant by DM-27005 ?

            Show
            tjenness Tim Jenness added a comment - Has this ticket been made irrelevant by DM-27005 ?
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            We're not putting this into scipipe_conda_env, but into rubin-env instead. Still marking this done.

            Show
            ktl Kian-Tat Lim added a comment - We're not putting this into scipipe_conda_env, but into rubin-env instead. Still marking this done.

              People

              Assignee:
              bvan Brian Van Klaveren
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Brian Van Klaveren, Gabriele Comoretto [X] (Inactive), Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.