Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-583

Add Python Package numba

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Withdrawn
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      In preparation for the next phase of DESC's DC2 processing, we are adding metacalibration (https://github.com/lsst-dm/meas_extensions_ngmix) to our workflow.  This requires numba (https://github.com/numba/numba).

      Supporting this processing would be much easier if numba were already part of the stack's conda environment, rather than trying to juggle DM dependencies as we attempt to install numba ourselves.

      Please note that currently, we must use either numba==v0.40 or numba>=v0.43 due to a bug present in the intervening versions.

        Attachments

          Issue Links

            Activity

            hkelly Heather Kelly created issue -
            hkelly Heather Kelly made changes -
            Field Original Value New Value
            Assignee Heather Kelly [ hkelly ] Gabriele Comoretto [ gcomoretto ]
            hkelly Heather Kelly made changes -
            Watchers Heather Kelly [ Heather Kelly ] Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Johann Cohen-Tanugi, Robert Lupton [ Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Johann Cohen-Tanugi, Robert Lupton ]
            tjenness Tim Jenness made changes -
            Watchers Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Johann Cohen-Tanugi, Robert Lupton, Tim Jenness [ Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Johann Cohen-Tanugi, Robert Lupton, Tim Jenness ] Colin Slater, Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Johann Cohen-Tanugi, John Swinbank, Kian-Tat Lim, Leanne Guy, Robert Lupton, Tim Jenness, Wil O'Mullane [ Colin Slater, Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Johann Cohen-Tanugi, John Swinbank, Kian-Tat Lim, Leanne Guy, Robert Lupton, Tim Jenness, Wil O'Mullane ]
            tjenness Tim Jenness made changes -
            Watchers Colin Slater, Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Johann Cohen-Tanugi, John Swinbank, Kian-Tat Lim, Leanne Guy, Robert Lupton, Tim Jenness, Wil O'Mullane [ Colin Slater, Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Johann Cohen-Tanugi, John Swinbank, Kian-Tat Lim, Leanne Guy, Robert Lupton, Tim Jenness, Wil O'Mullane ] Colin Slater, Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Jim Bosch, Johann Cohen-Tanugi, John Swinbank, Kian-Tat Lim, Leanne Guy, Robert Lupton, Tim Jenness, Wil O'Mullane [ Colin Slater, Dominique Boutigny, Fabio Hernandez, Heather Kelly, James Chiang, Jim Bosch, Johann Cohen-Tanugi, John Swinbank, Kian-Tat Lim, Leanne Guy, Robert Lupton, Tim Jenness, Wil O'Mullane ]
            gcomoretto Gabriele Comoretto (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 20088 ]
            Hide
            frossie Frossie Economou added a comment -

            I support this RFC under the Krughoff Criterion (is it in pyviz.org? yes)

            Show
            frossie Frossie Economou added a comment - I support this RFC under the Krughoff Criterion (is it in pyviz.org? yes)
            Hide
            jbosch Jim Bosch added a comment - - edited

            I'm not fundamentally opposed to this, but given that that Numba is installable via conda, and (last I checked) the desired version of Numba is compatible with the conda package versions DM uses, I'm curious what's wrong with DESC (et al) installing Numba into their DM-generated conda stacks directly (with --no-update-deps).  If our installer is producing a conda environment that doesn't support that kind of downstream extension, I think that's a serious problem.

            If this is just about installing conda into DM-controlled conda environments (e.g. on lsst-dev), I think that's a separate issue (and one I'd wholeheartedly support).

            Show
            jbosch Jim Bosch added a comment - - edited I'm not fundamentally opposed to this, but given that that Numba is installable via conda, and (last I checked) the desired version of Numba is compatible with the conda package versions DM uses, I'm curious what's wrong with DESC (et al) installing Numba into their DM-generated conda stacks directly (with --no-update-deps ).  If our installer is producing a conda environment that doesn't support that kind of downstream extension, I think that's a serious problem. If this is just about installing conda into DM-controlled conda environments (e.g. on lsst-dev), I think that's a separate issue (and one I'd wholeheartedly support).
            Hide
            tjenness Tim Jenness added a comment -

            I think that the LSP and shared stack on lsst-dev already have mechanisms for adding additional packages to the environment. I'm a bit worried that if we add packages to our base environment based on external users requests, then science pipelines packages will start to rely on them without anybody really noticing the growth of dependencies.

            Show
            tjenness Tim Jenness added a comment - I think that the LSP and shared stack on lsst-dev already have mechanisms for adding additional packages to the environment. I'm a bit worried that if we add packages to our base environment based on external users requests, then science pipelines packages will start to rely on them without anybody really noticing the growth of dependencies.
            Hide
            rhl Robert Lupton added a comment -

            I think that the problem was that they couldn't just add numba due to dependency problems (there was a buggy release of numba). I agree that we have to avoid adding dependencies may not be desirable, and an alternative would have been to upgrade our conda – I'm not sure if that's better.

            Show
            rhl Robert Lupton added a comment - I think that the problem was that they couldn't just add numba due to dependency problems (there was a buggy release of numba ). I agree that we have to avoid adding dependencies may not be desirable, and an alternative would have been to upgrade our conda – I'm not sure if that's better.
            Hide
            tjenness Tim Jenness added a comment -

            We hope to update our base conda environment in the next few days.

            Show
            tjenness Tim Jenness added a comment - We hope to update our base conda environment in the next few days.
            Hide
            cohen Johann Cohen-Tanugi added a comment -

            to add specifics as I understood them (Erin and Fabio are the experts) : numba 0.42 is buggy, and a 0.43 release was cut a few days ago which includes a bug fix. But libstdc++ dependency seems to force the install of 0.39 numba release on the CVMFS conda end.

            Show
            cohen Johann Cohen-Tanugi added a comment - to add specifics as I understood them (Erin and Fabio are the experts) : numba 0.42 is buggy, and a 0.43 release was cut a few days ago which includes a bug fix. But libstdc++ dependency seems to force the install of 0.39 numba release on the CVMFS conda end.
            Hide
            hkelly Heather Kelly added a comment - - edited

            Generally I would be against asking DM to add random python packages that DESC just happens to use.  In this case, we need numba to support the use of https://github.com/lsst-dm/meas_extensions_ngmix
            Is that considered a DM software product, however loosely? If DESC is to run metacal, we have to have include numba.  If using numba is not as simple as "conda install numba==blah" within the stack conda environment, then we need to have some clear support on which version(s) of numba to use that are compatible with the version(s) of the stack that DESC is using for DC2.

            Johann reminds me that meas_extensions_ngmix also requires the addition of ngmix and I don't even know if it's sensible to ask that this be added too.  numba is a true external dependency, while I'm not sure how to classify: https://github.com/esheldon/ngmix  To me, that slides off the slippery slope.  I think we (DESC) have to coordinate our installations of ngmix with Erin Sheldon for the purposes of DC2.

            Show
            hkelly Heather Kelly added a comment - - edited Generally I would be against asking DM to add random python packages that DESC just happens to use.  In this case, we need numba to support the use of https://github.com/lsst-dm/meas_extensions_ngmix Is that considered a DM software product, however loosely? If DESC is to run metacal, we have to have include numba.  If using numba is not as simple as "conda install numba==blah" within the stack conda environment, then we need to have some clear support on which version(s) of numba to use that are compatible with the version(s) of the stack that DESC is using for DC2. Johann reminds me that meas_extensions_ngmix also requires the addition of ngmix and I don't even know if it's sensible to ask that this be added too.  numba is a true external dependency, while I'm not sure how to classify: https://github.com/esheldon/ngmix   To me, that slides off the slippery slope.  I think we (DESC) have to coordinate our installations of ngmix with Erin Sheldon for the purposes of DC2.
            Hide
            tjenness Tim Jenness added a comment -

            ngmix needs numba and meas_extensions_ngmix is (currently) not an official LSST software product. If the request was for meas_extensions_ngmix to be included in lsst_distrib then this would be a very different conversation. Would your problems be solved in the short term with our entire conda environment being made current?

            Show
            tjenness Tim Jenness added a comment - ngmix needs numba and meas_extensions_ngmix is (currently) not an official LSST software product. If the request was for meas_extensions_ngmix to be included in lsst_distrib then this would be a very different conversation. Would your problems be solved in the short term with our entire conda environment being made current?
            Hide
            hkelly Heather Kelly added a comment -

            For the short term - having the conda environment being made current will probably help for now.  The problem is that I fear this will be an ongoing issue as I do not know the development plans for meas_extensions_ngmix or ngmix (and maybe that's a conversation for elsewhere). If they need a further upgraded numba or ngmix between now and through the summer - we may be having this conversation again.

            Ultimately, I hope meas_extensions_ngmix will become part of lsst_distrib - but currently I have no real window into that possibility!

            Show
            hkelly Heather Kelly added a comment - For the short term - having the conda environment being made current will probably help for now.  The problem is that I fear this will be an ongoing issue as I do not know the development plans for meas_extensions_ngmix or ngmix (and maybe that's a conversation for elsewhere). If they need a further upgraded numba or ngmix between now and through the summer - we may be having this conversation again. Ultimately, I hope meas_extensions_ngmix will become part of lsst_distrib - but currently I have no real window into that possibility!
            Hide
            rhl Robert Lupton added a comment -

            We do expect/hope to pull ngmix into the stack (if it works!), but that's a different RFC

            Show
            rhl Robert Lupton added a comment - We do expect/hope to pull ngmix into the stack (if it works!), but that's a different RFC
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            As Tim Jenness said above, it is currently possible to add additional python packages to a given release of lsst_distrib via conda. We have been doing this with the cvmfs distribution to extend it with several packages useful for analysis (e.g. jupyter, ginga, etc.) using

            conda install --no-update-deps <blah>

            We have not had any issue with this mechanism.

            The specific issue is that DESC needs numba 0.43. Currently (i.e. as of w_2019_11) it is not possible to install that specific version of numba because it requires libstdcxx-ng[version='>=7.3.0']. lsst_distrib itself requires libstdcxx-ng=7.2.0=hdf63c60_3. Therefore, when we ask conda to install numba, without updating the packages already installed, it installs the latest version compatible, that is numba 0.39.

            I understand from RFC-584 that there are plans to update conda. As far as I can see, those plans should solve this specific issue with numba.

            Show
            FabioHernandez Fabio Hernandez added a comment - As Tim Jenness said above, it is currently possible to add additional python packages to a given release of lsst_distrib via conda . We have been doing this with the cvmfs distribution to extend it with several packages useful for analysis (e.g. jupyter, ginga, etc.) using conda install --no-update-deps <blah> We have not had any issue with this mechanism. The specific issue is that DESC needs numba 0.43 . Currently (i.e. as of w_2019_11 ) it is not possible to install that specific version of numba because it requires libstdcxx-ng [version='>=7.3.0'] . lsst_distrib itself requires libstdcxx-ng=7.2.0=hdf63c60_3 . Therefore, when we ask conda to install numba , without updating the packages already installed, it installs the latest version compatible, that is numba 0.39 . I understand from RFC-584 that there are plans to update conda . As far as I can see, those plans should solve this specific issue with numba .
            gcomoretto Gabriele Comoretto (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 20137 ]
            Hide
            gcomoretto Gabriele Comoretto (Inactive) added a comment -

            The new conda environment is very close to be operative.

            When testing the installation of numba on the new environment, no issue have been found.

            The following packages are installed (linux platform):

            • llvmlite pkgs/main/linux-64::llvmlite-0.28.0-py37hd408876_0
            • numba pkgs/main/linux-64::numba-0.43.1-py37h962f231_0

             

            Therefore I suggest waiting for the implementation of RFC-584 before deciding if numba need to be added to the conda environment or if it can be handled on top of it.

            Show
            gcomoretto Gabriele Comoretto (Inactive) added a comment - The new conda environment is very close to be operative. When testing the installation of numba on the new environment, no issue have been found. The following packages are installed (linux platform): llvmlite pkgs/main/linux-64::llvmlite-0.28.0-py37hd408876_0 numba pkgs/main/linux-64::numba-0.43.1-py37h962f231_0   Therefore I suggest waiting for the implementation of RFC-584 before deciding if numba need to be added to the conda environment or if it can be handled on top of it.
            gcomoretto Gabriele Comoretto (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 20173 ]
            Hide
            tjenness Tim Jenness added a comment -

            Heather Kelly can you please let us know whether we can withdraw this RFC for now?

            Show
            tjenness Tim Jenness added a comment - Heather Kelly can you please let us know whether we can withdraw this RFC for now?
            Hide
            hkelly Heather Kelly added a comment -

            Over the weekend, Johann Cohen-Tanugi ran another test of metacalibration which uses numba. While the evaluation of the output is ongoing, at the very least it appears that adding numba 0.43 to the CVMFS installation of DMstack worked as expected. 

            This RFC can be withdrawn for now.

            Show
            hkelly Heather Kelly added a comment - Over the weekend, Johann Cohen-Tanugi ran another test of metacalibration which uses numba. While the evaluation of the output is ongoing, at the very least it appears that adding numba 0.43 to the CVMFS installation of DMstack worked as expected.  This RFC can be withdrawn for now.
            Hide
            tjenness Tim Jenness added a comment -

            Withdrawing as discussed.

            Show
            tjenness Tim Jenness added a comment - Withdrawing as discussed.
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Proposed [ 10805 ] Withdrawn [ 10605 ]
            gcomoretto Gabriele Comoretto (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 20321 ]

              People

              Assignee:
              gcomoretto Gabriele Comoretto (Inactive)
              Reporter:
              hkelly Heather Kelly
              Watchers:
              Colin Slater, Dominique Boutigny, Fabio Hernandez, Frossie Economou, Gabriele Comoretto (Inactive), Heather Kelly, James Chiang, Jim Bosch, Johann Cohen-Tanugi, John Parejko, John Swinbank, Kian-Tat Lim, Leanne Guy, Robert Lupton, Tim Jenness, Wil O'Mullane
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End: