Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-583

Add Python Package numba

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Withdrawn
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      In preparation for the next phase of DESC's DC2 processing, we are adding metacalibration (https://github.com/lsst-dm/meas_extensions_ngmix) to our workflow.  This requires numba (https://github.com/numba/numba).

      Supporting this processing would be much easier if numba were already part of the stack's conda environment, rather than trying to juggle DM dependencies as we attempt to install numba ourselves.

      Please note that currently, we must use either numba==v0.40 or numba>=v0.43 due to a bug present in the intervening versions.

        Attachments

          Issue Links

            Activity

            Hide
            FabioHernandez Fabio Hernandez added a comment -

            As Tim Jenness said above, it is currently possible to add additional python packages to a given release of lsst_distrib via conda. We have been doing this with the cvmfs distribution to extend it with several packages useful for analysis (e.g. jupyter, ginga, etc.) using

            conda install --no-update-deps <blah>

            We have not had any issue with this mechanism.

            The specific issue is that DESC needs numba 0.43. Currently (i.e. as of w_2019_11) it is not possible to install that specific version of numba because it requires libstdcxx-ng[version='>=7.3.0']. lsst_distrib itself requires libstdcxx-ng=7.2.0=hdf63c60_3. Therefore, when we ask conda to install numba, without updating the packages already installed, it installs the latest version compatible, that is numba 0.39.

            I understand from RFC-584 that there are plans to update conda. As far as I can see, those plans should solve this specific issue with numba.

            Show
            FabioHernandez Fabio Hernandez added a comment - As Tim Jenness said above, it is currently possible to add additional python packages to a given release of lsst_distrib via conda . We have been doing this with the cvmfs distribution to extend it with several packages useful for analysis (e.g. jupyter, ginga, etc.) using conda install --no-update-deps <blah> We have not had any issue with this mechanism. The specific issue is that DESC needs numba 0.43 . Currently (i.e. as of w_2019_11 ) it is not possible to install that specific version of numba because it requires libstdcxx-ng [version='>=7.3.0'] . lsst_distrib itself requires libstdcxx-ng=7.2.0=hdf63c60_3 . Therefore, when we ask conda to install numba , without updating the packages already installed, it installs the latest version compatible, that is numba 0.39 . I understand from RFC-584 that there are plans to update conda . As far as I can see, those plans should solve this specific issue with numba .
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            The new conda environment is very close to be operative.

            When testing the installation of numba on the new environment, no issue have been found.

            The following packages are installed (linux platform):

            • llvmlite pkgs/main/linux-64::llvmlite-0.28.0-py37hd408876_0
            • numba pkgs/main/linux-64::numba-0.43.1-py37h962f231_0

             

            Therefore I suggest waiting for the implementation of RFC-584 before deciding if numba need to be added to the conda environment or if it can be handled on top of it.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - The new conda environment is very close to be operative. When testing the installation of numba on the new environment, no issue have been found. The following packages are installed (linux platform): llvmlite pkgs/main/linux-64::llvmlite-0.28.0-py37hd408876_0 numba pkgs/main/linux-64::numba-0.43.1-py37h962f231_0   Therefore I suggest waiting for the implementation of RFC-584 before deciding if numba need to be added to the conda environment or if it can be handled on top of it.
            Hide
            tjenness Tim Jenness added a comment -

            Heather Kelly can you please let us know whether we can withdraw this RFC for now?

            Show
            tjenness Tim Jenness added a comment - Heather Kelly can you please let us know whether we can withdraw this RFC for now?
            Hide
            hkelly Heather Kelly added a comment -

            Over the weekend, Johann Cohen-Tanugi ran another test of metacalibration which uses numba. While the evaluation of the output is ongoing, at the very least it appears that adding numba 0.43 to the CVMFS installation of DMstack worked as expected. 

            This RFC can be withdrawn for now.

            Show
            hkelly Heather Kelly added a comment - Over the weekend, Johann Cohen-Tanugi ran another test of metacalibration which uses numba. While the evaluation of the output is ongoing, at the very least it appears that adding numba 0.43 to the CVMFS installation of DMstack worked as expected.  This RFC can be withdrawn for now.
            Hide
            tjenness Tim Jenness added a comment -

            Withdrawing as discussed.

            Show
            tjenness Tim Jenness added a comment - Withdrawing as discussed.

              People

              Assignee:
              gcomoretto Gabriele Comoretto [X] (Inactive)
              Reporter:
              hkelly Heather Kelly
              Watchers:
              Colin Slater, Dominique Boutigny, Fabio Hernandez, Frossie Economou, Gabriele Comoretto [X] (Inactive), Heather Kelly, James Chiang, Jim Bosch, Johann Cohen-Tanugi, John Parejko, John Swinbank, Kian-Tat Lim, Leanne Guy, Robert Lupton, Tim Jenness, Wil O'Mullane
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.