Details
-
Type:
RFC
-
Status: Withdrawn
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
In preparation for the next phase of DESC's DC2 processing, we are adding metacalibration (https://github.com/lsst-dm/meas_extensions_ngmix) to our workflow. This requires numba (https://github.com/numba/numba).
Supporting this processing would be much easier if numba were already part of the stack's conda environment, rather than trying to juggle DM dependencies as we attempt to install numba ourselves.
Please note that currently, we must use either numba==v0.40 or numba>=v0.43 due to a bug present in the intervening versions.
Attachments
Activity
The new conda environment is very close to be operative.
When testing the installation of numba on the new environment, no issue have been found.
The following packages are installed (linux platform):
- llvmlite pkgs/main/linux-64::llvmlite-0.28.0-py37hd408876_0
- numba pkgs/main/linux-64::numba-0.43.1-py37h962f231_0
Therefore I suggest waiting for the implementation of RFC-584 before deciding if numba need to be added to the conda environment or if it can be handled on top of it.
Heather Kelly can you please let us know whether we can withdraw this RFC for now?
Over the weekend, Johann Cohen-Tanugi ran another test of metacalibration which uses numba. While the evaluation of the output is ongoing, at the very least it appears that adding numba 0.43 to the CVMFS installation of DMstack worked as expected.
This RFC can be withdrawn for now.
Heather Kelly
added a comment - Over the weekend, Johann Cohen-Tanugi ran another test of metacalibration which uses numba. While the evaluation of the output is ongoing, at the very least it appears that adding numba 0.43 to the CVMFS installation of DMstack worked as expected.
This RFC can be withdrawn for now. Withdrawing as discussed.
As Tim Jenness said above, it is currently possible to add additional python packages to a given release of lsst_distrib via conda. We have been doing this with the cvmfs distribution to extend it with several packages useful for analysis (e.g. jupyter, ginga, etc.) using
conda install --no-update-deps <blah>
We have not had any issue with this mechanism.
The specific issue is that DESC needs numba 0.43. Currently (i.e. as of w_2019_11) it is not possible to install that specific version of numba because it requires libstdcxx-ng[version='>=7.3.0']. lsst_distrib itself requires libstdcxx-ng=7.2.0=hdf63c60_3. Therefore, when we ask conda to install numba, without updating the packages already installed, it installs the latest version compatible, that is numba 0.39.
I understand from RFC-584 that there are plans to update conda. As far as I can see, those plans should solve this specific issue with numba.