Details
-
Type:
RFC
-
Status: Withdrawn
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
In preparation for the next phase of DESC's DC2 processing, we are adding metacalibration (https://github.com/lsst-dm/meas_extensions_ngmix) to our workflow. This requires numba (https://github.com/numba/numba).
Supporting this processing would be much easier if numba were already part of the stack's conda environment, rather than trying to juggle DM dependencies as we attempt to install numba ourselves.
Please note that currently, we must use either numba==v0.40 or numba>=v0.43 due to a bug present in the intervening versions.
As Tim Jenness said above, it is currently possible to add additional python packages to a given release of lsst_distrib via conda. We have been doing this with the cvmfs distribution to extend it with several packages useful for analysis (e.g. jupyter, ginga, etc.) using
conda install --no-update-deps <blah>
We have not had any issue with this mechanism.
The specific issue is that DESC needs numba 0.43. Currently (i.e. as of w_2019_11) it is not possible to install that specific version of numba because it requires libstdcxx-ng[version='>=7.3.0']. lsst_distrib itself requires libstdcxx-ng=7.2.0=hdf63c60_3. Therefore, when we ask conda to install numba, without updating the packages already installed, it installs the latest version compatible, that is numba 0.39.
I understand from RFC-584 that there are plans to update conda. As far as I can see, those plans should solve this specific issue with numba.