Details
-
Type:
RFC
-
Status: Withdrawn
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
In preparation for the next phase of DESC's DC2 processing, we are adding metacalibration (https://github.com/lsst-dm/meas_extensions_ngmix) to our workflow. This requires numba (https://github.com/numba/numba).
Supporting this processing would be much easier if numba were already part of the stack's conda environment, rather than trying to juggle DM dependencies as we attempt to install numba ourselves.
Please note that currently, we must use either numba==v0.40 or numba>=v0.43 due to a bug present in the intervening versions.
Attachments
Activity
I'm not fundamentally opposed to this, but given that that Numba is installable via conda, and (last I checked) the desired version of Numba is compatible with the conda package versions DM uses, I'm curious what's wrong with DESC (et al) installing Numba into their DM-generated conda stacks directly (with --no-update-deps). If our installer is producing a conda environment that doesn't support that kind of downstream extension, I think that's a serious problem.
If this is just about installing conda into DM-controlled conda environments (e.g. on lsst-dev), I think that's a separate issue (and one I'd wholeheartedly support).
I think that the LSP and shared stack on lsst-dev already have mechanisms for adding additional packages to the environment. I'm a bit worried that if we add packages to our base environment based on external users requests, then science pipelines packages will start to rely on them without anybody really noticing the growth of dependencies.
I think that the problem was that they couldn't just add numba due to dependency problems (there was a buggy release of numba). I agree that we have to avoid adding dependencies may not be desirable, and an alternative would have been to upgrade our conda – I'm not sure if that's better.
We hope to update our base conda environment in the next few days.
to add specifics as I understood them (Erin and Fabio are the experts) : numba 0.42 is buggy, and a 0.43 release was cut a few days ago which includes a bug fix. But libstdc++ dependency seems to force the install of 0.39 numba release on the CVMFS conda end.
Johann Cohen-Tanugi
added a comment - to add specifics as I understood them (Erin and Fabio are the experts) : numba 0.42 is buggy, and a 0.43 release was cut a few days ago which includes a bug fix. But libstdc++ dependency seems to force the install of 0.39 numba release on the CVMFS conda end. Generally I would be against asking DM to add random python packages that DESC just happens to use. In this case, we need numba to support the use of https://github.com/lsst-dm/meas_extensions_ngmix
Is that considered a DM software product, however loosely? If DESC is to run metacal, we have to have include numba. If using numba is not as simple as "conda install numba==blah" within the stack conda environment, then we need to have some clear support on which version(s) of numba to use that are compatible with the version(s) of the stack that DESC is using for DC2.
Johann reminds me that meas_extensions_ngmix also requires the addition of ngmix and I don't even know if it's sensible to ask that this be added too. numba is a true external dependency, while I'm not sure how to classify: https://github.com/esheldon/ngmix To me, that slides off the slippery slope. I think we (DESC) have to coordinate our installations of ngmix with Erin Sheldon for the purposes of DC2.
Heather Kelly
added a comment - - edited Generally I would be against asking DM to add random python packages that DESC just happens to use. In this case, we need numba to support the use of https://github.com/lsst-dm/meas_extensions_ngmix
Is that considered a DM software product, however loosely? If DESC is to run metacal, we have to have include numba. If using numba is not as simple as "conda install numba==blah" within the stack conda environment, then we need to have some clear support on which version(s) of numba to use that are compatible with the version(s) of the stack that DESC is using for DC2.
Johann reminds me that meas_extensions_ngmix also requires the addition of ngmix and I don't even know if it's sensible to ask that this be added too. numba is a true external dependency, while I'm not sure how to classify: https://github.com/esheldon/ngmix To me, that slides off the slippery slope. I think we (DESC) have to coordinate our installations of ngmix with Erin Sheldon for the purposes of DC2. ngmix needs numba and meas_extensions_ngmix is (currently) not an official LSST software product. If the request was for meas_extensions_ngmix to be included in lsst_distrib then this would be a very different conversation. Would your problems be solved in the short term with our entire conda environment being made current?
For the short term - having the conda environment being made current will probably help for now. The problem is that I fear this will be an ongoing issue as I do not know the development plans for meas_extensions_ngmix or ngmix (and maybe that's a conversation for elsewhere). If they need a further upgraded numba or ngmix between now and through the summer - we may be having this conversation again.
Ultimately, I hope meas_extensions_ngmix will become part of lsst_distrib - but currently I have no real window into that possibility!
Heather Kelly
added a comment - For the short term - having the conda environment being made current will probably help for now. The problem is that I fear this will be an ongoing issue as I do not know the development plans for meas_extensions_ngmix or ngmix (and maybe that's a conversation for elsewhere). If they need a further upgraded numba or ngmix between now and through the summer - we may be having this conversation again.
Ultimately, I hope meas_extensions_ngmix will become part of lsst_distrib - but currently I have no real window into that possibility! We do expect/hope to pull ngmix into the stack (if it works!), but that's a different RFC
As Tim Jenness said above, it is currently possible to add additional python packages to a given release of lsst_distrib via conda. We have been doing this with the cvmfs distribution to extend it with several packages useful for analysis (e.g. jupyter, ginga, etc.) using
conda install --no-update-deps <blah>
We have not had any issue with this mechanism.
The specific issue is that DESC needs numba 0.43. Currently (i.e. as of w_2019_11) it is not possible to install that specific version of numba because it requires libstdcxx-ng[version='>=7.3.0']. lsst_distrib itself requires libstdcxx-ng=7.2.0=hdf63c60_3. Therefore, when we ask conda to install numba, without updating the packages already installed, it installs the latest version compatible, that is numba 0.39.
I understand from RFC-584 that there are plans to update conda. As far as I can see, those plans should solve this specific issue with numba.
The new conda environment is very close to be operative.
When testing the installation of numba on the new environment, no issue have been found.
The following packages are installed (linux platform):
- llvmlite pkgs/main/linux-64::llvmlite-0.28.0-py37hd408876_0
- numba pkgs/main/linux-64::numba-0.43.1-py37h962f231_0
Therefore I suggest waiting for the implementation of RFC-584 before deciding if numba need to be added to the conda environment or if it can be handled on top of it.
Heather Kelly can you please let us know whether we can withdraw this RFC for now?
Over the weekend, Johann Cohen-Tanugi ran another test of metacalibration which uses numba. While the evaluation of the output is ongoing, at the very least it appears that adding numba 0.43 to the CVMFS installation of DMstack worked as expected.
This RFC can be withdrawn for now.
Heather Kelly
added a comment - Over the weekend, Johann Cohen-Tanugi ran another test of metacalibration which uses numba. While the evaluation of the output is ongoing, at the very least it appears that adding numba 0.43 to the CVMFS installation of DMstack worked as expected.
This RFC can be withdrawn for now. Withdrawing as discussed.
I support this RFC under the Krughoff Criterion (is it in pyviz.org? yes)