Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17924

Transition loadLSST.bash conda environment setup to current anaconda recommendations.

    XMLWordPrintable

    Details

    • Team:
      Architecture

      Description

      Currently, the setup to set the conda environment in loadLSST.bash looks something like (copied from my /cvmfs setup):

      export PATH="/cvmfs/sw.lsst.eu/darwin-x86_64/lsst_distrib/w_2019_06/python/miniconda3-4.5.4/bin:${PATH}"
      export LSST_CONDA_ENV_NAME=${LSST_CONDA_ENV_NAME:-lsst-scipipe-fcd27eb}
      source activate "$LSST_CONDA_ENV_NAME"

      This pattern is how it was supposed to be done for the  pre-conda 4.4 setup. The new recommended setup is described here:

      https://github.com/conda/conda/blob/master/CHANGELOG.md#440-2017-12-20

      Also, if you were just to add the path (like in the loadLSST.bash file) and then try the new conda activate command you will get this error message:

      CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
      If your shell is Bash or a Bourne variant, enable conda for the current user with

      $ echo ". /cvmfs/sw.lsst.eu/darwin-x86_64/lsst_distrib/w_2019_06/python/miniconda3-4.5.4/etc/profile.d/conda.sh" >> ~/.bash_profile

      or, for all users, enable conda with

      $ sudo ln -s /cvmfs/sw.lsst.eu/darwin-x86_64/lsst_distrib/w_2019_06/python/miniconda3-4.5.4/etc/profile.d/conda.sh /etc/profile.d/conda.sh

      The options above will permanently enable the 'conda' command, but they do NOT
      put conda's base (root) environment on PATH. To do so, run

      $ conda activate

      in your terminal, or to put the base environment on PATH permanently, run

      $ echo "conda activate" >> ~/.bash_profile

      Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
      your ~/.bash_profile file. You should manually remove the line that looks like

      export PATH="/cvmfs/sw.lsst.eu/darwin-x86_64/lsst_distrib/w_2019_06/python/miniconda3-4.5.4/bin:$PATH"

      ^^^ The above line should NO LONGER be in your ~/.bash_profile file! ^^^

      So, with the new scheme you should do something like this instead:

      # For Conda
      . ${LSST_DISTRIB}/python/current/etc/profile.d/conda.sh
      conda activate lsst-scipipe-fcd27eb

      The main difference is that conda is now defined as a function. In the setup above the base environment is always in the path (so conda can be found).  Aside from the other reasons anaconda wants you to switch, this means if there is some executable in the base environment it will be seen by the other environments too.

      I should note one other thing that I noticed in testing.  With conda 4.5.4 I often am not seeing the path completely reset when activating a new environment and often need to do "conda deactivate" first.  This didn't used to be the case. However, with conda 4.6 this problem completely goes away and everything acts as expected.

      I think the bottom line is that for the use case that only the scipipe environment is ever setup, the user doesn't use and switch to other environments, and the base environment is kept clean of packages you don't want the other environments to see, continuing to do things the old way will work as expected.

      However, for more complicated setups where environments are being switched between, we should  switch to the recommended setup and also update to conda 4.6. 

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Thanks for this. I didn't know about this change. I think we are planning to switch between multiple conda environments soon since we will need to keep around environments used for doing previous builds. I've added Gabriele Comoretto [X] and Joshua Hoblitt as watchers.

            Show
            tjenness Tim Jenness added a comment - Thanks for this. I didn't know about this change. I think we are planning to switch between multiple conda environments soon since we will need to keep around environments used for doing previous builds. I've added Gabriele Comoretto [X] and Joshua Hoblitt as watchers.
            Hide
            cwalter Chris Walter added a comment -

            Hi All,

            Since you are thinking about the conda environments going forward, I wanted to mention one other design criteria you might want to keep in mind. When running in a production environment or in batch mode on the grid etc I just setup the default scipipe environment as designed.  

            However, when working in an analysis/development environment using stack code I need a more complete/rich environment.  In the past I used my own anaconda paired with the stack for this. I recently learned that I can use the cvmfs distribution from IN2P3 and make my own environments which are put in ~/.conda.  

            Fabio Hernandez has handled this issue by adding extra packages to the scipipe environment which he distributes.  As more and more "users" start using the LSST software I think this will become an ever larger use case to it is good to also keep this in mind as you are thinking about what to put in the environments and how to organize them.

            Show
            cwalter Chris Walter added a comment - Hi All, Since you are thinking about the conda environments going forward, I wanted to mention one other design criteria you might want to keep in mind. When running in a production environment or in batch mode on the grid etc I just setup the default scipipe environment as designed.   However, when working in an analysis/development environment using stack code I need a more complete/rich environment.  In the past I used my own anaconda paired with the stack for this. I recently learned that I can use the cvmfs distribution from IN2P3 and make my own environments which are put in ~/.conda.   Fabio Hernandez has handled this issue by adding extra packages to the scipipe environment which he distributes.  As more and more "users" start using the LSST software I think this will become an ever larger use case to it is good to also keep this in mind as you are thinking about what to put in the environments and how to organize them.
            Hide
            swinbank John Swinbank added a comment -

            Frossie Economou, Kian-Tat Lim — I'm confused about whether this currently an Architecture or a SQuaRE responsibility. Can you clarify?

            Show
            swinbank John Swinbank added a comment - Frossie Economou , Kian-Tat Lim — I'm confused about whether this currently an Architecture or a SQuaRE responsibility. Can you clarify?
            Hide
            swinbank John Swinbank added a comment -

            Given the lack of input, I decided it was an Architecture responsibility. Congratulations, Kian-Tat Lim!

            Show
            swinbank John Swinbank added a comment - Given the lack of input, I decided it was an Architecture responsibility. Congratulations, Kian-Tat Lim !
            Hide
            cwalter Chris Walter added a comment -

            Since I see some new activity on this issue, I will note that since 4.6 the anaconda people made a few more changes.  What I wrote above will still work but they have a new system that puts everything into shell functions.  Mostly this was done to make it work with new/more shells I think.  

            There is a new 'conda init' command but this just puts a bunch of stuff into your .bashrc which checks for versions and then  sets things up in one of the three ways. The new way seems to be to add a line that looks like: (for example) 

            eval "$(Path/to/anaconda/bin/conda shell.bash hook)"

            You would follow this by your conda activate command.

            If you don't want the base env activated by default first you can add

            auto_activate_base: false

            to your .condarc

            More details here:

            https://askubuntu.com/a/1114436

            I haven't been able to play with this myself much since I mostly use the conda 4.5 version distributed with the stack and use the 

            // code placeholder
            # For Conda
            . ${LSST_DISTRIB}/python/current/etc/profile.d/conda.sh
            conda activate lsst-scipipe-1172c30
            

            setup I described above appropriate for 4.5

             

            Show
            cwalter Chris Walter added a comment - Since I see some new activity on this issue, I will note that since 4.6 the anaconda people made a few more changes.  What I wrote above will still work but they have a new system that puts everything into shell functions.  Mostly this was done to make it work with new/more shells I think.   There is a new 'conda init' command but this just puts a bunch of stuff into your .bashrc which checks for versions and then  sets things up in one of the three ways. The new way seems to be to add a line that looks like: (for example)  eval "$(Path/to/anaconda/bin/conda shell.bash hook)" You would follow this by your conda activate command. If you don't want the base env activated by default first you can add auto_activate_base: false to your .condarc More details here: https://askubuntu.com/a/1114436 I haven't been able to play with this myself much since I mostly use the conda 4.5 version distributed with the stack and use the  // code placeholder # For Conda . ${LSST_DISTRIB}/python/current/etc/profile.d/conda.sh conda activate lsst-scipipe-1172c30 setup I described above appropriate for 4.5  
            Hide
            ktl Kian-Tat Lim added a comment -

            We are using source $CONDA_DIR/etc/profile.d/conda.sh followed by conda activate throughout our scripts now.

            Show
            ktl Kian-Tat Lim added a comment - We are using source $CONDA_DIR/etc/profile.d/conda.sh followed by conda activate throughout our scripts now.

              People

              Assignee:
              ktl Kian-Tat Lim
              Reporter:
              cwalter Chris Walter
              Watchers:
              Chris Walter, Fabio Hernandez, Gabriele Comoretto [X] (Inactive), Heather Kelly, Joshua Hoblitt, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.