Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15040

lsst_dm_stack_demo drops assorted products from the environment, then fails


    • Type: Story
    • Status: Done
    • Priority: Undefined
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Templates:
    • Story Points:
    • Sprint:
      Arch 2018-10-15
    • Team:


      I installed lsst_distrib 16.0 following the instructions at https://pipelines.lsst.io/v/16-0/install/newinstall.html. I then tried to run the demo following the instructions at https://pipelines.lsst.io/v/16-0/install/demo.html. The result:

      [jds@magpie lsst_dm_stack_demo-16.0]$ ./bin/demo.sh 
      Setting up: astrometry_net_data             Flavor: DarwinX86  Version: LOCAL:/Users/jds/Projects/LSST/src/lsst_dm_stack_demo-16.0/astrometry_net_data
      Traceback (most recent call last):
        File "/Users/jds/Projects/LSST/stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/pipe_tasks/16.0+1/bin/processCcd.py", line 23, in <module>
          from lsst.pipe.tasks.processCcd import ProcessCcdTask
        File "/Users/jds/Projects/LSST/stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/pipe_tasks/16.0+1/python/lsst/pipe/tasks/processCcd.py", line 22, in <module>
          from lsst.ip.isr import IsrTask
        File "/Users/jds/Projects/LSST/stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/ip_isr/16.0+1/python/lsst/ip/isr/__init__.py", line 27, in <module>
          from .isrFunctions import *
        File "/Users/jds/Projects/LSST/stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/ip_isr/16.0+1/python/lsst/ip/isr/isrFunctions.py", line 26, in <module>
          import lsst.afw.geom as afwGeom
        File "/Users/jds/Projects/LSST/stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/afw/16.0+1/python/lsst/afw/__init__.py", line 23, in <module>
          import lsst.utils
        File "/Users/jds/Projects/LSST/stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/utils/16.0/python/lsst/utils/__init__.py", line 28, in <module>
          import lsst.pex.exceptions
        File "/Users/jds/Projects/LSST/stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/pex_exceptions/16.0/python/lsst/pex/exceptions/__init__.py", line 26, in <module>
          from .wrappers import *
        File "/Users/jds/Projects/LSST/stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/pex_exceptions/16.0/python/lsst/pex/exceptions/wrappers.py", line 33, in <module>
          from future.utils import with_metaclass
      ModuleNotFoundError: No module named 'future'

      (The above is on a Mac, but I get the same on Linux)

      The problem arises because of the following line in bin/demo.sh:

      # Tell the stack where to find astrometric reference catalogs
      setup --nolocks -v -r ./astrometry_net_data astrometry_net_data

      Having setup lsst_distrib, I already have a version of astrometry_net_data configured. It, in turn, depends on a bunch of other packages:

      $ eups list -m astrometry_net_data  | xargs cat
      if (type == exact) {
         setupRequired(sconsUtils      -j 16.0)
         setupRequired(scons           -j 3.0.0.lsst1+3)
         setupRequired(python          -j 0.0.8)
         setupRequired(python_future   -j 0.16.0+3)
         setupRequired(pytest          -j 3.2.0.lsst4+1)
         setupRequired(pytest_flake8   -j 0.9.1+8)
         setupRequired(flake8          -j 3.5.0+8)
         setupRequired(python_mccabe   -j 0.6.1+9)
         setupRequired(pycodestyle     -j 2.3.1+3)
         setupRequired(pyflakes        -j 1.6.0+2)
         setupRequired(pep8_naming     -j 0.4.1+3)
         setupRequired(pytest_xdist    -j 1.20.1.lsst4+1)
         setupRequired(pytest_forked   -j 0.2.lsst4+1)
         setupRequired(python_execnet  -j 1.4.1.lsst4+1)
         setupRequired(pytest_session2file -j 0.1.9+9)
         setupOptional(doxygen         -j 1.8.13.lsst2)
      } else {
         setupRequired(sconsUtils 16.0 [>= 16.0])

      The lsst_dm_stack_demo version of astrometry_net_data has no dependencies:

      $ cat astrometry_net_data/ups/astrometry_net_data.table 

      When I set up this version, all the dependencies are removed from my configured stack, and it breaks.

      An easy workaround is to change the line to read:

      setup --nolocks -v -j -r ./astrometry_net_data astrometry_net_data

      Which will not drop the dependencies of the old astrometry_net_data when setting up the new version.


          Container Issues

            Issue Links



                • Assignee:
                  tjenness Tim Jenness
                  swinbank John Swinbank
                  John Swinbank
                  Gabriele Comoretto, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Simon Krughoff, Tim Jenness
                • Votes:
                  0 Vote for this issue
                  6 Start watching this issue


                  • Created:

                    Summary Panel