Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22769

Please do not strip {{tests/}} from Pipelines Docker images

    Details

    • Templates:
    • Team:
      SQuaRE

      Description

      The lsstsqre/centos Docker images are explicitly constructed without the tests directory.

      Unfortunately, the tests for some packages rely on the contents of the tests directory in other packages. For example, when trying to build pipe_tasks against a Dockerized obs_base, I get:

      ____________________________________________ ReadDefectsTestCase.test_read_defects ____________________________________________
      [gw3] linux -- Python 3.7.2 /opt/lsst/software/stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/bin/python3.7
       
      self = <test_read_CuratedCalibs.ReadDefectsTestCase testMethod=test_read_defects>
       
          def setUp(self):
      >       butler = dafPersist.ButlerFactory(mapper=BaseMapper()).create()
       
      tests/test_read_CuratedCalibs.py:61: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      tests/test_read_CuratedCalibs.py:48: in __init__
          policy = dafPersist.Policy(os.path.join(ROOT, "BaseMapper.yaml"))
      /opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0-1-g6fe20d0+1/python/lsst/daf/persistence/policy.py:80: in __init__
          self.__initFromFile(other)
      /opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0-1-g6fe20d0+1/python/lsst/daf/persistence/policy.py:111: in __initFromFile
          self.__initFromYamlFile(path)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
       
      self = {}
      path = '/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_base/19.0.0-9-ge91d8c4+1/tests/BaseMapper.yaml'
       
          def __initFromYamlFile(self, path):
              """Opens a file at a given path and attempts to load it in from yaml.
          
              :param path:
              :return:
              """
      >       with open(path, 'r') as f:
      E       FileNotFoundError: [Errno 2] No such file or directory: '/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_base/19.0.0-9-ge91d8c4+1/tests/BaseMapper.yaml'
       
      /opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0-1-g6fe20d0+1/python/lsst/daf/persistence/policy.py:145: FileNotFoundError
      

      This is happening because $PIPE_TASKS_DIR/tests/test_read_CuratedCalibs.py depends upon $OBS_BASE_DIR/tests/BaseMapper.yaml, which has been removed from the Docker images.

      This renders the Docker images much less useful for development than they might otherwise be.

      I don't know what the original motivation for stripping tests was (just to save space?). In general, I'd suggest that the Docker images should contain exactly the contents of the packages published at eups.lsst.codes — if it's appropriate to strip something from the Docker image, it must be appropriate to strip it from the package, and vice versa. Please stop special-casing this directory in image construction.

      Adding Josh and Simon as watchers here, as respectively the author of the Docker image building code and the tests that are being broken.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                frossie Frossie Economou
                Reporter:
                swinbank John Swinbank
                Watchers:
                John Swinbank, Joshua Hoblitt, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel