Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2917

obs_cfht unit tests are broken

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_cfht
    • Labels:
      None

      Description

      obs_cfht has one unit test "testButler" that uses git://git.lsstcorp.org/contrib/price/testdata_cfht. 4 of the tests fail, as shown below.

      In addition, testdata_cfht is huge, and the tests barely use any of it. It's worth considering making a new test repo that is smaller, or if the amount of data is small enough, move it into afwdata or obs_cfht itself.

      localhost$ tests/testButler.py 
      CameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3
      CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3
      ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3
      CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3
      ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3
      CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3
      ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3
      CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3
      .CameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3
      CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3
      E.
      ======================================================================
      ERROR: testBias (__main__.GetRawTestCase)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "tests/testButler.py", line 122, in testBias
          self.getDetrend("bias")
        File "tests/testButler.py", line 110, in getDetrend
          flat = self.butler.get(detrend, self.dataId, ccd=ccd)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 218, in get
          location = self.mapper.map(datasetType, dataId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py", line 116, in map
          return func(self.validate(dataId), write)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py", line 287, in mapClosure
          return mapping.map(mapper, dataId, write)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 118, in map
          actualId = self.need(self.keyDict.iterkeys(), dataId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 199, in need
          lookups = self.lookup(newProps, newId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 345, in lookup
          return Mapping.lookup(self, properties, newId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup
          where, self.range, values)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery
          c = self.conn.execute(cmd, values)
      OperationalError: no such column: extension
       
      ======================================================================
      ERROR: testFlat (__main__.GetRawTestCase)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "tests/testButler.py", line 117, in testFlat
          self.getDetrend("flat")
        File "tests/testButler.py", line 110, in getDetrend
          flat = self.butler.get(detrend, self.dataId, ccd=ccd)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 218, in get
          location = self.mapper.map(datasetType, dataId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py", line 116, in map
          return func(self.validate(dataId), write)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py", line 287, in mapClosure
          return mapping.map(mapper, dataId, write)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 118, in map
          actualId = self.need(self.keyDict.iterkeys(), dataId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 199, in need
          lookups = self.lookup(newProps, newId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 345, in lookup
          return Mapping.lookup(self, properties, newId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup
          where, self.range, values)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery
          c = self.conn.execute(cmd, values)
      OperationalError: no such column: extension
       
      ======================================================================
      ERROR: testFringe (__main__.GetRawTestCase)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "tests/testButler.py", line 127, in testFringe
          self.getDetrend("fringe")
        File "tests/testButler.py", line 110, in getDetrend
          flat = self.butler.get(detrend, self.dataId, ccd=ccd)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 218, in get
          location = self.mapper.map(datasetType, dataId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py", line 116, in map
          return func(self.validate(dataId), write)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py", line 287, in mapClosure
          return mapping.map(mapper, dataId, write)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 118, in map
          actualId = self.need(self.keyDict.iterkeys(), dataId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 199, in need
          lookups = self.lookup(newProps, newId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 345, in lookup
          return Mapping.lookup(self, properties, newId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup
          where, self.range, values)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery
          c = self.conn.execute(cmd, values)
      OperationalError: no such column: extension
       
      ======================================================================
      ERROR: testRaw (__main__.GetRawTestCase)
      Test retrieval of raw image
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "tests/testButler.py", line 101, in testRaw
          raw = self.butler.get("raw", self.dataId, ccd=ccd, immediate=True)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 244, in get
          return callback()
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 242, in <lambda>
          innerCallback(), dataId)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 238, in <lambda>
          callback = lambda: self._read(pythonType, location)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 426, in _read
          location.getCppType(), storageList, additionalData)
        File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/persistenceLib.py", line 1430, in unsafeRetrieve
          return _persistenceLib.Persistence_unsafeRetrieve(self, *args)
      FitsError: 
        File "src/fits.cc", line 1064, in lsst::afw::fits::Fits::Fits(const std::string &, const std::string &, int)
          cfitsio error: could not open the named file (104) : Opening file '/Users/rowen/LSST/code/testdata/testdata_cfht/DATA/raw/08BL05/w2.+2+2/2008-11-01/i2/1038843o.fits.fz[1]' with mode 'r' {0}
      lsst::afw::fits::FitsError: 'cfitsio error: could not open the named file (104) : Opening file '/Users/rowen/LSST/code/testdata/testdata_cfht/DATA/raw/08BL05/w2.+2+2/2008-11-01/i2/1038843o.fits.fz[1]' with mode 'r''
       
       
      ----------------------------------------------------------------------
      Ran 6 tests in 3.544s
       
      FAILED (errors=4)
      

        Attachments

          Issue Links

            Activity

            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Having no optional tests (as a long term goal) makes sense.

            When I think test data for an `obs_*` package the only thing that comes to mind is images from that telescope+camera. These images, in general, will be large for any large format imager.

            If we don't want to large test packages as part of testing an `obs_*` package, that would seem to rule out images.

            What kind of test data would that leave?

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Having no optional tests (as a long term goal) makes sense. When I think test data for an `obs_*` package the only thing that comes to mind is images from that telescope+camera. These images, in general, will be large for any large format imager. If we don't want to large test packages as part of testing an `obs_*` package, that would seem to rule out images. What kind of test data would that leave?
            Hide
            rowen Russell Owen added a comment - - edited

            I don't think the data needed to test obs_cfht has to be so large. I understand that natively CFHT writes single FITS files that contain all images from one focal plane, but surely it is practical to strip out all but one or two images (or, to avoid indexing issues, replace the earlier HDUs with tiny images or empty HDUs) and just fetch what remains.

            Ideally the data would becomes small enough that it could be included in the obs_cfht package itself, but MegaCam has large CCDs, so that may not be practical. If obs_cfht itself is allowed to use git-lfs for some files, then I think it would be acceptable to include a few images and that would be my recommendation.

            Another alternative is to generate data on the fly, but then the generating code would have to be carefully verified.

            Show
            rowen Russell Owen added a comment - - edited I don't think the data needed to test obs_cfht has to be so large. I understand that natively CFHT writes single FITS files that contain all images from one focal plane, but surely it is practical to strip out all but one or two images (or, to avoid indexing issues, replace the earlier HDUs with tiny images or empty HDUs) and just fetch what remains. Ideally the data would becomes small enough that it could be included in the obs_cfht package itself, but MegaCam has large CCDs, so that may not be practical. If obs_cfht itself is allowed to use git-lfs for some files, then I think it would be acceptable to include a few images and that would be my recommendation. Another alternative is to generate data on the fly, but then the generating code would have to be carefully verified.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Russell Owen I hear you about the longer-term goals.

            Do you have any objections to the pull request to at least just compress the files?

            https://github.com/lsst/testdata_cfht/pull/2

            Tim Jenness says that J Matt Peterson [X] says under git-lfs, it is helpful to compress files and that only files actually used by checked out versions get downloaded.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Russell Owen I hear you about the longer-term goals. Do you have any objections to the pull request to at least just compress the files? https://github.com/lsst/testdata_cfht/pull/2 Tim Jenness says that J Matt Peterson [X] says under git-lfs, it is helpful to compress files and that only files actually used by checked out versions get downloaded.
            Hide
            rowen Russell Owen added a comment -

            I have no objection to compressing the files (I assume it did saved enough space to be interesting).

            Show
            rowen Russell Owen added a comment - I have no objection to compressing the files (I assume it did saved enough space to be interesting).
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            I am taking this Issues as implemented. Thanks to all for the discussion.

            The outstanding unresolved issues is a larger discussion on what should obs_ packages test. I propose that we spend 6 months (perhaps 9 months include the next short+standard cycle) with new ci_ capabilities to give us a better picture of what the distinction between ci_ and obs_ tests should be and what the relationship between validation_data_ and testdata_ packages should be.

            If it becomes clear that we want to revisit the testdata_* packages at that time a new ticket asking to do so would be appropriate to submit.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - I am taking this Issues as implemented. Thanks to all for the discussion. The outstanding unresolved issues is a larger discussion on what should obs_ packages test. I propose that we spend 6 months (perhaps 9 months include the next short+standard cycle) with new ci_ capabilities to give us a better picture of what the distinction between ci_ and obs_ tests should be and what the relationship between validation_data_ and testdata_ packages should be. If it becomes clear that we want to revisit the testdata_* packages at that time a new ticket asking to do so would be appropriate to submit.

              People

              • Assignee:
                wmwood-vasey Michael Wood-Vasey
                Reporter:
                rowen Russell Owen
                Reviewers:
                Russell Owen
                Watchers:
                David Nidever [X] (Inactive), Dominique Boutigny, Hsin-Fang Chiang, Joshua Hoblitt, Michael Wood-Vasey, Paul Price, Russell Owen, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel