# obs_cfht unit tests are broken

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
SQuaRE

#### Description

obs_cfht has one unit test "testButler" that uses git://git.lsstcorp.org/contrib/price/testdata_cfht. 4 of the tests fail, as shown below.

In addition, testdata_cfht is huge, and the tests barely use any of it. It's worth considering making a new test repo that is smaller, or if the amount of data is small enough, move it into afwdata or obs_cfht itself.

 localhost\$ tests/testButler.py  CameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 .CameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 E. ====================================================================== ERROR: testBias (__main__.GetRawTestCase) ---------------------------------------------------------------------- Traceback (most recent call last):  File "tests/testButler.py", line 122, in testBias  self.getDetrend("bias")  File "tests/testButler.py", line 110, in getDetrend  flat = self.butler.get(detrend, self.dataId, ccd=ccd)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 218, in get  location = self.mapper.map(datasetType, dataId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py", line 116, in map  return func(self.validate(dataId), write)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py", line 287, in mapClosure  return mapping.map(mapper, dataId, write)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 118, in map  actualId = self.need(self.keyDict.iterkeys(), dataId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 199, in need  lookups = self.lookup(newProps, newId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 345, in lookup  return Mapping.lookup(self, properties, newId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup  where, self.range, values)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery  c = self.conn.execute(cmd, values) OperationalError: no such column: extension   ====================================================================== ERROR: testFlat (__main__.GetRawTestCase) ---------------------------------------------------------------------- Traceback (most recent call last):  File "tests/testButler.py", line 117, in testFlat  self.getDetrend("flat")  File "tests/testButler.py", line 110, in getDetrend  flat = self.butler.get(detrend, self.dataId, ccd=ccd)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 218, in get  location = self.mapper.map(datasetType, dataId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py", line 116, in map  return func(self.validate(dataId), write)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py", line 287, in mapClosure  return mapping.map(mapper, dataId, write)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 118, in map  actualId = self.need(self.keyDict.iterkeys(), dataId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 199, in need  lookups = self.lookup(newProps, newId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 345, in lookup  return Mapping.lookup(self, properties, newId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup  where, self.range, values)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery  c = self.conn.execute(cmd, values) OperationalError: no such column: extension   ====================================================================== ERROR: testFringe (__main__.GetRawTestCase) ---------------------------------------------------------------------- Traceback (most recent call last):  File "tests/testButler.py", line 127, in testFringe  self.getDetrend("fringe")  File "tests/testButler.py", line 110, in getDetrend  flat = self.butler.get(detrend, self.dataId, ccd=ccd)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 218, in get  location = self.mapper.map(datasetType, dataId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py", line 116, in map  return func(self.validate(dataId), write)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py", line 287, in mapClosure  return mapping.map(mapper, dataId, write)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 118, in map  actualId = self.need(self.keyDict.iterkeys(), dataId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 199, in need  lookups = self.lookup(newProps, newId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 345, in lookup  return Mapping.lookup(self, properties, newId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup  where, self.range, values)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery  c = self.conn.execute(cmd, values) OperationalError: no such column: extension   ====================================================================== ERROR: testRaw (__main__.GetRawTestCase) Test retrieval of raw image ---------------------------------------------------------------------- Traceback (most recent call last):  File "tests/testButler.py", line 101, in testRaw  raw = self.butler.get("raw", self.dataId, ccd=ccd, immediate=True)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 244, in get  return callback()  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 242, in   innerCallback(), dataId)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 238, in   callback = lambda: self._read(pythonType, location)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py", line 426, in _read  location.getCppType(), storageList, additionalData)  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/persistenceLib.py", line 1430, in unsafeRetrieve  return _persistenceLib.Persistence_unsafeRetrieve(self, *args) FitsError:   File "src/fits.cc", line 1064, in lsst::afw::fits::Fits::Fits(const std::string &, const std::string &, int)  cfitsio error: could not open the named file (104) : Opening file '/Users/rowen/LSST/code/testdata/testdata_cfht/DATA/raw/08BL05/w2.+2+2/2008-11-01/i2/1038843o.fits.fz[1]' with mode 'r' {0} lsst::afw::fits::FitsError: 'cfitsio error: could not open the named file (104) : Opening file '/Users/rowen/LSST/code/testdata/testdata_cfht/DATA/raw/08BL05/w2.+2+2/2008-11-01/i2/1038843o.fits.fz[1]' with mode 'r''     ---------------------------------------------------------------------- Ran 6 tests in 3.544s   FAILED (errors=4) 

#### Activity

Hide
Michael Wood-Vasey [X] (Inactive) added a comment -

Having no optional tests (as a long term goal) makes sense.

When I think test data for an obs_* package the only thing that comes to mind is images from that telescope+camera. These images, in general, will be large for any large format imager.

If we don't want to large test packages as part of testing an obs_* package, that would seem to rule out images.

What kind of test data would that leave?

Show
Michael Wood-Vasey [X] (Inactive) added a comment - Having no optional tests (as a long term goal) makes sense. When I think test data for an obs_* package the only thing that comes to mind is images from that telescope+camera. These images, in general, will be large for any large format imager. If we don't want to large test packages as part of testing an obs_* package, that would seem to rule out images. What kind of test data would that leave?
Hide
Russell Owen added a comment - - edited

I don't think the data needed to test obs_cfht has to be so large. I understand that natively CFHT writes single FITS files that contain all images from one focal plane, but surely it is practical to strip out all but one or two images (or, to avoid indexing issues, replace the earlier HDUs with tiny images or empty HDUs) and just fetch what remains.

Ideally the data would becomes small enough that it could be included in the obs_cfht package itself, but MegaCam has large CCDs, so that may not be practical. If obs_cfht itself is allowed to use git-lfs for some files, then I think it would be acceptable to include a few images and that would be my recommendation.

Another alternative is to generate data on the fly, but then the generating code would have to be carefully verified.

Show
Russell Owen added a comment - - edited I don't think the data needed to test obs_cfht has to be so large. I understand that natively CFHT writes single FITS files that contain all images from one focal plane, but surely it is practical to strip out all but one or two images (or, to avoid indexing issues, replace the earlier HDUs with tiny images or empty HDUs) and just fetch what remains. Ideally the data would becomes small enough that it could be included in the obs_cfht package itself, but MegaCam has large CCDs, so that may not be practical. If obs_cfht itself is allowed to use git-lfs for some files, then I think it would be acceptable to include a few images and that would be my recommendation. Another alternative is to generate data on the fly, but then the generating code would have to be carefully verified.
Hide
Michael Wood-Vasey [X] (Inactive) added a comment -

Russell Owen I hear you about the longer-term goals.

Do you have any objections to the pull request to at least just compress the files?

https://github.com/lsst/testdata_cfht/pull/2

Tim Jenness says that J Matt Peterson [X] says under git-lfs, it is helpful to compress files and that only files actually used by checked out versions get downloaded.

Show
Michael Wood-Vasey [X] (Inactive) added a comment - Russell Owen I hear you about the longer-term goals. Do you have any objections to the pull request to at least just compress the files? https://github.com/lsst/testdata_cfht/pull/2 Tim Jenness says that J Matt Peterson [X] says under git-lfs, it is helpful to compress files and that only files actually used by checked out versions get downloaded.
Hide
Russell Owen added a comment -

I have no objection to compressing the files (I assume it did saved enough space to be interesting).

Show
Russell Owen added a comment - I have no objection to compressing the files (I assume it did saved enough space to be interesting).
Hide
Michael Wood-Vasey [X] (Inactive) added a comment -

I am taking this Issues as implemented. Thanks to all for the discussion.

The outstanding unresolved issues is a larger discussion on what should obs_ packages test. I propose that we spend 6 months (perhaps 9 months include the next short+standard cycle) with new ci_ capabilities to give us a better picture of what the distinction between ci_ and obs_ tests should be and what the relationship between validation_data_ and testdata_ packages should be.

If it becomes clear that we want to revisit the testdata_* packages at that time a new ticket asking to do so would be appropriate to submit.

Show
Michael Wood-Vasey [X] (Inactive) added a comment - I am taking this Issues as implemented. Thanks to all for the discussion. The outstanding unresolved issues is a larger discussion on what should obs_ packages test. I propose that we spend 6 months (perhaps 9 months include the next short+standard cycle) with new ci_ capabilities to give us a better picture of what the distinction between ci_ and obs_ tests should be and what the relationship between validation_data_ and testdata_ packages should be. If it becomes clear that we want to revisit the testdata_* packages at that time a new ticket asking to do so would be appropriate to submit.

#### People

Assignee:
Michael Wood-Vasey [X] (Inactive)
Reporter:
Russell Owen
Reviewers:
Russell Owen
Watchers:
David Nidever [X] (Inactive), Dominique Boutigny, Hsin-Fang Chiang, Joshua Hoblitt, Michael Wood-Vasey [X] (Inactive), Paul Price, Russell Owen, Simon Krughoff, Tim Jenness