# ci_hsc fails test requiring >95% of PSF stars to be stars on the coadd

XMLWordPrintable

## Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Sprint:
DRP W16-7
• Team:
Data Release Production

## Description

Since the first week of March 2016, ci_hsc fails its test that requires that >95% of the PSF stars be identified as stars in the coadd. I suspect this is related to the DM-4692 merge.

Here is a sample job that fails:
https://ci.lsst.codes/job/stack-os-matrix/9084/label=centos-6/console

The relevant snippet of the failure is:

 [2016-03-10T17:12:06.667778Z] : Validating dataset measureCoaddSources_config for {'filter': 'HSC-R', 'tract': 0, 'patch': '5,4'} [2016-03-10T17:12:06.697383Z] CameraMapper: Loading registry registry from /home/build0/lsstsw/build/ci_hsc/DATA/registry.sqlite3 [2016-03-10T17:12:06.697615Z] CameraMapper: Loading calibRegistry registry from /home/build0/lsstsw/build/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 [2016-03-10T17:12:07.716310Z] CameraMapper: Loading registry registry from /home/build0/lsstsw/build/ci_hsc/DATA/registry.sqlite3 [2016-03-10T17:12:07.716443Z] CameraMapper: Loading calibRegistry registry from /home/build0/lsstsw/build/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 [2016-03-10T17:12:08.663566Z] : measureCoaddSources_config exists: PASS [2016-03-10T17:12:08.721051Z] : measureCoaddSources_config readable (): PASS [2016-03-10T17:12:08.721077Z] : Validating dataset measureCoaddSources_metadata for {'filter': 'HSC-R', 'tract': 0, 'patch': '5,4'} [2016-03-10T17:12:08.721249Z] : measureCoaddSources_metadata exists: PASS [2016-03-10T17:12:08.721663Z] : measureCoaddSources_metadata readable (): PASS [2016-03-10T17:12:08.721715Z] : Validating dataset deepCoadd_meas_schema for {'filter': 'HSC-R', 'tract': 0, 'patch': '5,4'} [2016-03-10T17:12:08.721878Z] : deepCoadd_meas_schema exists: PASS [2016-03-10T17:12:08.726703Z] : deepCoadd_meas_schema readable (): PASS [2016-03-10T17:12:08.726834Z] : Validating source output for {'filter': 'HSC-R', 'tract': 0, 'patch': '5,4'} [2016-03-10T17:12:10.203469Z] : Number of sources (7595 > 100): PASS [2016-03-10T17:12:10.204166Z] : calib_psfCandidate field exists in deepCoadd_meas catalog: PASS [2016-03-10T17:12:10.204772Z] : calib_psfUsed field exists in deepCoadd_meas catalog: PASS [2016-03-10T17:12:10.205468Z] : Aperture correction fields for base_PsfFlux are present.: PASS [2016-03-10T17:12:10.206159Z] : Aperture correction fields for base_GaussianFlux are present.: PASS [2016-03-10T17:12:10.207193Z] FATAL: 95% of sources used to build the PSF are classified as stars on the coadd (0 > 0): FAIL [2016-03-10T17:12:10.207455Z] scons: *** [.scons/measure-HSC-R] AssertionError : Failed test: 95% of sources used to build the PSF are classified as stars on the coadd (0 > 0) [2016-03-10T17:12:10.207481Z] Traceback (most recent call last): [2016-03-10T17:12:10.207525Z] File "/home/build0/lsstsw/stack/Linux64/scons/2.3.5/lib/scons/SCons/Action.py", line 1063, in execute [2016-03-10T17:12:10.207556Z] result = self.execfunction(target=target, source=rsources, env=env) [2016-03-10T17:12:10.207593Z] File "/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py", line 133, in scons [2016-03-10T17:12:10.207611Z] return self.run(*args, **kwargs) [2016-03-10T17:12:10.207646Z] File "/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py", line 122, in run [2016-03-10T17:12:10.207663Z] self.validateSources(dataId) [2016-03-10T17:12:10.207732Z] File "/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py", line 191, in validateSources [2016-03-10T17:12:10.207749Z] 0.95*psfStars.sum() [2016-03-10T17:12:10.207786Z] File "/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py", line 52, in assertGreater [2016-03-10T17:12:10.207816Z] self.assertTrue(description + " (%d > %d)" % (num1, num2), num1 > num2) [2016-03-10T17:12:10.207853Z] File "/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py", line 43, in assertTrue [2016-03-10T17:12:10.207877Z] raise AssertionError("Failed test: %s" % description) [2016-03-10T17:12:10.207919Z] AssertionError: Failed test: 95% of sources used to build the PSF are classified as stars on the coadd (0 > 0) [2016-03-10T17:12:10.209935Z] scons: building terminated because of errors. 

This is the test that fails

https://github.com/lsst/ci_hsc/blob/74303a818eb5049a2015b5e885df2781053748c9/python/lsst/ci/hsc/validate.py#L169

 class MeasureValidation(Validation):  _datasets = ["measureCoaddSources_config", "measureCoaddSources_metadata", "deepCoadd_meas_schema"]  _sourceDataset = "deepCoadd_meas"  _matchDataset = "deepCoadd_srcMatch"    def validateSources(self, dataId):  catalog = Validation.validateSources(self, dataId)  self.assertTrue("calib_psfCandidate field exists in deepCoadd_meas catalog",  "calib_psfCandidate" in catalog.schema)  self.assertTrue("calib_psfUsed field exists in deepCoadd_meas catalog",  "calib_psfUsed" in catalog.schema)  self.checkApertureCorrections(catalog)  # Check that at least 95% of the stars we used to model the PSF end up classified as stars  # on the coadd. We certainly need much more purity than that to build good PSF models, but  # this should verify that flag propagation, aperture correction, and extendendess are all  # running and configured reasonably (but it may not be sensitive enough to detect subtle  # bugs).  psfStars = catalog.get("calib_psfUsed")  extStars = catalog.get("base_ClassificationExtendedness_value") < 0.5  self.assertGreater(  "95% of sources used to build the PSF are classified as stars on the coadd",  numpy.logical_and(extStars, psfStars).sum(),  0.95*psfStars.sum()  ) 

Note that the assertion failure messages is a bit confusing. It should say
"Fewer than 95% of the sources used to build the PSF are classified as stars on the coadd."

## Activity

Hide
Jim Bosch added a comment -

Assuming the regression is due to single-frame processing, it could be a problem in setting the psfUsed flag, or a problem in aperture corrections: if PsfFlux and GaussianFlux aren't tied to the same system for stars by aperture corrections, classification on the coadd will fail.

Show
Jim Bosch added a comment - Assuming the regression is due to single-frame processing, it could be a problem in setting the psfUsed flag, or a problem in aperture corrections: if PsfFlux and GaussianFlux aren't tied to the same system for stars by aperture corrections, classification on the coadd will fail.
Hide
John Swinbank added a comment -

Paul Price, do you have any ideas here? I'm (optimistically!) hoping given your experience writing ci_hsc and reviewing DM-4692 you might know an easy fix.

Show
John Swinbank added a comment - Paul Price , do you have any ideas here? I'm (optimistically!) hoping given your experience writing ci_hsc and reviewing DM-4692 you might know an easy fix.
Hide
Jim Bosch added a comment -

So, half of the problem is quite straightforward: PropagateVisitFlags is loading icSrc and expecting it to have its coord fields filled in. That's easy enough to fix, since it also has access to the corresponding Wcs. I'll commit a fix for that shortly on a tickets/DM-5419 branch of pipe_tasks.

But since we're also not seeing any objects classified as stars on the coadd (the comparison that failed was actually "0 < 0", and neither of those numbers should be zero), there's another problem.

Show
Jim Bosch added a comment - So, half of the problem is quite straightforward: PropagateVisitFlags is loading icSrc and expecting it to have its coord fields filled in. That's easy enough to fix, since it also has access to the corresponding Wcs . I'll commit a fix for that shortly on a tickets/ DM-5419 branch of pipe_tasks. But since we're also not seeing any objects classified as stars on the coadd (the comparison that failed was actually "0 < 0", and neither of those numbers should be zero), there's another problem.
Hide
Jim Bosch added a comment -

Fix is in on pipe_tasks tickets/DM-5419.

There was only the one problem after all; I just guessed incorrectly the second zero in the comparison message (both numbers are now nonzero, and the comparison is successful).

Show
Jim Bosch added a comment - Fix is in on pipe_tasks tickets/ DM-5419 . There was only the one problem after all; I just guessed incorrectly the second zero in the comparison message (both numbers are now nonzero, and the comparison is successful).
Hide
Russell Owen added a comment - - edited

This looks great. My one request, if you are willing to make the change, is to change the assertion failure message, as suggested by Michael Wood-Vasey at the end of this ticket's description. Thank you for the quick fix.

Show
Russell Owen added a comment - - edited This looks great. My one request, if you are willing to make the change, is to change the assertion failure message, as suggested by Michael Wood-Vasey at the end of this ticket's description. Thank you for the quick fix.
Hide
Jim Bosch added a comment -

My one request, if you are willing to make the change, is to change the assertion failure message, as suggested by Michael Wood-Vasey at the end of this ticket's description.

Done, and merged to master.

Show
Jim Bosch added a comment - My one request, if you are willing to make the change, is to change the assertion failure message, as suggested by Michael Wood-Vasey at the end of this ticket's description. Done, and merged to master.
Hide
Jim Bosch added a comment -

DM-5424 should make the fix already committed here unnecessary.

Show
Jim Bosch added a comment - DM-5424 should make the fix already committed here unnecessary.

## People

• Assignee:
Jim Bosch
Reporter:
Michael Wood-Vasey
Reviewers:
Russell Owen
Watchers:
Hsin-Fang Chiang, Jim Bosch, John Swinbank, Michael Wood-Vasey, Russell Owen