Identify changes in HSC processCcd matching between w_2017_39 and w_2017_38

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
1
• Team:
Data Release Production
• Urgent?:
No

Description

Using the Python 2 shared stack on lsst-dev (either on /software or on /ssd), processCcd.py was able to process HSC data --id visit=9868 ccd=77 with w_2017_38 but failed with w_2017_39 at "Unable to match sources".

The command to reproduce can be:

 processCcd.py /datasets/hsc/repo/ --rerun private/usename/ --id visit=9868 ccd=77 

With w_2017_38:

 processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 57 matches with scatter = 0.040 +- 0.026 arcsec processCcd.calibrate.photoRefObjLoader INFO: Loading reference objects using center Fk5Coord(34.7345186, -2.9528981, 2000.00) and radius 0.121483483029 deg processCcd.calibrate.photoRefObjLoader INFO: Loaded 3186 reference objects processCcd.calibrate.photoCal.matcher INFO: filterStars purged 0 reference stars, leaving 3186 stars processCcd.calibrate.photoCal.matcher INFO: Purged 778 unusable sources, leaving 101 usable sources processCcd.calibrate.photoCal.matcher INFO: Matched 72 sources processCcd.calibrate.photoCal INFO: Found 72 matches with scatter = 0.045 +- 0.032 arcsec;  processCcd.calibrate.photoCal WARN: No 'photometric' flag key found in reference schema. processCcd.calibrate.photoCal INFO: Applying color terms for filterName=u'g', config.photoCatName=ps1_pv3_3pi_20170110 because config.applyColorTerms is True processCcd.calibrate.photoCal INFO: Magnitude zero point: 32.157198 +/- 0.000578 from 19 stars processCcd.calibrate INFO: Photometric zero-point: 32.157198 

With w_2017_39:

 processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 56 matches with scatter = 0.039 +- 0.026 arcsec processCcd.calibrate.photoRefObjLoader INFO: Loading reference objects using center Fk5Coord(34.7345155, -2.9528997, 2000.00) and radius 0.121496056094 deg processCcd.calibrate.photoRefObjLoader INFO: Loaded 3184 reference objects processCcd.calibrate.photoCal.matcher INFO: filterStars purged 0 reference stars, leaving 3184 stars processCcd.calibrate.photoCal.matcher INFO: Purged 778 unusable sources, leaving 99 usable sources processCcd FATAL: Failed on dataId={'taiObs': '2014-10-01', 'pointing': 1004, 'visit': 9868, 'dateObs': '2014-10-01', 'filter': 'HSC-G', 'field': 'SSP_WIDE', 'ccd': 77, 'expTime': 150.0}: Unable to match sources Traceback (most recent call last):  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/cmdLineTask.py", line 407, in __call__  result = task.run(dataRef, **kwargs)  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-66-gfbf2f2ce+2/python/lsst/pipe/tasks/processCcd.py", line 199, in run  icSourceCat=charRes.sourceCat,  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-66-gfbf2f2ce+2/python/lsst/pipe/tasks/calibrate.py", line 433, in run  icSourceCat=icSourceCat,  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-66-gfbf2f2ce+2/python/lsst/pipe/tasks/calibrate.py", line 538, in calibrate  photoRes = self.photoCal.run(exposure, sourceCat=sourceCat, expId=exposureIdInfo.expId)  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-66-gfbf2f2ce+2/python/lsst/pipe/tasks/photoCal.py", line 666, in run  res = self.loadAndMatch(exposure, sourceCat)  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/ssd/lsstsw/stack_20170409/Linux64/meas_astrom/13.0-16-g30f6da5+20/python/lsst/meas/astrom/ref_match.py", line 116, in loadAndMatch  match_tolerance=None,  File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/ssd/lsstsw/stack_20170409/Linux64/meas_astrom/13.0-16-g30f6da5+20/python/lsst/meas/astrom/matchOptimisticB/matchOptimisticBContinued.py", line 276, in matchObjectsToSources  raise RuntimeError("Unable to match sources") 

Activity

Hide
Hsin-Fang Chiang added a comment -

The difference seemed to come from DM-9579. Using w_2017_38 plus the DM-9579 branch, processing this ccd failed.

Show
Hsin-Fang Chiang added a comment - The difference seemed to come from DM-9579 . Using w_2017_38 plus the DM-9579 branch, processing this ccd failed.
Hide
Lauren MacArthur added a comment -

The difference seems to start with differing astrometric fits. The reference loading search center & radius for the two stack runs differ, leading to a different set & number of photoCal reference objects getting loaded. There are still a large number of “usable sources”, so the failure may still be somewhat surprising...

Show
Lauren MacArthur added a comment - The difference seems to start with differing astrometric fits. The reference loading search center & radius for the two stack runs differ, leading to a different set & number of photoCal reference objects getting loaded. There are still a large number of “usable sources”, so the failure may still be somewhat surprising...
Hide
Hsin-Fang Chiang added a comment -

Now I finished running the full HSC RC set in DM-11980, I found a few examples that failed in w_2017_38 but passed now, so maybe I should not call this a regression.

In the HSC RC set the data IDs of which the outcome changes include:
passed in w_2017_38 but failed now
visit=9868 ccd=77
visit=6528 ccd=59

failed in w_2017_38 but passed now
visit=28392 ccd=102
visit=28394 ccd=93

Show
Hsin-Fang Chiang added a comment - Now I finished running the full HSC RC set in DM-11980 , I found a few examples that failed in w_2017_38 but passed now, so maybe I should not call this a regression. In the HSC RC set the data IDs of which the outcome changes include: passed in w_2017_38 but failed now visit=9868 ccd=77 visit=6528 ccd=59 failed in w_2017_38 but passed now visit=28392 ccd=102 visit=28394 ccd=93
Hide
John Swinbank added a comment -

A little confused by the above — is it obvious why DM-9579 would lead to the reference loading search centre & radius changing?

Regardless of that, it seems reasonable to accept that the changes on DM-9579 could lead to a change in behaviour. Hopefully, the new behaviour is repeatable, where the old behaviour was not. Given that, I don't think there's a problem that we need waste time chasing down here. Lauren MacArthur, Jim Bosch, do you agree?

Show
John Swinbank added a comment - A little confused by the above — is it obvious why DM-9579 would lead to the reference loading search centre & radius changing? Regardless of that, it seems reasonable to accept that the changes on DM-9579 could lead to a change in behaviour. Hopefully, the new behaviour is repeatable, where the old behaviour was not. Given that, I don't think there's a problem that we need waste time chasing down here. Lauren MacArthur , Jim Bosch , do you agree?
Hide
Lauren MacArthur added a comment -

My guess is that the non-deterministic object reservation prior to DM-9579 could have led to different reference sources for the calibration – and thus slightly different calibration results – than post DM-9579. If this suspicion is indeed the cause of the differences, then I do indeed believe things should now be deterministic and repeatable. My only hesitation in accepting this theory is the fit pass/fail pre/post when there were still many usable sources (i.e. I would only really expect the pass/fail to occur for edge cases where there are few calibration candidates to begin with).

Show
Lauren MacArthur added a comment - My guess is that the non-deterministic object reservation prior to DM-9579 could have led to different reference sources for the calibration – and thus slightly different calibration results – than post DM-9579 . If this suspicion is indeed the cause of the differences, then I do indeed believe things should now be deterministic and repeatable. My only hesitation in accepting this theory is the fit pass/fail pre/post when there were still many usable sources (i.e. I would only really expect the pass/fail to occur for edge cases where there are few calibration candidates to begin with).
Hide
Jim Bosch added a comment -

Given Lauren MacArthur's explanation, I'm still a bit bothered by how sensitive PhotoCal seems to be to the number of input sources, but if it's a matching step in PhotoCal that's failing, that should be addressed by the implementation of Paul Price's RFC-405.

Show
Jim Bosch added a comment - Given Lauren MacArthur 's explanation, I'm still a bit bothered by how sensitive PhotoCal seems to be to the number of input sources, but if it's a matching step in PhotoCal that's failing, that should be addressed by the implementation of Paul Price 's RFC-405 .
Hide

This looks like it's been resolved. Can we mark as Done?

Show
Yusra AlSayyad added a comment - This looks like it's been resolved. Can we mark as Done?

People

Assignee:
Lauren MacArthur
Reporter:
Hsin-Fang Chiang
Watchers:
Hsin-Fang Chiang, Jim Bosch, Lauren MacArthur, Tim Jenness, Yusra AlSayyad