Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12179

Identify changes in HSC processCcd matching between w_2017_39 and w_2017_38

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Using the Python 2 shared stack on lsst-dev (either on /software or on /ssd), processCcd.py was able to process HSC data --id visit=9868 ccd=77 with w_2017_38 but failed with w_2017_39 at "Unable to match sources".

      The command to reproduce can be:

      processCcd.py /datasets/hsc/repo/ --rerun private/usename/ --id visit=9868 ccd=77
      

      With w_2017_38:

      processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 57 matches with scatter = 0.040 +- 0.026 arcsec
      processCcd.calibrate.photoRefObjLoader INFO: Loading reference objects using center Fk5Coord(34.7345186, -2.9528981, 2000.00) and radius 0.121483483029 deg
      processCcd.calibrate.photoRefObjLoader INFO: Loaded 3186 reference objects
      processCcd.calibrate.photoCal.matcher INFO: filterStars purged 0 reference stars, leaving 3186 stars
      processCcd.calibrate.photoCal.matcher INFO: Purged 778 unusable sources, leaving 101 usable sources
      processCcd.calibrate.photoCal.matcher INFO: Matched 72 sources
      processCcd.calibrate.photoCal INFO: Found 72 matches with scatter = 0.045 +- 0.032 arcsec; 
      processCcd.calibrate.photoCal WARN: No 'photometric' flag key found in reference schema.
      processCcd.calibrate.photoCal INFO: Applying color terms for filterName=u'g', config.photoCatName=ps1_pv3_3pi_20170110 because config.applyColorTerms is True
      processCcd.calibrate.photoCal INFO: Magnitude zero point: 32.157198 +/- 0.000578 from 19 stars
      processCcd.calibrate INFO: Photometric zero-point: 32.157198
      

      With w_2017_39:

      processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 56 matches with scatter = 0.039 +- 0.026 arcsec
      processCcd.calibrate.photoRefObjLoader INFO: Loading reference objects using center Fk5Coord(34.7345155, -2.9528997, 2000.00) and radius 0.121496056094 deg
      processCcd.calibrate.photoRefObjLoader INFO: Loaded 3184 reference objects
      processCcd.calibrate.photoCal.matcher INFO: filterStars purged 0 reference stars, leaving 3184 stars
      processCcd.calibrate.photoCal.matcher INFO: Purged 778 unusable sources, leaving 99 usable sources
      processCcd FATAL: Failed on dataId={'taiObs': '2014-10-01', 'pointing': 1004, 'visit': 9868, 'dateObs': '2014-10-01', 'filter': 'HSC-G', 'field': 'SSP_WIDE', 'ccd': 77, 'expTime': 150.0}: Unable to match sources
      Traceback (most recent call last):
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/cmdLineTask.py", line 407, in __call__
          result = task.run(dataRef, **kwargs)
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-66-gfbf2f2ce+2/python/lsst/pipe/tasks/processCcd.py", line 199, in run
          icSourceCat=charRes.sourceCat,
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-66-gfbf2f2ce+2/python/lsst/pipe/tasks/calibrate.py", line 433, in run
          icSourceCat=icSourceCat,
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-66-gfbf2f2ce+2/python/lsst/pipe/tasks/calibrate.py", line 538, in calibrate
          photoRes = self.photoCal.run(exposure, sourceCat=sourceCat, expId=exposureIdInfo.expId)
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-66-gfbf2f2ce+2/python/lsst/pipe/tasks/photoCal.py", line 666, in run
          res = self.loadAndMatch(exposure, sourceCat)
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack_20170409/Linux64/meas_astrom/13.0-16-g30f6da5+20/python/lsst/meas/astrom/ref_match.py", line 116, in loadAndMatch
          match_tolerance=None,
        File "/ssd/lsstsw/stack_20170409/Linux64/pipe_base/13.0-14-g8b3bf66+27/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack_20170409/Linux64/meas_astrom/13.0-16-g30f6da5+20/python/lsst/meas/astrom/matchOptimisticB/matchOptimisticBContinued.py", line 276, in matchObjectsToSources
          raise RuntimeError("Unable to match sources")
      

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The difference seemed to come from DM-9579. Using w_2017_38 plus the DM-9579 branch, processing this ccd failed.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The difference seemed to come from DM-9579 . Using w_2017_38 plus the DM-9579 branch, processing this ccd failed.
            Hide
            lauren Lauren MacArthur added a comment -

            The difference seems to start with differing astrometric fits. The reference loading search center & radius for the two stack runs differ, leading to a different set & number of photoCal reference objects getting loaded. There are still a large number of “usable sources”, so the failure may still be somewhat surprising...

            Show
            lauren Lauren MacArthur added a comment - The difference seems to start with differing astrometric fits. The reference loading search center & radius for the two stack runs differ, leading to a different set & number of photoCal reference objects getting loaded. There are still a large number of “usable sources”, so the failure may still be somewhat surprising...
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Now I finished running the full HSC RC set in DM-11980, I found a few examples that failed in w_2017_38 but passed now, so maybe I should not call this a regression.

            In the HSC RC set the data IDs of which the outcome changes include:
            passed in w_2017_38 but failed now
            visit=9868 ccd=77
            visit=6528 ccd=59

            failed in w_2017_38 but passed now
            visit=28392 ccd=102
            visit=28394 ccd=93

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Now I finished running the full HSC RC set in DM-11980 , I found a few examples that failed in w_2017_38 but passed now, so maybe I should not call this a regression. In the HSC RC set the data IDs of which the outcome changes include: passed in w_2017_38 but failed now visit=9868 ccd=77 visit=6528 ccd=59 failed in w_2017_38 but passed now visit=28392 ccd=102 visit=28394 ccd=93
            Hide
            swinbank John Swinbank added a comment -

            A little confused by the above — is it obvious why DM-9579 would lead to the reference loading search centre & radius changing?

            Regardless of that, it seems reasonable to accept that the changes on DM-9579 could lead to a change in behaviour. Hopefully, the new behaviour is repeatable, where the old behaviour was not. Given that, I don't think there's a problem that we need waste time chasing down here. Lauren MacArthur, Jim Bosch, do you agree?

            Show
            swinbank John Swinbank added a comment - A little confused by the above — is it obvious why DM-9579 would lead to the reference loading search centre & radius changing? Regardless of that, it seems reasonable to accept that the changes on DM-9579 could lead to a change in behaviour. Hopefully, the new behaviour is repeatable, where the old behaviour was not. Given that, I don't think there's a problem that we need waste time chasing down here. Lauren MacArthur , Jim Bosch , do you agree?
            Hide
            lauren Lauren MacArthur added a comment -

            My guess is that the non-deterministic object reservation prior to DM-9579 could have led to different reference sources for the calibration – and thus slightly different calibration results – than post DM-9579. If this suspicion is indeed the cause of the differences, then I do indeed believe things should now be deterministic and repeatable. My only hesitation in accepting this theory is the fit pass/fail pre/post when there were still many usable sources (i.e. I would only really expect the pass/fail to occur for edge cases where there are few calibration candidates to begin with).

            Show
            lauren Lauren MacArthur added a comment - My guess is that the non-deterministic object reservation prior to DM-9579 could have led to different reference sources for the calibration – and thus slightly different calibration results – than post DM-9579 . If this suspicion is indeed the cause of the differences, then I do indeed believe things should now be deterministic and repeatable. My only hesitation in accepting this theory is the fit pass/fail pre/post when there were still many usable sources (i.e. I would only really expect the pass/fail to occur for edge cases where there are few calibration candidates to begin with).
            Hide
            jbosch Jim Bosch added a comment -

            Given Lauren MacArthur's explanation, I'm still a bit bothered by how sensitive PhotoCal seems to be to the number of input sources, but if it's a matching step in PhotoCal that's failing, that should be addressed by the implementation of Paul Price's RFC-405.

            Show
            jbosch Jim Bosch added a comment - Given Lauren MacArthur 's explanation, I'm still a bit bothered by how sensitive PhotoCal seems to be to the number of input sources, but if it's a matching step in PhotoCal that's failing, that should be addressed by the implementation of Paul Price 's RFC-405 .
            Hide
            yusra Yusra AlSayyad added a comment -

            This looks like it's been resolved. Can we mark as Done?

            Show
            yusra Yusra AlSayyad added a comment - This looks like it's been resolved. Can we mark as Done?

              People

              • Assignee:
                Unassigned
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Jim Bosch, John Swinbank, Lauren MacArthur, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel