Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12180

Different processCcd matching outcomes using Py2 and Py3 stacks

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None

      Description

      Running ProcessCcd with HSC data --id visit=9868 ccd=77 using Stack of the same tag w_2017_38 of the Python 2 shared stack at /ssd/lsstsw/stack and the Python 3 shared stack at /ssd/lsstsw/stack3/ gives different matching results. The Py2 stack passed but the Py3 stack failed with "Unable to match sources".

      The command to reproduce:

      processCcd.py /datasets/hsc/repo/ --rerun private/user/path --id visit=9868 ccd=77
      

      With Python 2 on /ssd/lsstsw/stack_20170409/:

      processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 57 matches with scatter = 0.040 +- 0.026 arcsec
      processCcd.calibrate.photoRefObjLoader INFO: Loading reference objects using center Fk5Coord(34.7345186, -2.9528981, 2000.00) and radius 0.121483483029 deg
      processCcd.calibrate.photoRefObjLoader INFO: Loaded 3186 reference objects
      processCcd.calibrate.photoCal.matcher INFO: filterStars purged 0 reference stars, leaving 3186 stars
      processCcd.calibrate.photoCal.matcher INFO: Purged 778 unusable sources, leaving 101 usable sources
      processCcd.calibrate.photoCal.matcher INFO: Matched 72 sources
      processCcd.calibrate.photoCal INFO: Found 72 matches with scatter = 0.045 +- 0.032 arcsec; 
      processCcd.calibrate.photoCal WARN: No 'photometric' flag key found in reference schema.
      processCcd.calibrate.photoCal INFO: Applying color terms for filterName=u'g', config.photoCatName=ps1_pv3_3pi_20170110 because config.applyColorTerms is True
      processCcd.calibrate.photoCal INFO: Magnitude zero point: 32.157198 +/- 0.000578 from 19 stars
      processCcd.calibrate INFO: Photometric zero-point: 32.157198
      

      With Python 3 on /ssd/lsstsw/stack3_2017-09-11/

      processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 60 matches with scatter = 0.044 +- 0.029 arcsec
      processCcd.calibrate.photoRefObjLoader INFO: Loading reference objects using center Fk5Coord(34.7345178, -2.9529006, 2000.00) and radius 0.12148793072780815 deg
      processCcd.calibrate.photoRefObjLoader INFO: Loaded 3187 reference objects
      processCcd.calibrate.photoCal.matcher INFO: filterStars purged 0 reference stars, leaving 3187 stars
      processCcd.calibrate.photoCal.matcher INFO: Purged 782 unusable sources, leaving 97 usable sources
      processCcd FATAL: Failed on dataId={'field': 'SSP_WIDE', 'ccd': 77, 'taiObs': '2014-10-01', 'dateObs': '2014-10-01', 'visit': 9868, 'expTime': 150.0, 'filter': 'HSC-G', 'pointing': 1004}: Unable to match sources
      Traceback (most recent call last):
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-14-g8b3bf66+22/python/lsst/pipe/base/cmdLineTask.py", line 407, in __call__
          result = task.run(dataRef, **kwargs)
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-14-g8b3bf66+22/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-64-gd461ca2f+2/python/lsst/pipe/tasks/processCcd.py", line 199, in run
          icSourceCat=charRes.sourceCat,
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-14-g8b3bf66+22/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-64-gd461ca2f+2/python/lsst/pipe/tasks/calibrate.py", line 433, in run
          icSourceCat=icSourceCat,
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-64-gd461ca2f+2/python/lsst/pipe/tasks/calibrate.py", line 538, in calibrate
          photoRes = self.photoCal.run(exposure, sourceCat=sourceCat)
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-14-g8b3bf66+22/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-64-gd461ca2f+2/python/lsst/pipe/tasks/photoCal.py", line 667, in run
          res = self.loadAndMatch(exposure, sourceCat)
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-14-g8b3bf66+22/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_astrom/13.0-16-g30f6da5+15/python/lsst/meas/astrom/ref_match.py", line 116, in loadAndMatch
          match_tolerance=None,
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-14-g8b3bf66+22/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/ssd/lsstsw/stack3_2017-09-11/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_astrom/13.0-16-g30f6da5+15/python/lsst/meas/astrom/matchOptimisticB/matchOptimisticBContinued.py", line 276, in matchObjectsToSources
          raise RuntimeError("Unable to match sources")
      RuntimeError: Unable to match sources
      

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Paul Price pointed out that matchOptimisticB has from __future__ import division and might cause py2/3 differences.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Paul Price pointed out that matchOptimisticB has from __future__ import division and might cause py2/3 differences.
            Hide
            tjenness Tim Jenness added a comment -

            Importing division should make the code behave the same on both versions. It should not cause differences. The usual difference is caused by division not being imported such that py2 and py3 use different divisions.

            Show
            tjenness Tim Jenness added a comment - Importing division should make the code behave the same on both versions. It should not cause differences. The usual difference is caused by division not being imported such that py2 and py3 use different divisions.
            Hide
            yusra Yusra AlSayyad added a comment - - edited

            Is it possible the cause of this produced the symptom that appeared in DM-11779Sophie Reed, Dan Taranu?

            On DM-11779 We confirmed that for w_2018_14 the  py2 and py3 gave identical results when switching a line to np.random. I suspect that it covers this too. 

            Show
            yusra Yusra AlSayyad added a comment - - edited Is it possible the cause of this produced the symptom that appeared in  DM-11779 ?  Sophie Reed , Dan Taranu ? On DM-11779 We confirmed that for w_2018_14 the  py2 and py3 gave identical results when switching a line to np.random. I suspect that it covers this too. 

              People

              • Assignee:
                Unassigned
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel