Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11779

Different processCcd outcomes using Py2 and Py3 stacks

    Details

    • Story Points:
      8
    • Sprint:
      DRP F18-1
    • Team:
      Data Release Production

      Description

      Processing using Python 2 and Python 3 stacks of the same weekly tag w_2017_33 gave different results for HSC data --id visit=6528 ccd=17 (available in /datasets/hsc/repo)

      The Py3 stack fails with the error below, while the Py2 stack can finish processCcd.

      processCcd.charImage.measurePsf INFO: PSF determination using 11/12 stars.
      processCcd.charImage INFO: iter 2; PSF sigma=1.29, dimensions=(41, 41); median background=7562.40
      processCcd.charImage.repair INFO: Identified 812 cosmic rays.
      processCcd.charImage.measurement INFO: Measuring 38 sources (38 parents, 0 children)
      processCcd.charImage.measureApCorr INFO: Measuring aperture corrections for 23 flux fields
      processCcd FATAL: Failed on dataId={'pointing': 991, 'expTime': 200.0, 'visit': 6528, 'ccd': 17, 'field': 'SSP_WIDE', 'filter': 'HSC-Y', 'taiObs': '2014-09-18', 'dateObs': '2014-09-18'}: Unable to measure aperture correction for required algorithm 'modelfit_CModel_initial': only 1 sources, but require at least 2.
      

      Same part of logs in Py2:

      processCcd.charImage.measurePsf INFO: PSF determination using 11/12 stars.
      processCcd.charImage INFO: iter 2; PSF sigma=1.28, dimensions=(41, 41); median background=7562.41
      processCcd.charImage.repair INFO: Identified 813 cosmic rays.
      processCcd.charImage.measurement INFO: Measuring 38 sources (38 parents, 0 children)
      processCcd.charImage.measureApCorr INFO: Measuring aperture corrections for 23 flux fields
      processCcd.charImage.measureApCorr INFO: Aperture correction for ext_convolved_ConvolvedFlux_1_4_5: RMS 0.126533 from 11
      /software/lsstsw/stack/Linux64/meas_algorithms/13.0-18-gc4ad422+4/python/lsst/meas/algorithms/measureApCorr.py:237: RuntimeWarning: invalid value encountered in greater
        fluxes > 0.0,
      processCcd.charImage.measureApCorr INFO: Aperture correction for modelfit_CModel_exp: RMS 0.046453 from 3
      processCcd.charImage.measureApCorr INFO: Aperture correction for ext_convolved_ConvolvedFlux_1_6_0: RMS 0.097525 from 11
      

        Attachments

          Issue Links

            Activity

            Hide
            sophiereed Sophie Reed added a comment - - edited

            Output of python 2:

            (Pdb) random.seed(0)
            (Pdb) random.sample(range(35), 7)
            [29, 25, 13, 8, 15, 12, 22]

            Output of python 3:

            (Pdb) random.seed(0)
            (Pdb) random.sample(range(35), 7)
            [24, 26, 2, 16, 30, 31, 15]

            These are different between the two python version leading to different stars being used for the reserve list. In the current stack this is done differently.

            Output of python 2:

            (Pdb) np.random.seed(0)
            (Pdb) x = np.arange(7)
            (Pdb) rng = np.random.RandomState(0)
            (Pdb) rng.shuffle(x)
            (Pdb) print(x)
            [6 2 1 3 0 5 4]
            

            Output of python 3:

            (Pdb) x = np.arange(7)
            (Pdb) rng = np.random.RandomState(0)
            (Pdb) rng.shuffle(x)
            (Pdb) print(x)
            [6 2 1 3 0 5 4]
            

            Show
            sophiereed Sophie Reed added a comment - - edited Output of python 2: (Pdb) random.seed(0) (Pdb) random.sample(range(35), 7) [29, 25, 13, 8, 15, 12, 22] Output of python 3: (Pdb) random.seed(0) (Pdb) random.sample(range(35), 7) [24, 26, 2, 16, 30, 31, 15] These are different between the two python version leading to different stars being used for the reserve list. In the current stack this is done differently. Output of python 2: (Pdb) np.random.seed(0) (Pdb) x = np.arange(7) (Pdb) rng = np.random.RandomState(0) (Pdb) rng.shuffle(x) (Pdb) print(x) [6 2 1 3 0 5 4] Output of python 3: (Pdb) x = np.arange(7) (Pdb) rng = np.random.RandomState(0) (Pdb) rng.shuffle(x) (Pdb) print(x) [6 2 1 3 0 5 4]
            Hide
            yusra Yusra AlSayyad added a comment -

            Since you have a repo with the right version to test right now, will you fix that line and run it in py2 and py3 and store the outputs and logs in both for comparison so we can say definitively that this was the only difference?

            Show
            yusra Yusra AlSayyad added a comment - Since you have a repo with the right version to test right now, will you fix that line and run it in py2 and py3 and store the outputs and logs in both for comparison so we can say definitively that this was the only difference?
            Hide
            sophiereed Sophie Reed added a comment - - edited

            Forcing measurePsf.py to use np.random instead for both produces identical results. These can be found in /datasets/hsc/repo/rerun/private/sr525/DM-11779-3/ and /datasets/hsc/repo/rerun/private/sr525/DM-11779-2/ which each also contain a log file py3.log and py2.log respectively.

            Show
            sophiereed Sophie Reed added a comment - - edited Forcing measurePsf.py to use np.random instead for both produces identical results. These can be found in /datasets/hsc/repo/rerun/private/sr525/ DM-11779 -3/ and /datasets/hsc/repo/rerun/private/sr525/ DM-11779 -2/ which each also contain a log file py3.log and py2.log respectively.
            Hide
            yusra Yusra AlSayyad added a comment -

            TL;DR  An older version of the stack had an instance of random.sample being used instead of numpy.random.  It was demonstrated that py2 and py3 processCcd's give identical results, when that line is changed.  That line no longer exists in the current, so there is no code to merge. Resolution of this ticket is that the mystery was solved, and we believe no existing problems still lurk. 

            Show
            yusra Yusra AlSayyad added a comment - TL;DR  An older version of the stack had an instance of random.sample being used instead of numpy.random .  It was demonstrated that py2 and py3 processCcd's give identical results, when that line is changed.  That line no longer exists in the current, so there is no code to merge. Resolution of this ticket is that the mystery was solved, and we believe no existing problems still lurk. 
            Hide
            yusra Yusra AlSayyad added a comment -

            This was a joint effort between Jim Bosch Dan Taranu and Sophie Reed during a pair programming session.

            Show
            yusra Yusra AlSayyad added a comment - This was a joint effort between Jim Bosch Dan Taranu and Sophie Reed during a pair programming session.

              People

              • Assignee:
                dtaranu Dan Taranu
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Yusra AlSayyad
                Watchers:
                Colin Slater, Hsin-Fang Chiang, Jim Bosch, John Swinbank, Sophie Reed, Tim Jenness, Yusra AlSayyad
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel