Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16289

validate_drp seems to fail on Python 3.7

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Team:
      SQuaRE

      Description

      I have tried out the fix from DM-15462 and whilst it does work on my numpy1.15 system, I get now a new failure on python 3.7 when running lsst_ci:

      tests/test_obs.py:50: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      ../../stack/DarwinX86/utils/16.0-13-g80874fd+2/python/lsst/utils/tests.py:297: in assertExecutable
          self.fail(msg)
      E   AssertionError: DECam Quick Test failed
      ----------------------------- Captured stdout call -----------------------------
      Running executable 'scripts/runDecamQuickTest.sh' with arguments "-- --noplot"...
      Ingesting Raw data
      root INFO: Loading config overrride file '/Users/timj/work/lsstsw3/stack/DarwinX86/obs_decam/16.0-11-g0300050/config/ingest.py'
      CameraMapper INFO: Loading Posix exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/instcal/instcal0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/instcal0176837.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/dqmask/dqmask0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/dqmask0176837.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/wtmap/wtmap0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/wtmap0176837.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/instcal/instcal0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/instcal0176846.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/dqmask/dqmask0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/dqmask0176846.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/wtmap/wtmap0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/wtmap0176846.fits.fz
      running singleFrameDriver
      validating
      CameraMapper INFO: Loading exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/registry.sqlite3
      No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 10}, tag=set())
      Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 10}
      Skipping this dataId.
      No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 10}, tag=set())
      Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 10}
      Skipping this dataId.
       
      ----------------------------- Captured stderr call -----------------------------
      Traceback (most recent call last):
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/bin/validateDrp.py", line 104, in <module>
          validate.run(args.repo, **kwargs)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/validate.py", line 152, in run
          metrics_package=metrics_package, **kwargs)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
          metrics_package=metrics_package, **kwargs)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/validate.py", line 301, in runOneFilter
          skipTEx=skipTEx)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/matchreduce.py", line 126, in build_matched_dataset
          useJointCal=useJointCal, skipTEx=skipTEx)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/matchreduce.py", line 293, in _loadAndMatchCatalogs
          matchCat = mmatch.finish()
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/afw/16.0-55-ged26810fc/python/lsst/afw/table/multiMatch.py", line 123, in finish
          for record in self.result:
      TypeError: 'NoneType' object is not iterable
       generated xml file: /Users/timj/work/lsstsw3/build/lsst_ci/tests/.tests/pytest-lsst_ci.xml 
      

      The DECam and CFHT tests fail in the same way.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Closing this as INVALID since everything now seems to be working with python3.7. Not sure what changed recently but all is now good.

            Show
            tjenness Tim Jenness added a comment - Closing this as INVALID since everything now seems to be working with python3.7. Not sure what changed recently but all is now good.
            Hide
            krughoff Simon Krughoff added a comment -

            The fix to DM-15462 also fixed this. Tim Jenness maybe we can close as DONE instead of INVALID?

            Show
            krughoff Simon Krughoff added a comment - The fix to DM-15462 also fixed this. Tim Jenness maybe we can close as DONE instead of INVALID?
            Hide
            tjenness Tim Jenness added a comment -

            I think INVALID is right since no work was done for this and it was probably my user error (leaving a broken validate_drp lying around).

            Show
            tjenness Tim Jenness added a comment - I think INVALID is right since no work was done for this and it was probably my user error (leaving a broken validate_drp lying around).
            Hide
            krughoff Simon Krughoff added a comment -

            OK. Mostly I just wanted to point out that this was not fixed by magic. It was the work on the other ticket that caused this to be fixed.

            Show
            krughoff Simon Krughoff added a comment - OK. Mostly I just wanted to point out that this was not fixed by magic. It was the work on the other ticket that caused this to be fixed.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Frossie Economou, Jeffrey Carlin, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.