Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16289

validate_drp seems to fail on Python 3.7

    XMLWordPrintable

Details

    • Story
    • Status: Invalid
    • Resolution: Done
    • None
    • None
    • None
    • SQuaRE

    Description

      I have tried out the fix from DM-15462 and whilst it does work on my numpy1.15 system, I get now a new failure on python 3.7 when running lsst_ci:

      tests/test_obs.py:50: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      ../../stack/DarwinX86/utils/16.0-13-g80874fd+2/python/lsst/utils/tests.py:297: in assertExecutable
          self.fail(msg)
      E   AssertionError: DECam Quick Test failed
      ----------------------------- Captured stdout call -----------------------------
      Running executable 'scripts/runDecamQuickTest.sh' with arguments "-- --noplot"...
      Ingesting Raw data
      root INFO: Loading config overrride file '/Users/timj/work/lsstsw3/stack/DarwinX86/obs_decam/16.0-11-g0300050/config/ingest.py'
      CameraMapper INFO: Loading Posix exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/instcal/instcal0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/instcal0176837.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/dqmask/dqmask0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/dqmask0176837.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/wtmap/wtmap0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/wtmap0176837.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/instcal/instcal0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/instcal0176846.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/dqmask/dqmask0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/dqmask0176846.fits.fz
      ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0-1-gd5cb667+2/wtmap/wtmap0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/wtmap0176846.fits.fz
      running singleFrameDriver
      validating
      CameraMapper INFO: Loading exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/registry.sqlite3
      No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 10}, tag=set())
      Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 10}
      Skipping this dataId.
      No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 10}, tag=set())
      Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 10}
      Skipping this dataId.
       
      ----------------------------- Captured stderr call -----------------------------
      Traceback (most recent call last):
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/bin/validateDrp.py", line 104, in <module>
          validate.run(args.repo, **kwargs)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/validate.py", line 152, in run
          metrics_package=metrics_package, **kwargs)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
          metrics_package=metrics_package, **kwargs)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/validate.py", line 301, in runOneFilter
          skipTEx=skipTEx)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/matchreduce.py", line 126, in build_matched_dataset
          useJointCal=useJointCal, skipTEx=skipTEx)
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-12-g7d69e54+1/python/lsst/validate/drp/matchreduce.py", line 293, in _loadAndMatchCatalogs
          matchCat = mmatch.finish()
        File "/Users/timj/work/lsstsw3/stack/DarwinX86/afw/16.0-55-ged26810fc/python/lsst/afw/table/multiMatch.py", line 123, in finish
          for record in self.result:
      TypeError: 'NoneType' object is not iterable
       generated xml file: /Users/timj/work/lsstsw3/build/lsst_ci/tests/.tests/pytest-lsst_ci.xml 
      

      The DECam and CFHT tests fail in the same way.

      Attachments

        Issue Links

          Activity

            tjenness Tim Jenness added a comment -

            Closing this as INVALID since everything now seems to be working with python3.7. Not sure what changed recently but all is now good.

            tjenness Tim Jenness added a comment - Closing this as INVALID since everything now seems to be working with python3.7. Not sure what changed recently but all is now good.

            The fix to DM-15462 also fixed this. tjenness maybe we can close as DONE instead of INVALID?

            krughoff Simon Krughoff (Inactive) added a comment - The fix to DM-15462 also fixed this. tjenness maybe we can close as DONE instead of INVALID?
            tjenness Tim Jenness added a comment -

            I think INVALID is right since no work was done for this and it was probably my user error (leaving a broken validate_drp lying around).

            tjenness Tim Jenness added a comment - I think INVALID is right since no work was done for this and it was probably my user error (leaving a broken validate_drp lying around).

            OK. Mostly I just wanted to point out that this was not fixed by magic. It was the work on the other ticket that caused this to be fixed.

            krughoff Simon Krughoff (Inactive) added a comment - OK. Mostly I just wanted to point out that this was not fixed by magic. It was the work on the other ticket that caused this to be fixed.

            People

              Unassigned Unassigned
              tjenness Tim Jenness
              Frossie Economou, Jeffrey Carlin, Simon Krughoff (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.