Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15462

Failure in lsst_ci with numpy1.15

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • lsst_ci, validate_drp
    • None
    • External

    Description

      Testing bin/deploy -b with DM-15385, DM-15419, DM-15394, DM-15389, and DM-15443 (Python 3.7.0, numpy 1.15.0, and astropy 3.0.4) I get the following error from lsst_ci:

      [2018-08-18T00:45:21.894199Z] Traceback (most recent call last):
      [2018-08-18T00:45:21.894202Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/bin/validateDrp.py", line 104, in <module>
      [2018-08-18T00:45:21.894204Z]     validate.run(args.repo, **kwargs)
      [2018-08-18T00:45:21.894206Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 152, in run
      [2018-08-18T00:45:21.894208Z]     metrics_package=metrics_package, **kwargs)
      [2018-08-18T00:45:21.894210Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
      [2018-08-18T00:45:21.894212Z]     metrics_package=metrics_package, **kwargs)
      [2018-08-18T00:45:21.894214Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 350, in runOneFilter
      [2018-08-18T00:45:21.894216Z]     pa2 = measurePA2(metrics[pa2_spec.metric_name], pa1, pf1_spec.threshold)
      [2018-08-18T00:45:21.894218Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/calcsrd/pa2.py", line 62, in measurePA2
      [2018-08-18T00:45:21.894220Z]     return Measurement(metric, np.percentile(np.abs(magDiffs), pf1Percentile) * magDiffs.unit,
      [2018-08-18T00:45:21.894222Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3540, in percentile
      [2018-08-18T00:45:21.894224Z]     a, q, axis, out, overwrite_input, interpolation, keepdims)
      [2018-08-18T00:45:21.894226Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked
      [2018-08-18T00:45:21.894229Z]     interpolation=interpolation)
      [2018-08-18T00:45:21.894231Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce
      [2018-08-18T00:45:21.894233Z]     r = func(a, **kwargs)
      [2018-08-18T00:45:21.894235Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3754, in _quantile_ureduce_func
      [2018-08-18T00:45:21.894237Z]     ap.partition(concatenate((indices_below, indices_above)), axis=axis)
      [2018-08-18T00:45:21.894239Z] TypeError: Partition index must be integer
      [2018-08-18T00:45:21.894241Z] ===================== 2 failed, 1 passed in 409.12 seconds =====================
      

      The fix is to edit lsst.validate.drp.calcsrd.measurePA2 to cast pf1Percentile to a float before calling np.percentile. I have built this without problem with numpy1.14.5, astropy3.0.3 and python 3.6.6.

      The problem is numpy 1.15. Using python 3.6.6, astropy 3.0.4 and numpy 1.15 also fails in the same way.

      Attachments

        Issue Links

          Activity

            tjenness Tim Jenness added a comment -

            If I modify measurePA2 to use a float runDecamQuickTest.sh does complete although it fails 12/16 measurements:

            $ ./scripts/runDecamQuickTest.sh -- --noplot
            Ingesting Raw data
            root INFO: Loading config overrride file '/Users/timj/work/lsstsw3/stack/DarwinX86/obs_decam/16.0-5-g9631a7a/config/ingest.py'
            CameraMapper INFO: Loading Posix exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/instcal0176837.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/dqmask0176837.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/wtmap0176837.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/instcal0176846.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/dqmask0176846.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/wtmap0176846.fits.fz
            running singleFrameDriver
            validating
            CameraMapper INFO: Loading exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/registry.sqlite3
            5301 sources in ccd 10  visit 176837
            3157 sources in ccd 10  visit 176846
            /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/fromnumeric.py:83: RuntimeWarning: invalid value encountered in reduce
              return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
            Photometric scatter (median) - SNR > 100.0 : 50.8 mmag
            /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py:794: OptimizeWarning: Covariance of the parameters could not be estimated
              category=OptimizeWarning)
            Median value of the astrometric scatter - all magnitudes: 93.447 marcsec
            Astrometric scatter (median) - snr > 100.0 : 65.8 marcsec
            No stars found that are 19.0 arcmin--21.0 arcmin apart.
            No stars found that are 199.0 arcmin--201.0 arcmin apart.
            =================================================================
            z band metric measurements
            =================================================================
            validate_drp.PA2_minimum_uzy - LPM-17, p. 21, http://ls.st/lpm-17
            At most PF1_minimum_uzy % of magnitudes may deviate by more
            than this magnitude from the mean.
            	Passed minimum      0.7705 mmag <= 22.5 mmag
            	Passed design       0.809 mmag <= 22.5 mmag
            	Passed stretch      0.8283 mmag <= 15 mmag
            validate_drp.PF1_minimum_uzy - LPM-17, p. 21, http://ls.st/lpm-17
            The maximum fraction of magnitudes deviating by more than
            PA2_minimum_uzy from mean.
            	Failed minimum      32.26 % <= 20 %
            	Failed design       32.26 % <= 10 %
            	Failed stretch      61.29 % <= 5 %
            validate_drp.PA1 - LPM-17, p. 21, http://ls.st/lpm-17
            The maximum rms of the unresolved source magnitude
            distribution around the mean value.
            	Failed design       24.79 mmag <= 7.5 mmag
            	Failed minimum      24.79 mmag <= 12 mmag
            	Failed stretch      24.79 mmag <= 4.5 mmag
            validate_drp.AD1_design - LPM-17, p. 23, http://ls.st/lpm-17
            No more than AF1 of the astrometric distances will deviate by
            more than this from the median (see AM1, AF1).
            	Failed design       60.38 marcsec <= 20 marcsec
            	Passed minimum      20.22 marcsec <= 40 marcsec
            	Failed stretch      88.09 marcsec <= 10 marcsec
            validate_drp.AD2_design - LPM-17, p. 23, http://ls.st/lpm-17
            No more than AF2 of the astrometric distances will deviate by
            more than this from the median (see AM2, AF2).
            	Skipped AD2.design   no measurement
            	Skipped AD2.minimum  no measurement
            	Skipped AD2.stretch  no measurement
            validate_drp.AD3_design - LPM-17, p. 23, http://ls.st/lpm-17
            No more than AF3 of the astrometric distances will deviate by
            more than this from the median (see AM3, AF3).
            	Skipped AD3.design   no measurement
            	Skipped AD3.minimum  no measurement
            	Skipped AD3.stretch  no measurement
            validate_drp.AF1_design - LPM-17, p. 23, http://ls.st/lpm-17
            The maximum fraction of astrometric distances which deviate
            by more than AD1 milliarcsec (see AM1).
            	Failed design       20.29 % <= 10 %
            	Passed minimum      13.04 % <= 20 %
            	Failed stretch      36.23 % <= 5 %
            validate_drp.AF2_design - LPM-17, p. 23, http://ls.st/lpm-17
            The maximum fraction of astrometric distances which deviate
            by more than AD2 milliarcsec (see AM2).
            	Skipped AF2.design   no measurement
            	Skipped AF2.minimum  no measurement
            	Skipped AF2.stretch  no measurement
            validate_drp.AF3_design - LPM-17, p. 23, http://ls.st/lpm-17
            The maximum fraction of astrometric distances which deviate
            by more than AD3 milliarcsec (see AM3).
            	Skipped AF3.design   no measurement
            	Skipped AF3.minimum  no measurement
            	Skipped AF3.stretch  no measurement
            validate_drp.AM1 - LPM-17, p. 23, http://ls.st/lpm-17
            The maximum rms of the astrometric distance distribution for
            stellar pairs with separations of D=5 arcmin (repeatability).
            	Failed design       23.87 marcsec <= 10 marcsec
            	Failed minimum      23.87 marcsec <= 20 marcsec
            	Failed stretch      23.87 marcsec <= 5 marcsec
            validate_drp.AM2 - LPM-17, p. 23, http://ls.st/lpm-17
            The maximum rms of the astrometric distance distribution for
            stellar pairs with separations of D=20 arcmin (repeatability).
            	Skipped AM2.design   no measurement
            	Skipped AM2.minimum  no measurement
            	Skipped AM2.stretch  no measurement
            validate_drp.AM3 - LPM-17, p. 23, http://ls.st/lpm-17
            The maximum rms of the astrometric distance distribution for
            stellar pairs with separations of D=200 arcmin (repeatability).
            	Skipped AM3.design   no measurement
            	Skipped AM3.minimum  no measurement
            	Skipped AM3.stretch  no measurement
            validate_drp.TE1 - LPM-17, p. 31, http://ls.st/lpm-17
            The averaged E1, E2, and Ex residual PSF ellipticity
            correlations must have median absolute value less than this for
            separations < 1 arcmin.
            	Failed design       0.0001572 <= 2e-05
            	Failed minimum      0.0001572 <= 3e-05
            	Failed stretch      0.0001572 <= 1e-05
            validate_drp.TE2 - LPM-17, p. 31, http://ls.st/lpm-17
            The averaged E1, E2, and Ex residual PSF ellipticity
            correlations must have median absolute value less than this for
            separations > 5 arcmin.
            	Passed design       1.248e-05 <= 5e-05
            	Passed minimum      1.248e-05 <= 3e-05
            	Failed stretch      1.248e-05 <= 1e-05
             
            =================================================================
            z band summary
            =================================================================
            Failed minimum      10 of 16 failed
            Failed design       12 of 16 failed
            Failed stretch      13 of 16 failed
            =================================================================
             
            =================================================================
            design level summary
            =================================================================
            FAILED (12/16 measurements)
            =================================================================
            

            tjenness Tim Jenness added a comment - If I modify measurePA2 to use a float runDecamQuickTest.sh does complete although it fails 12/16 measurements: $ ./scripts/runDecamQuickTest.sh -- --noplot Ingesting Raw data root INFO: Loading config overrride file '/Users/timj/work/lsstsw3/stack/DarwinX86/obs_decam/16.0-5-g9631a7a/config/ingest.py' CameraMapper INFO: Loading Posix exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/instcal0176837.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/dqmask0176837.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/wtmap0176837.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/instcal0176846.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/dqmask0176846.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/wtmap0176846.fits.fz running singleFrameDriver validating CameraMapper INFO: Loading exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/registry.sqlite3 5301 sources in ccd 10 visit 176837 3157 sources in ccd 10 visit 176846 /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/fromnumeric.py:83: RuntimeWarning: invalid value encountered in reduce return ufunc.reduce(obj, axis, dtype, out, **passkwargs) Photometric scatter (median) - SNR > 100.0 : 50.8 mmag /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py:794: OptimizeWarning: Covariance of the parameters could not be estimated category=OptimizeWarning) Median value of the astrometric scatter - all magnitudes: 93.447 marcsec Astrometric scatter (median) - snr > 100.0 : 65.8 marcsec No stars found that are 19.0 arcmin--21.0 arcmin apart. No stars found that are 199.0 arcmin--201.0 arcmin apart. ================================================================= z band metric measurements ================================================================= validate_drp.PA2_minimum_uzy - LPM-17, p. 21, http://ls.st/lpm-17 At most PF1_minimum_uzy % of magnitudes may deviate by more than this magnitude from the mean. Passed minimum 0.7705 mmag <= 22.5 mmag Passed design 0.809 mmag <= 22.5 mmag Passed stretch 0.8283 mmag <= 15 mmag validate_drp.PF1_minimum_uzy - LPM-17, p. 21, http://ls.st/lpm-17 The maximum fraction of magnitudes deviating by more than PA2_minimum_uzy from mean. Failed minimum 32.26 % <= 20 % Failed design 32.26 % <= 10 % Failed stretch 61.29 % <= 5 % validate_drp.PA1 - LPM-17, p. 21, http://ls.st/lpm-17 The maximum rms of the unresolved source magnitude distribution around the mean value. Failed design 24.79 mmag <= 7.5 mmag Failed minimum 24.79 mmag <= 12 mmag Failed stretch 24.79 mmag <= 4.5 mmag validate_drp.AD1_design - LPM-17, p. 23, http://ls.st/lpm-17 No more than AF1 of the astrometric distances will deviate by more than this from the median (see AM1, AF1). Failed design 60.38 marcsec <= 20 marcsec Passed minimum 20.22 marcsec <= 40 marcsec Failed stretch 88.09 marcsec <= 10 marcsec validate_drp.AD2_design - LPM-17, p. 23, http://ls.st/lpm-17 No more than AF2 of the astrometric distances will deviate by more than this from the median (see AM2, AF2). Skipped AD2.design no measurement Skipped AD2.minimum no measurement Skipped AD2.stretch no measurement validate_drp.AD3_design - LPM-17, p. 23, http://ls.st/lpm-17 No more than AF3 of the astrometric distances will deviate by more than this from the median (see AM3, AF3). Skipped AD3.design no measurement Skipped AD3.minimum no measurement Skipped AD3.stretch no measurement validate_drp.AF1_design - LPM-17, p. 23, http://ls.st/lpm-17 The maximum fraction of astrometric distances which deviate by more than AD1 milliarcsec (see AM1). Failed design 20.29 % <= 10 % Passed minimum 13.04 % <= 20 % Failed stretch 36.23 % <= 5 % validate_drp.AF2_design - LPM-17, p. 23, http://ls.st/lpm-17 The maximum fraction of astrometric distances which deviate by more than AD2 milliarcsec (see AM2). Skipped AF2.design no measurement Skipped AF2.minimum no measurement Skipped AF2.stretch no measurement validate_drp.AF3_design - LPM-17, p. 23, http://ls.st/lpm-17 The maximum fraction of astrometric distances which deviate by more than AD3 milliarcsec (see AM3). Skipped AF3.design no measurement Skipped AF3.minimum no measurement Skipped AF3.stretch no measurement validate_drp.AM1 - LPM-17, p. 23, http://ls.st/lpm-17 The maximum rms of the astrometric distance distribution for stellar pairs with separations of D=5 arcmin (repeatability). Failed design 23.87 marcsec <= 10 marcsec Failed minimum 23.87 marcsec <= 20 marcsec Failed stretch 23.87 marcsec <= 5 marcsec validate_drp.AM2 - LPM-17, p. 23, http://ls.st/lpm-17 The maximum rms of the astrometric distance distribution for stellar pairs with separations of D=20 arcmin (repeatability). Skipped AM2.design no measurement Skipped AM2.minimum no measurement Skipped AM2.stretch no measurement validate_drp.AM3 - LPM-17, p. 23, http://ls.st/lpm-17 The maximum rms of the astrometric distance distribution for stellar pairs with separations of D=200 arcmin (repeatability). Skipped AM3.design no measurement Skipped AM3.minimum no measurement Skipped AM3.stretch no measurement validate_drp.TE1 - LPM-17, p. 31, http://ls.st/lpm-17 The averaged E1, E2, and Ex residual PSF ellipticity correlations must have median absolute value less than this for separations < 1 arcmin. Failed design 0.0001572 <= 2e-05 Failed minimum 0.0001572 <= 3e-05 Failed stretch 0.0001572 <= 1e-05 validate_drp.TE2 - LPM-17, p. 31, http://ls.st/lpm-17 The averaged E1, E2, and Ex residual PSF ellipticity correlations must have median absolute value less than this for separations > 5 arcmin. Passed design 1.248e-05 <= 5e-05 Passed minimum 1.248e-05 <= 3e-05 Failed stretch 1.248e-05 <= 1e-05   ================================================================= z band summary ================================================================= Failed minimum 10 of 16 failed Failed design 12 of 16 failed Failed stretch 13 of 16 failed =================================================================   ================================================================= design level summary ================================================================= FAILED (12/16 measurements) =================================================================
            tjenness Tim Jenness added a comment -

            runDecamTest.sh doesn't work either.

            $ ./scripts/runDecamTest.sh -- --noplot
            Ingesting Raw data
            root INFO: Loading config overrride file '/Users/timj/work/lsstsw3/stack/DarwinX86/obs_decam/16.0-5-g9631a7a/config/ingest.py'
            CameraMapper INFO: Loading Posix exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/instcal0176837.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/dqmask0176837.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/wtmap0176837.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/instcal0176846.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/dqmask0176846.fits.fz
            ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/wtmap0176846.fits.fz
            running singleFrameDriver
            validating
            CameraMapper INFO: Loading exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/registry.sqlite3
            5301 sources in ccd 10  visit 176837
            5501 sources in ccd 11  visit 176837
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 12}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 12}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 13}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 13}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 14}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 14}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 15}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 15}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 16}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 16}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 17}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 17}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 18}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176837, 'ccdnum': 18}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 10}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 10}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 11}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 11}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 12}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 12}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 13}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 13}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 14}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 14}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 15}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 15}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 16}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 16}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 17}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 17}
            Skipping this dataId.
            No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 18}, tag=set())
            Could not open calibrated image file for  {'filter': 'z', 'visit': 176846, 'ccdnum': 18}
            Skipping this dataId.
            /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2920: RuntimeWarning: Mean of empty slice.
              out=out, **kwargs)
            /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars
              ret = ret.dtype.type(ret / rcount)
            Photometric scatter (median) - SNR > 100.0 : nan mmag
            Traceback (most recent call last):
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/bin/validateDrp.py", line 104, in <module>
                validate.run(args.repo, **kwargs)
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 152, in run
                metrics_package=metrics_package, **kwargs)
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
                metrics_package=metrics_package, **kwargs)
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 303, in runOneFilter
                photomModel = build_photometric_error_model(matchedDataset)
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 211, in build_photometric_error_model
                matchRef)
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 229, in _compute
                fit_params = fitPhotErrModel(mag[bright], magErr[bright])
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 130, in fitPhotErrModel
                photErrModel, mag, mag_err, p0=p0)
              File "/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py", line 751, in curve_fit
                res = leastsq(func, p0, Dfun=jac, full_output=1, **kwargs)
              File "/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py", line 386, in leastsq
                raise TypeError('Improper input: N=%s must not exceed M=%s' % (n, m))
            TypeError: Improper input: N=3 must not exceed M=0
            

            tjenness Tim Jenness added a comment - runDecamTest.sh doesn't work either. $ ./scripts/runDecamTest.sh -- --noplot Ingesting Raw data root INFO: Loading config overrride file '/Users/timj/work/lsstsw3/stack/DarwinX86/obs_decam/16.0-5-g9631a7a/config/ingest.py' CameraMapper INFO: Loading Posix exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/instcal0176837.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/dqmask0176837.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/wtmap0176837.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/instcal0176846.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/dqmask0176846.fits.fz ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/wtmap0176846.fits.fz running singleFrameDriver validating CameraMapper INFO: Loading exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/registry.sqlite3 5301 sources in ccd 10 visit 176837 5501 sources in ccd 11 visit 176837 No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 12}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 12} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 13}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 13} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 14}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 14} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 15}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 15} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 16}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 16} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 17}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 17} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 18}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 18} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 10}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 10} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 11}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 11} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 12}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 12} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 13}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 13} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 14}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 14} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 15}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 15} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 16}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 16} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 17}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 17} Skipping this dataId. No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 18}, tag=set()) Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 18} Skipping this dataId. /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2920: RuntimeWarning: Mean of empty slice. out=out, **kwargs) /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) Photometric scatter (median) - SNR > 100.0 : nan mmag Traceback (most recent call last): File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/bin/validateDrp.py", line 104, in <module> validate.run(args.repo, **kwargs) File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 152, in run metrics_package=metrics_package, **kwargs) File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo metrics_package=metrics_package, **kwargs) File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 303, in runOneFilter photomModel = build_photometric_error_model(matchedDataset) File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 211, in build_photometric_error_model matchRef) File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 229, in _compute fit_params = fitPhotErrModel(mag[bright], magErr[bright]) File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 130, in fitPhotErrModel photErrModel, mag, mag_err, p0=p0) File "/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py", line 751, in curve_fit res = leastsq(func, p0, Dfun=jac, full_output=1, **kwargs) File "/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py", line 386, in leastsq raise TypeError('Improper input: N=%s must not exceed M=%s' % (n, m)) TypeError: Improper input: N=3 must not exceed M=0

            One-line fix to pass numpy.percentile a value rather than an astropy Quantity. I compared the outputs from runDecamTest.sh using numpy 1.14 and numpy 1.15, and confirmed that they are the same after applying this fix.

            jcarlin Jeffrey Carlin added a comment - One-line fix to pass numpy.percentile a value rather than an astropy Quantity. I compared the outputs from runDecamTest.sh using numpy 1.14 and numpy 1.15, and confirmed that they are the same after applying this fix.
            tjenness Tim Jenness added a comment -

            Thank you for doing the right fix. Sorry that my broken fix messed things up. This did all wake me up to the scripts running in lsst_ci being far too lenient about the values they are receiving.

            tjenness Tim Jenness added a comment - Thank you for doing the right fix. Sorry that my broken fix messed things up. This did all wake me up to the scripts running in lsst_ci being far too lenient about the values they are receiving.

            Note: the crazy/unreasonable values that Tim saw after making this fix arose because casting an astropy Quantity with units "percent" to a float gives back the value scaled 0-1 (i.e., 75% is returned as 0.75). You must use the .value (or .to_value) method instead. Example (also showing the fail when passing an astropy quantity to np.percentile):

            >>> import numpy as np
            >>> aa = np.random.random(1000)
            >>> import astropy.units as u
            >>> pc = 75 * u.percent
            >>> pc
            <Quantity 75. %>
            >>> print(np.percentile(aa, pc))
            Traceback (most recent call last):
            {{ File "<stdin>", line 1, in <module>}}
            {{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3540, in percentile}}
            {{ a, q, axis, out, overwrite_input, interpolation, keepdims)}}
            {{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked}}
            {{ interpolation=interpolation)}}
            {{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce}}
            {{ r = func(a, **kwargs)}}
            {{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3754, in _quantile_ureduce_func}}
            {{ ap.partition(concatenate((indices_below, indices_above)), axis=axis)}}
            TypeError: Partition index must be integer
            >>> print(np.percentile(aa, pc.value))
            0.7310401145848625
            >>> print(np.percentile(aa, np.float(pc)))
            0.008769740232925993
            >>> np.float(pc)
            0.75

            jcarlin Jeffrey Carlin added a comment - Note: the crazy/unreasonable values that Tim saw after making this fix arose because casting an astropy Quantity with units "percent" to a float gives back the value scaled 0-1 (i.e., 75% is returned as 0.75). You must use the .value (or .to_value) method instead. Example (also showing the fail when passing an astropy quantity to np.percentile): >>> import numpy as np >>> aa = np.random.random(1000) >>> import astropy.units as u >>> pc = 75 * u.percent >>> pc <Quantity 75. %> >>> print(np.percentile(aa, pc)) Traceback (most recent call last): {{ File "<stdin>", line 1, in <module>}} {{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3540, in percentile}} {{ a, q, axis, out, overwrite_input, interpolation, keepdims)}} {{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked}} {{ interpolation=interpolation)}} {{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce}} {{ r = func(a, **kwargs)}} {{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3754, in _quantile_ureduce_func}} {{ ap.partition(concatenate((indices_below, indices_above)), axis=axis)}} TypeError: Partition index must be integer >>> print(np.percentile(aa, pc.value)) 0.7310401145848625 >>> print(np.percentile(aa, np.float(pc))) 0.008769740232925993 >>> np.float(pc) 0.75

            People

              jcarlin Jeffrey Carlin
              tjenness Tim Jenness
              Tim Jenness
              Jeffrey Carlin, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Simon Krughoff (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.