Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: lsst_ci, validate_drp
-
Labels:None
-
Team:External
Description
Testing bin/deploy -b with DM-15385, DM-15419, DM-15394, DM-15389, and DM-15443 (Python 3.7.0, numpy 1.15.0, and astropy 3.0.4) I get the following error from lsst_ci:
[2018-08-18T00:45:21.894199Z] Traceback (most recent call last):
|
[2018-08-18T00:45:21.894202Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/bin/validateDrp.py", line 104, in <module>
|
[2018-08-18T00:45:21.894204Z] validate.run(args.repo, **kwargs)
|
[2018-08-18T00:45:21.894206Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 152, in run
|
[2018-08-18T00:45:21.894208Z] metrics_package=metrics_package, **kwargs)
|
[2018-08-18T00:45:21.894210Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
|
[2018-08-18T00:45:21.894212Z] metrics_package=metrics_package, **kwargs)
|
[2018-08-18T00:45:21.894214Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 350, in runOneFilter
|
[2018-08-18T00:45:21.894216Z] pa2 = measurePA2(metrics[pa2_spec.metric_name], pa1, pf1_spec.threshold)
|
[2018-08-18T00:45:21.894218Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/calcsrd/pa2.py", line 62, in measurePA2
|
[2018-08-18T00:45:21.894220Z] return Measurement(metric, np.percentile(np.abs(magDiffs), pf1Percentile) * magDiffs.unit,
|
[2018-08-18T00:45:21.894222Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3540, in percentile
|
[2018-08-18T00:45:21.894224Z] a, q, axis, out, overwrite_input, interpolation, keepdims)
|
[2018-08-18T00:45:21.894226Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked
|
[2018-08-18T00:45:21.894229Z] interpolation=interpolation)
|
[2018-08-18T00:45:21.894231Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce
|
[2018-08-18T00:45:21.894233Z] r = func(a, **kwargs)
|
[2018-08-18T00:45:21.894235Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3754, in _quantile_ureduce_func
|
[2018-08-18T00:45:21.894237Z] ap.partition(concatenate((indices_below, indices_above)), axis=axis)
|
[2018-08-18T00:45:21.894239Z] TypeError: Partition index must be integer
|
[2018-08-18T00:45:21.894241Z] ===================== 2 failed, 1 passed in 409.12 seconds =====================
|
The fix is to edit lsst.validate.drp.calcsrd.measurePA2 to cast pf1Percentile to a float before calling np.percentile. I have built this without problem with numpy1.14.5, astropy3.0.3 and python 3.6.6.
The problem is numpy 1.15. Using python 3.6.6, astropy 3.0.4 and numpy 1.15 also fails in the same way.
Attachments
Issue Links
- relates to
-
DM-16289 validate_drp seems to fail on Python 3.7
- Invalid
Activity
runDecamTest.sh doesn't work either.
$ ./scripts/runDecamTest.sh -- --noplot
|
Ingesting Raw data
|
root INFO: Loading config overrride file '/Users/timj/work/lsstsw3/stack/DarwinX86/obs_decam/16.0-5-g9631a7a/config/ingest.py'
|
CameraMapper INFO: Loading Posix exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input
|
ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/instcal0176837.fits.fz
|
ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/dqmask0176837.fits.fz
|
ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176837/wtmap0176837.fits.fz
|
ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/instcal0176846.fits.fz
|
ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/dqmask0176846.fits.fz
|
ingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/0176846/wtmap0176846.fits.fz
|
running singleFrameDriver
|
validating
|
CameraMapper INFO: Loading exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/Decam/input/registry.sqlite3
|
5301 sources in ccd 10 visit 176837
|
5501 sources in ccd 11 visit 176837
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 12}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 12}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 13}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 13}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 14}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 14}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 15}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 15}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 16}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 16}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 17}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 17}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176837, 'ccdnum': 18}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176837, 'ccdnum': 18}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 10}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 10}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 11}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 11}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 12}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 12}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 13}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 13}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 14}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 14}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 15}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 15}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 16}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 16}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 17}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 17}
|
Skipping this dataId.
|
No locations for get: datasetType:calexp_calib dataId:DataId(initialdata={'filter': 'z', 'visit': 176846, 'ccdnum': 18}, tag=set())
|
Could not open calibrated image file for {'filter': 'z', 'visit': 176846, 'ccdnum': 18}
|
Skipping this dataId.
|
/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2920: RuntimeWarning: Mean of empty slice.
|
out=out, **kwargs)
|
/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars
|
ret = ret.dtype.type(ret / rcount)
|
Photometric scatter (median) - SNR > 100.0 : nan mmag
|
Traceback (most recent call last):
|
File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/bin/validateDrp.py", line 104, in <module>
|
validate.run(args.repo, **kwargs)
|
File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 152, in run
|
metrics_package=metrics_package, **kwargs)
|
File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
|
metrics_package=metrics_package, **kwargs)
|
File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 303, in runOneFilter
|
photomModel = build_photometric_error_model(matchedDataset)
|
File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 211, in build_photometric_error_model
|
matchRef)
|
File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 229, in _compute
|
fit_params = fitPhotErrModel(mag[bright], magErr[bright])
|
File "/Users/timj/work/lsstsw3/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/photerrmodel.py", line 130, in fitPhotErrModel
|
photErrModel, mag, mag_err, p0=p0)
|
File "/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py", line 751, in curve_fit
|
res = leastsq(func, p0, Dfun=jac, full_output=1, **kwargs)
|
File "/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py", line 386, in leastsq
|
raise TypeError('Improper input: N=%s must not exceed M=%s' % (n, m))
|
TypeError: Improper input: N=3 must not exceed M=0
|
One-line fix to pass numpy.percentile a value rather than an astropy Quantity. I compared the outputs from runDecamTest.sh using numpy 1.14 and numpy 1.15, and confirmed that they are the same after applying this fix.
Thank you for doing the right fix. Sorry that my broken fix messed things up. This did all wake me up to the scripts running in lsst_ci being far too lenient about the values they are receiving.
Note: the crazy/unreasonable values that Tim saw after making this fix arose because casting an astropy Quantity with units "percent" to a float gives back the value scaled 0-1 (i.e., 75% is returned as 0.75). You must use the .value (or .to_value) method instead. Example (also showing the fail when passing an astropy quantity to np.percentile):
>>> import numpy as np
>>> aa = np.random.random(1000)
>>> import astropy.units as u
>>> pc = 75 * u.percent
>>> pc
<Quantity 75. %>
>>> print(np.percentile(aa, pc))
Traceback (most recent call last):
{{ File "<stdin>", line 1, in <module>}}
{{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3540, in percentile}}
{{ a, q, axis, out, overwrite_input, interpolation, keepdims)}}
{{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked}}
{{ interpolation=interpolation)}}
{{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce}}
{{ r = func(a, **kwargs)}}
{{ File "/home/jcarlin/.local/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3754, in _quantile_ureduce_func}}
{{ ap.partition(concatenate((indices_below, indices_above)), axis=axis)}}
TypeError: Partition index must be integer
>>> print(np.percentile(aa, pc.value))
0.7310401145848625
>>> print(np.percentile(aa, np.float(pc)))
0.008769740232925993
>>> np.float(pc)
0.75
If I modify measurePA2 to use a float runDecamQuickTest.sh does complete although it fails 12/16 measurements:
$ ./scripts/runDecamQuickTest.sh -- --noplotIngesting Raw dataroot INFO: Loading config overrride file '/Users/timj/work/lsstsw3/stack/DarwinX86/obs_decam/16.0-5-g9631a7a/config/ingest.py'CameraMapper INFO: Loading Posix exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/inputingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/instcal0176837.fits.fzingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/dqmask0176837.fits.fzingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176837.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176837/wtmap0176837.fits.fzingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/instcal/instcal0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/instcal0176846.fits.fzingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/dqmask/dqmask0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/dqmask0176846.fits.fzingest INFO: /Users/timj/work/lsstsw3/stack/DarwinX86/validation_data_decam/16.0+1/wtmap/wtmap0176846.fits.fz --<link>--> /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/0176846/wtmap0176846.fits.fzrunning singleFrameDrivervalidatingCameraMapper INFO: Loading exposure registry from /Users/timj/work/lsstsw3/build/lsst_ci/DecamQuick/input/registry.sqlite35301 sources in ccd 10 visit 1768373157 sources in ccd 10 visit 176846/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/numpy/core/fromnumeric.py:83: RuntimeWarning: invalid value encountered in reducereturn ufunc.reduce(obj, axis, dtype, out, **passkwargs)Photometric scatter (median) - SNR > 100.0 : 50.8 mmag/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.6/site-packages/scipy/optimize/minpack.py:794: OptimizeWarning: Covariance of the parameters could not be estimatedcategory=OptimizeWarning)Median value of the astrometric scatter - all magnitudes: 93.447 marcsecAstrometric scatter (median) - snr > 100.0 : 65.8 marcsecNo stars found that are 19.0 arcmin--21.0 arcmin apart.No stars found that are 199.0 arcmin--201.0 arcmin apart.=================================================================z band metric measurements=================================================================validate_drp.PA2_minimum_uzy - LPM-17, p. 21, http://ls.st/lpm-17At most PF1_minimum_uzy % of magnitudes may deviate by morethan this magnitude from the mean.Passed minimum 0.7705 mmag <= 22.5 mmagPassed design 0.809 mmag <= 22.5 mmagPassed stretch 0.8283 mmag <= 15 mmagvalidate_drp.PF1_minimum_uzy - LPM-17, p. 21, http://ls.st/lpm-17The maximum fraction of magnitudes deviating by more thanPA2_minimum_uzy from mean.Failed minimum 32.26 % <= 20 %Failed design 32.26 % <= 10 %Failed stretch 61.29 % <= 5 %validate_drp.PA1 - LPM-17, p. 21, http://ls.st/lpm-17The maximum rms of the unresolved source magnitudedistribution around the mean value.Failed design 24.79 mmag <= 7.5 mmagFailed minimum 24.79 mmag <= 12 mmagFailed stretch 24.79 mmag <= 4.5 mmagvalidate_drp.AD1_design - LPM-17, p. 23, http://ls.st/lpm-17No more than AF1 of the astrometric distances will deviate bymore than this from the median (see AM1, AF1).Failed design 60.38 marcsec <= 20 marcsecPassed minimum 20.22 marcsec <= 40 marcsecFailed stretch 88.09 marcsec <= 10 marcsecvalidate_drp.AD2_design - LPM-17, p. 23, http://ls.st/lpm-17No more than AF2 of the astrometric distances will deviate bymore than this from the median (see AM2, AF2).Skipped AD2.design no measurementSkipped AD2.minimum no measurementSkipped AD2.stretch no measurementvalidate_drp.AD3_design - LPM-17, p. 23, http://ls.st/lpm-17No more than AF3 of the astrometric distances will deviate bymore than this from the median (see AM3, AF3).Skipped AD3.design no measurementSkipped AD3.minimum no measurementSkipped AD3.stretch no measurementvalidate_drp.AF1_design - LPM-17, p. 23, http://ls.st/lpm-17The maximum fraction of astrometric distances which deviateby more than AD1 milliarcsec (see AM1).Failed design 20.29 % <= 10 %Passed minimum 13.04 % <= 20 %Failed stretch 36.23 % <= 5 %validate_drp.AF2_design - LPM-17, p. 23, http://ls.st/lpm-17The maximum fraction of astrometric distances which deviateby more than AD2 milliarcsec (see AM2).Skipped AF2.design no measurementSkipped AF2.minimum no measurementSkipped AF2.stretch no measurementvalidate_drp.AF3_design - LPM-17, p. 23, http://ls.st/lpm-17The maximum fraction of astrometric distances which deviateby more than AD3 milliarcsec (see AM3).Skipped AF3.design no measurementSkipped AF3.minimum no measurementSkipped AF3.stretch no measurementvalidate_drp.AM1 - LPM-17, p. 23, http://ls.st/lpm-17The maximum rms of the astrometric distance distribution forstellar pairs with separations of D=5 arcmin (repeatability).Failed design 23.87 marcsec <= 10 marcsecFailed minimum 23.87 marcsec <= 20 marcsecFailed stretch 23.87 marcsec <= 5 marcsecvalidate_drp.AM2 - LPM-17, p. 23, http://ls.st/lpm-17The maximum rms of the astrometric distance distribution forstellar pairs with separations of D=20 arcmin (repeatability).Skipped AM2.design no measurementSkipped AM2.minimum no measurementSkipped AM2.stretch no measurementvalidate_drp.AM3 - LPM-17, p. 23, http://ls.st/lpm-17The maximum rms of the astrometric distance distribution forstellar pairs with separations of D=200 arcmin (repeatability).Skipped AM3.design no measurementSkipped AM3.minimum no measurementSkipped AM3.stretch no measurementvalidate_drp.TE1 - LPM-17, p. 31, http://ls.st/lpm-17The averaged E1, E2, and Ex residual PSF ellipticitycorrelations must have median absolute value less than this forseparations < 1 arcmin.Failed design 0.0001572 <= 2e-05Failed minimum 0.0001572 <= 3e-05Failed stretch 0.0001572 <= 1e-05validate_drp.TE2 - LPM-17, p. 31, http://ls.st/lpm-17The averaged E1, E2, and Ex residual PSF ellipticitycorrelations must have median absolute value less than this forseparations > 5 arcmin.Passed design 1.248e-05 <= 5e-05Passed minimum 1.248e-05 <= 3e-05Failed stretch 1.248e-05 <= 1e-05=================================================================z band summary=================================================================Failed minimum 10 of 16 failedFailed design 12 of 16 failedFailed stretch 13 of 16 failed==================================================================================================================================design level summary=================================================================FAILED (12/16 measurements)=================================================================