Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: validate_drp
-
Labels:None
-
Story Points:1
-
Sprint:Arch 2019-12-16
-
Team:Architecture
Description
I'm testing the science pipelines with numpy1.17/astropy4/matplotlib3.1 and validate_drp fails:
======================================================================
|
ERROR: testPlotMetricsFromJsonJob (__main__.ParseJsonJob)
|
Does plotting the metrics run and produce the correct filenames?
|
----------------------------------------------------------------------
|
Traceback (most recent call last):
|
File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/axis.py", line 1550, in convert_units
|
ret = self.converter.convert(x, self.units, self)
|
File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/astropy/visualization/units.py", line 101, in convert
|
elif isinstance(val, list) and isinstance(val[0], u.Quantity):
|
IndexError: list index out of range
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
Traceback (most recent call last):
|
File "tests/test_load_json.py", line 93, in testPlotMetricsFromJsonJob
|
plot_metrics(job, filterName)
|
File "/Users/timj/work/lsst/tmp/lsstsw/build/validate_drp/python/lsst/validate/drp/validate.py", line 455, in plot_metrics
|
outputPrefix=outputPrefix)
|
File "/Users/timj/work/lsst/tmp/lsstsw/build/validate_drp/python/lsst/validate/drp/plot.py", line 314, in plotPhotometryErrorModel
|
histtype='stepfilled', orientation='horizontal')
|
File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/__init__.py", line 1601, in inner
|
return func(ax, *map(sanitize_sequence, args), **kwargs)
|
File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 6694, in hist
|
x = [self.convert_xunits(xi) for xi in x]
|
File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 6694, in <listcomp>
|
x = [self.convert_xunits(xi) for xi in x]
|
File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/artist.py", line 180, in convert_xunits
|
return ax.xaxis.convert_units(x)
|
File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/axis.py", line 1553, in convert_units
|
f'units: {x!r}') from e
|
matplotlib.units.ConversionError: Failed to convert value(s) to axis units: []
|
The root cause seems to be that in plotPhotometryErrorModel the mmagRmsHighSnr variable is empty because there are no high signal to noise measurements. For the current astropy/numpy/matplotlib this triggers:
/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.
|
out=out, **kwargs)
|
/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars
|
ret = ret.dtype.type(ret / rcount)
|
but continues regardless. With newer versions it fails badly in Astropy with an index lookup gone bad. This is arguably a bug in Astropy that should be a bit more defensive, but it also seems to me that we shouldn't be asking matplotlib to calculate a histogram with no data in it.
The quick fix would seem to me to be to not try to plot the histogram if there are no data. Is that acceptable? Are we expecting there to be any numbers from the bright mask?
This does make the tests pass:
diff --git a/python/lsst/validate/drp/plot.py b/python/lsst/validate/drp/plot.py
|
index 36a3cff..3ab6c58 100644
|
--- a/python/lsst/validate/drp/plot.py
|
+++ b/python/lsst/validate/drp/plot.py
|
@@ -118,8 +118,9 @@ def plotAstrometryErrorModel(dataset, astromModel, outputPrefix=''):
|
|
ax[0].hist(dist, bins=100, color=color['all'],
|
histtype='stepfilled', orientation='horizontal')
|
- ax[0].hist(dist[bright], bins=100, color=color['bright'],
|
- histtype='stepfilled', orientation='horizontal')
|
+ if len(dist[bright]):
|
+ ax[0].hist(dist[bright], bins=100, color=color['bright'],
|
+ histtype='stepfilled', orientation='horizontal')
|
|
ax[0].set_ylim([0., 500.])
|
ax[0].set_ylabel("Distance [{unit:latex}]".format(unit=dist.unit))
|
@@ -307,10 +308,11 @@ def plotPhotometryErrorModel(dataset, photomModel,
|
ax[0][0].hist(mmagRms,
|
bins=100, range=(0, 500), color=color['all'],
|
histtype='stepfilled', orientation='horizontal')
|
- ax[0][0].hist(mmagRmsHighSnr,
|
- bins=100, range=(0, 500),
|
- color=color['bright'],
|
- histtype='stepfilled', orientation='horizontal')
|
+ if len(mmagRmsHighSnr):
|
+ ax[0][0].hist(mmagRmsHighSnr,
|
+ bins=100, range=(0, 500),
|
+ color=color['bright'],
|
+ histtype='stepfilled', orientation='horizontal')
|
plotOutlinedAxline(
|
ax[0][0].axhline,
|
mmagrms_median.value, |
Attachments
Issue Links
- is triggered by
-
- Done
- links to