Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: meas_algorithms, pipe_tasks
-
Labels:None
-
Team:External
Description
This error is seen near the edge of the COSMOS tract. It used to write out a tiny corner of a coadd.
Traceback (most recent call last):
|
File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-1-ga2912ff+25/python/lsst/ctrl/pool/parallel.py", line 496, in logOperation |
yield
|
File "/home/hchiang2/stack/pipe_drivers/python/lsst/pipe/drivers/coaddDriver.py", line 316, in coadd |
detResults = self.detectCoaddSources.runDetection(coadd, idFactory, expId=expId)
|
File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-39-g03bf09b5+1/python/lsst/pipe/tasks/multiBand.py", line 303, in runDetection |
detections = self.detection.makeSourceCatalog(table, exposure, expId=expId)
|
File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+33/python/lsst/pipe/base/timer.py", line 150, in wrapper |
res = func(self, *args, **keyArgs)
|
File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-10-g59393990+2/python/lsst/meas/algorithms/detection.py", line 292, in run |
clearMask=clearMask, expId=expId)
|
File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-10-g59393990+2/python/lsst/meas/algorithms/dynamicDetection.py", line 180, in detectFootprints |
factor = self.calculateThreshold(exposure, seed, sigma=sigma)
|
File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-10-g59393990+2/python/lsst/meas/algorithms/dynamicDetection.py", line 107, in calculateThreshold |
lq, uq = np.percentile(fluxes, [25.0, 75.0]) |
File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4269, in percentile |
interpolation=interpolation)
|
File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4011, in _ureduce |
r = func(a, **kwargs)
|
File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4386, in _percentile |
x1 = take(ap, indices_below, axis=axis) * weights_below
|
File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 134, in take |
return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode) |
File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc |
return getattr(obj, method)(*args, **kwds) |
IndexError: cannot do a non-empty take from an empty axes. |
Attachments
Issue Links
Activity
In w_2018_12's RC2 run I'm getting fewer of this error, compared to w_2018_10. I'd guess DM-13741 might be related.
Hsin-Fang Chiang
added a comment - In w_2018_12's RC2 run I'm getting fewer of this error, compared to w_2018_10. I'd guess DM-13741 might be related. Hsin-Fang Chiang, would you be willing to review these small fixes?
price@pap-laptop-2:~/LSST/meas_algorithms (tickets/DM-13517=) $ git sub
|
commit be77d0d7aa0b4b7c271e926fc9d5cf5c12f6367a (HEAD -> tickets/DM-13517, origin/tickets/DM-13517)
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Tue Apr 10 15:21:32 2018 -0400
|
|
|
dynamicDetection: require sufficient sources for statistics
|
|
Makes the minimum number of sources for statistics configurable,
|
avoiding obscure numpy errors from having zero sources.
|
|
|
python/lsst/meas/algorithms/dynamicDetection.py | 9 +++++++++
|
tests/test_dynamicDetection.py | 4 ++++
|
2 files changed, 13 insertions(+)
|
|
|
|
|
price@pap-laptop-2:~/LSST/pipe_tasks (tickets/DM-13517=) $ git sub
|
commit 2fd1a86b4b035d32a70ee87724f5eefb9babe4bc
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Tue Apr 10 15:29:25 2018 -0400
|
|
|
scaleVariance: catch NAN rescaling factor
|
|
If the rescaling factor is NAN, *bad* things can happen...
|
|
|
python/lsst/pipe/tasks/scaleVariance.py | 4 ++--
|
1 file changed, 2 insertions(+), 2 deletions(-)
|
|
|
commit 1e3c7fecc89a3b12bbdb90018b88a6ef165fd356
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Tue Apr 10 15:30:30 2018 -0400
|
|
|
scaleVariance: ensure variance is postive
|
|
Imaginary universes come crashing down on you when the variance is negative.
|
|
|
python/lsst/pipe/tasks/scaleVariance.py | 4 ++--
|
1 file changed, 2 insertions(+), 2 deletions(-)
|
|
|
commit 0f5851be82104accccf88cf8ce784c225dea707c (HEAD -> tickets/DM-13517, origin/tickets/DM-13517)
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Tue Apr 10 15:31:02 2018 -0400
|
|
|
scaleVariance: fix undefined variable
|
|
maskVal wasn't defined.
|
|
|
python/lsst/pipe/tasks/scaleVariance.py | 1 +
|
1 file changed, 1 insertion(+)
|
Paul Price
added a comment - Hsin-Fang Chiang , would you be willing to review these small fixes?
price@pap-laptop-2:~/LSST/meas_algorithms (tickets/DM-13517=) $ git sub
commit be77d0d7aa0b4b7c271e926fc9d5cf5c12f6367a (HEAD -> tickets/DM-13517, origin/tickets/DM-13517)
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Apr 10 15:21:32 2018 -0400
dynamicDetection: require sufficient sources for statistics
Makes the minimum number of sources for statistics configurable,
avoiding obscure numpy errors from having zero sources.
python/lsst/meas/algorithms/dynamicDetection.py | 9 +++++++++
tests/test_dynamicDetection.py | 4 ++++
2 files changed, 13 insertions(+)
price@pap-laptop-2:~/LSST/pipe_tasks (tickets/DM-13517=) $ git sub
commit 2fd1a86b4b035d32a70ee87724f5eefb9babe4bc
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Apr 10 15:29:25 2018 -0400
scaleVariance: catch NAN rescaling factor
If the rescaling factor is NAN, *bad* things can happen...
python/lsst/pipe/tasks/scaleVariance.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
commit 1e3c7fecc89a3b12bbdb90018b88a6ef165fd356
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Apr 10 15:30:30 2018 -0400
scaleVariance: ensure variance is postive
Imaginary universes come crashing down on you when the variance is negative.
python/lsst/pipe/tasks/scaleVariance.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
commit 0f5851be82104accccf88cf8ce784c225dea707c (HEAD -> tickets/DM-13517, origin/tickets/DM-13517)
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Apr 10 15:31:02 2018 -0400
scaleVariance: fix undefined variable
maskVal wasn't defined.
python/lsst/pipe/tasks/scaleVariance.py | 1 +
1 file changed, 1 insertion(+)
The error was seen at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7' in w_2018_14's RC2 reprocessing (DM-13890), and can be reproduced with w_2018_14:
coaddDriver.py /datasets/hsc/repo/ --rerun RC/w_2018_14/DM-13890:private/hchiang2/DM-13517 --id tract=9813 patch=0,7 filter=HSC-Z --selectId visit=17930 -c makeCoaddTempExp.doApplySkyCorr=False --cores 1 |
With the ticket branches, the same command runs without the error. Got
coaddDriver.detectCoaddSources.detection WARN: Insufficient good flux measurements (0 < 10) for dynamic threshold calculation |
as expected and finished detectCoaddSources.
Hsin-Fang Chiang
added a comment - The error was seen at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7' in w_2018_14's RC2 reprocessing ( DM-13890 ), and can be reproduced with w_2018_14 :
coaddDriver.py /datasets/hsc/repo/ --rerun RC/w_2018_14/DM- 13890 : private /hchiang2/DM- 13517 --id tract= 9813 patch= 0 , 7 filter=HSC-Z --selectId visit= 17930 -c makeCoaddTempExp.doApplySkyCorr=False --cores 1
With the ticket branches, the same command runs without the error. Got
coaddDriver.detectCoaddSources.detection WARN: Insufficient good flux measurements ( 0 < 10 ) for dynamic threshold calculation
as expected and finished detectCoaddSources. Jenkins seems to have passed: failed on two elements because it couldn't find the demo script. But ci_hsc passed on all, and my changes should have nothing to do with the demo.
Paul Price
added a comment - Jenkins seems to have passed: failed on two elements because it couldn't find the demo script. But ci_hsc passed on all, and my changes should have nothing to do with the demo. Merged to master.
Thanks Hsin-Fang Chiang!
Paul Price
added a comment - Merged to master.
Thanks Hsin-Fang Chiang ! 
The above error was with w_2018_04 plus a bug fix of
DM-13396.This can also be reproduced with w_2018_05
`