Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_analysis
-
Labels:None
-
Story Points:3
-
Epic Link:
-
Sprint:DRP F18-6
-
Team:Data Release Production
Description
coaddAnalysis with HSC-RC2 tract=9615 filter=HSC-R data gave the following error:
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-13-gb122224+8/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
|
result = self.runTask(task, dataRef, kwargs)
|
File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-13-gb122224+8/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
|
return task.runDataRef(dataRef, **kwargs)
|
File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 334, in runDataRef
|
postFix="_unforced")
|
File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 577, in plotSizes
|
labeller=StarGalaxyLabeller(), flagsCat=flagsCat,
|
File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/analysis.py", line 104, in __init__
|
self.good = np.isfinite(self.quantity) & np.isfinite(self.mag) if self.quantity is not None else None
|
ValueError: operands could not be broadcast together with shapes (23872,) (23873,)
|
Other tracts/filters in HSC-RC2 finish without errors. A command to reproduce is
coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_42/DM-16095:private/user/name/coaddAnalysis --id tract=9615 filter=HSC-R
|
This used to finish fine in early September with w_2018_36 (before the DM-15869 issue).
Attachments
Activity
I tracked this down to a single object in that tract that had the calib_psf_used flag set in the coadd catalogs, but for which the shape measurement failed. I wasn't selecting against shape flags, but nan's are excluded, so one array was ending up with N - 1 entries in a column that was being "broadcast" with one that had the full N entries. The reason we did not see this before was because the calib_* flags were not getting propagated to the coadd catalogs, so these plots were getting skipped.
While an object that was used in visit-level PSF modeling failing to get a shape measurement on the coadd is a bit surprising (and, indeed, rare...only a single case in three tracts), the reason this can happen is that the calib_psf_used flag only requires that 20% of the visits contributing have that flag set (so perhaps there were enough visits contributing that were funky in that area that the resulting object in the coadd was also too funky to get a proper fit).
I've fixed the problem by updating my flag culling in those plot functions.
Would you mind giving this a look? I successfully ran through the tract/band that was failing.
LGTM. Thanks for tracking this down!
Hsin-Fang Chiang
added a comment - LGTM. Thanks for tracking this down! Thank YOU for spotting and reporting it (and the review)! Merged to master.
I'm not sure if my comment is relevant either. It just seemed weird that two bug reports appeared close to each complaining about mismatched data arrays being combined.