# ValueError in coaddAnalysis with HSC-RC2 tract=9615 filter=HSC-R

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
3
• Sprint:
DRP F18-6
• Team:
Data Release Production

## Description

coaddAnalysis with HSC-RC2 tract=9615 filter=HSC-R data gave the following error:

 Traceback (most recent call last):  File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-13-gb122224+8/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__  result = self.runTask(task, dataRef, kwargs)  File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-13-gb122224+8/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask  return task.runDataRef(dataRef, **kwargs)  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 334, in runDataRef  postFix="_unforced")  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 577, in plotSizes  labeller=StarGalaxyLabeller(), flagsCat=flagsCat,  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/analysis.py", line 104, in __init__  self.good = np.isfinite(self.quantity) & np.isfinite(self.mag) if self.quantity is not None else None ValueError: operands could not be broadcast together with shapes (23872,) (23873,) 

Other tracts/filters in HSC-RC2 finish without errors. A command to reproduce is

 coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_42/DM-16095:private/user/name/coaddAnalysis --id tract=9615 filter=HSC-R 

This used to finish fine in early September with w_2018_36 (before the DM-15869 issue).

## Activity

Hide
Tim Jenness added a comment -

I'm not sure if my comment is relevant either. It just seemed weird that two bug reports appeared close to each complaining about mismatched data arrays being combined.

Show
Tim Jenness added a comment - I'm not sure if my comment is relevant either. It just seemed weird that two bug reports appeared close to each complaining about mismatched data arrays being combined.
Hide
Lauren MacArthur added a comment -

I tracked this down to a single object in that tract that had the calib_psf_used flag set in the coadd catalogs, but for which the shape measurement failed. I wasn't selecting against shape flags, but nan's are excluded, so one array was ending up with N - 1 entries in a column that was being "broadcast" with one that had the full N entries. The reason we did not see this before was because the calib_* flags were not getting propagated to the coadd catalogs, so these plots were getting skipped.

While an object that was used in visit-level PSF modeling failing to get a shape measurement on the coadd is a bit surprising (and, indeed, rare...only a single case in three tracts), the reason this can happen is that the calib_psf_used flag only requires that 20% of the visits contributing have that flag set (so perhaps there were enough visits contributing that were funky in that area that the resulting object in the coadd was also too funky to get a proper fit).

I've fixed the problem by updating my flag culling in those plot functions.

Show
Lauren MacArthur added a comment - I tracked this down to a single object in that tract that had the calib_psf_used flag set in the coadd catalogs, but for which the shape measurement failed. I wasn't selecting against shape flags, but nan's are excluded, so one array was ending up with N - 1 entries in a column that was being "broadcast" with one that had the full N entries. The reason we did not see this before was because the calib_ * flags were not getting propagated to the coadd catalogs, so these plots were getting skipped. While an object that was used in visit-level PSF modeling failing to get a shape measurement on the coadd is a bit surprising (and, indeed, rare...only a single case in three tracts), the reason this can happen is that the calib_psf_used flag only requires that 20% of the visits contributing have that flag set (so perhaps there were enough visits contributing that were funky in that area that the resulting object in the coadd was also too funky to get a proper fit). I've fixed the problem by updating my flag culling in those plot functions.
Hide
Lauren MacArthur added a comment -

Would you mind giving this a look?  I successfully ran through the tract/band that was failing.

Show
Lauren MacArthur added a comment - Would you mind giving this a look?  I successfully ran through the tract/band that was failing.
Hide
Hsin-Fang Chiang added a comment -

LGTM. Thanks for tracking this down!

Show
Hsin-Fang Chiang added a comment - LGTM. Thanks for tracking this down!
Hide
Lauren MacArthur added a comment -

Thank YOU for spotting and reporting it (and the review)!  Merged to master.

Show
Lauren MacArthur added a comment - Thank YOU for spotting and reporting it (and the review)!  Merged to master.

## People

• Assignee:
Lauren MacArthur
Reporter:
Hsin-Fang Chiang
Reviewers:
Hsin-Fang Chiang
Watchers:
Hsin-Fang Chiang, John Swinbank, Lauren MacArthur, Tim Jenness