Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16363

ValueError in coaddAnalysis with HSC-RC2 tract=9615 filter=HSC-R

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      3
    • Epic Link:
    • Sprint:
      DRP F18-6
    • Team:
      Data Release Production

      Description

      coaddAnalysis with HSC-RC2 tract=9615 filter=HSC-R data gave the following error:

      Traceback (most recent call last):
        File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-13-gb122224+8/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-13-gb122224+8/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 334, in runDataRef
          postFix="_unforced")
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 577, in plotSizes
          labeller=StarGalaxyLabeller(), flagsCat=flagsCat,
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/analysis.py", line 104, in __init__
          self.good = np.isfinite(self.quantity) & np.isfinite(self.mag) if self.quantity is not None else None
      ValueError: operands could not be broadcast together with shapes (23872,) (23873,)
      

      Other tracts/filters in HSC-RC2 finish without errors. A command to reproduce is

      coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB  --rerun RC/w_2018_42/DM-16095:private/user/name/coaddAnalysis  --id tract=9615 filter=HSC-R 
      

      This used to finish fine in early September with w_2018_36 (before the DM-15869 issue).

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          I'm not sure if my comment is relevant either. It just seemed weird that two bug reports appeared close to each complaining about mismatched data arrays being combined.

          Show
          tjenness Tim Jenness added a comment - I'm not sure if my comment is relevant either. It just seemed weird that two bug reports appeared close to each complaining about mismatched data arrays being combined.
          Hide
          lauren Lauren MacArthur added a comment -

          I tracked this down to a single object in that tract that had the calib_psf_used flag set in the coadd catalogs, but for which the shape measurement failed. I wasn't selecting against shape flags, but nan's are excluded, so one array was ending up with N - 1 entries in a column that was being "broadcast" with one that had the full N entries. The reason we did not see this before was because the calib_* flags were not getting propagated to the coadd catalogs, so these plots were getting skipped.

          While an object that was used in visit-level PSF modeling failing to get a shape measurement on the coadd is a bit surprising (and, indeed, rare...only a single case in three tracts), the reason this can happen is that the calib_psf_used flag only requires that 20% of the visits contributing have that flag set (so perhaps there were enough visits contributing that were funky in that area that the resulting object in the coadd was also too funky to get a proper fit).

          I've fixed the problem by updating my flag culling in those plot functions.

          Show
          lauren Lauren MacArthur added a comment - I tracked this down to a single object in that tract that had the calib_psf_used flag set in the coadd catalogs, but for which the shape measurement failed. I wasn't selecting against shape flags, but nan's are excluded, so one array was ending up with N - 1 entries in a column that was being "broadcast" with one that had the full N entries. The reason we did not see this before was because the calib_ * flags were not getting propagated to the coadd catalogs, so these plots were getting skipped. While an object that was used in visit-level PSF modeling failing to get a shape measurement on the coadd is a bit surprising (and, indeed, rare...only a single case in three tracts), the reason this can happen is that the calib_psf_used flag only requires that 20% of the visits contributing have that flag set (so perhaps there were enough visits contributing that were funky in that area that the resulting object in the coadd was also too funky to get a proper fit). I've fixed the problem by updating my flag culling in those plot functions.
          Hide
          lauren Lauren MacArthur added a comment -

          Would you mind giving this a look?  I successfully ran through the tract/band that was failing.

          Show
          lauren Lauren MacArthur added a comment - Would you mind giving this a look?  I successfully ran through the tract/band that was failing.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          LGTM. Thanks for tracking this down! 

          Show
          hchiang2 Hsin-Fang Chiang added a comment - LGTM. Thanks for tracking this down! 
          Hide
          lauren Lauren MacArthur added a comment -

          Thank YOU for spotting and reporting it (and the review)!  Merged to master.

          Show
          lauren Lauren MacArthur added a comment - Thank YOU for spotting and reporting it (and the review)!  Merged to master.

            People

            • Assignee:
              lauren Lauren MacArthur
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, John Swinbank, Lauren MacArthur, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel