Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15133

Empty matches in coaddAnalysis.py at COSMOS field?

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      2
    • Epic Link:
    • Sprint:
      DRP F18-2
    • Team:
      Data Release Production

      Description

      Using w_2018_28 and pipe_analysis at commit ebd48cf, coaddAnalysis.py failed in the COSMOS tract=9813 filters HSC-G, HSC-R, HSC-Z, and HSC-Y:

      Traceback (most recent call last):
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-2-g852da13+6/python/lsst/pipe/base/cmdLineTask.py", line 392, in __call__
          result = task.run(dataRef, **kwargs)
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 348, in run
          hscRun=repoInfo.hscRun, wcs=repoInfo.wcs)
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 450, in readSrcMatches
          if not hasattr(matches[0].first, "schema"):
      IndexError: list index out of range
      

      The command I ran was
      coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_28/DM-14988 --id tract=9813 filter=HSC-G -c doWriteParquetTables=True
      (and 3 other filters)

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            So far I am unable to reproduce the failed coaddAnalysis issues.  I tried running the current master (just a trivial commit ahead of where you ran it) on just a handful of patches of HSC-G:

            coaddAnalysis.py /datasets/hsc/repo --rerun=RC/w_2018_28/DM-14988:private/lauren/testing6 --id filter=HSC-G tract=9813 --no-versions -c doWriteParquetTables=False
            

            and matches were successfully found.

            I'm now running the full tract for HSC-G and HSC-I to see what happens.

            Show
            lauren Lauren MacArthur added a comment - So far I am unable to reproduce the failed coaddAnalysis issues.  I tried running the current master (just a trivial commit ahead of where you ran it) on just a handful of patches of HSC-G: coaddAnalysis.py /datasets/hsc/repo --rerun=RC/w_2018_28/DM-14988:private/lauren/testing6 --id filter=HSC-G tract=9813 --no-versions -c doWriteParquetTables=False and matches were successfully found. I'm now running the full tract for HSC-G and HSC-I to see what happens.
            Hide
            lauren Lauren MacArthur added a comment -

            Huh...ok, that one did bomb...the mystery deepens!

            Show
            lauren Lauren MacArthur added a comment - Huh...ok, that one did bomb...the mystery deepens!
            Hide
            lauren Lauren MacArthur added a comment -

            Got it...just a few troublesome patches at the edges needed an extra check for zero good matches after I added some extra culling on flags.  I can confirm with this fix those bands now make it all the way through.  Results can be seen at:
            https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15133/w_2018_28/UD_COSMOS_9813/plots/

            Show
            lauren Lauren MacArthur added a comment - Got it...just a few troublesome patches at the edges needed an extra check for zero good matches after I added some extra culling on flags.  I can confirm with this fix those bands now make it all the way through.  Results can be seen at: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15133/w_2018_28/UD_COSMOS_9813/plots/
            Hide
            lauren Lauren MacArthur added a comment -

            Would you mind giving this a look?  PR is here.

            Show
            lauren Lauren MacArthur added a comment - Would you mind giving this a look?  PR is here .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            LGTM. Thanks for solving the mystery!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - LGTM. Thanks for solving the mystery!
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Merged per conversations with Lauren MacArthur when she's away from the computer.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Merged per conversations with Lauren MacArthur when she's away from the computer.

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel