Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15133

Empty matches in coaddAnalysis.py at COSMOS field?

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      2
    • Epic Link:
    • Sprint:
      DRP F18-2
    • Team:
      Data Release Production

      Description

      Using w_2018_28 and pipe_analysis at commit ebd48cf, coaddAnalysis.py failed in the COSMOS tract=9813 filters HSC-G, HSC-R, HSC-Z, and HSC-Y: 

      Traceback (most recent call last):
      		 
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-2-g852da13+6/python/lsst/pipe/base/cmdLineTask.py", line 392, in __call__
      		 
          result = task.run(dataRef, **kwargs)
      		 
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 348, in run
      		 
          hscRun=repoInfo.hscRun, wcs=repoInfo.wcs)
      		 
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 450, in readSrcMatches
      		 
          if not hasattr(matches[0].first, "schema"):
      		 
      IndexError: list index out of range

      The command I ran was
      coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_28/DM-14988 --id tract=9813 filter=HSC-G -c doWriteParquetTables=True
      (and 3 other filters)

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15119 Run pipe_analysis and validate_drp with w_2018_28 HSC RC2 outputs

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            So far I am unable to reproduce the failed coaddAnalysis issues.  I tried running the current master (just a trivial commit ahead of where you ran it) on just a handful of patches of HSC-G:

            coaddAnalysis.py /datasets/hsc/repo --rerun=RC/w_2018_28/DM-14988:private/lauren/testing6 --id filter=HSC-G tract=9813 --no-versions -c doWriteParquetTables=False

            and matches were successfully found.

            I'm now running the full tract for HSC-G and HSC-I to see what happens.

            Show
            lauren Lauren MacArthur added a comment - So far I am unable to reproduce the failed coaddAnalysis issues.  I tried running the current master (just a trivial commit ahead of where you ran it) on just a handful of patches of HSC-G: coaddAnalysis.py /datasets/hsc/repo --rerun=RC/w_2018_28/DM-14988:private/lauren/testing6 --id filter=HSC-G tract=9813 --no-versions -c doWriteParquetTables=False and matches were successfully found. I'm now running the full tract for HSC-G and HSC-I to see what happens.
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Huh...ok, that one did bomb...the mystery deepens!

            Show
            lauren Lauren MacArthur added a comment - Huh...ok, that one did bomb...the mystery deepens!
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Got it...just a few troublesome patches at the edges needed an extra check for zero good matches after I added some extra culling on flags.  I can confirm with this fix those bands now make it all the way through.  Results can be seen at:
            https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15133/w_2018_28/UD_COSMOS_9813/plots/

            Show
            lauren Lauren MacArthur added a comment - Got it...just a few troublesome patches at the edges needed an extra check for zero good matches after I added some extra culling on flags.  I can confirm with this fix those bands now make it all the way through.  Results can be seen at: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15133/w_2018_28/UD_COSMOS_9813/plots/
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Would you mind giving this a look?  PR is here.

            Show
            lauren Lauren MacArthur added a comment - Would you mind giving this a look?  PR is here .
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            LGTM. Thanks for solving the mystery!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - LGTM. Thanks for solving the mystery!
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Merged per conversations with Lauren MacArthur when she's away from the computer.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Merged per conversations with Lauren MacArthur when she's away from the computer.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.