Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15133

Empty matches in coaddAnalysis.py at COSMOS field?

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • pipe_analysis
    • None
    • 2
    • DRP F18-2
    • Data Release Production

    Description

      Using w_2018_28 and pipe_analysis at commit ebd48cf, coaddAnalysis.py failed in the COSMOS tract=9813 filters HSC-G, HSC-R, HSC-Z, and HSC-Y:

      Traceback (most recent call last):
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-2-g852da13+6/python/lsst/pipe/base/cmdLineTask.py", line 392, in __call__
          result = task.run(dataRef, **kwargs)
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 348, in run
          hscRun=repoInfo.hscRun, wcs=repoInfo.wcs)
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 450, in readSrcMatches
          if not hasattr(matches[0].first, "schema"):
      IndexError: list index out of range
      

      The command I ran was
      coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_28/DM-14988 --id tract=9813 filter=HSC-G -c doWriteParquetTables=True
      (and 3 other filters)

      Attachments

        Issue Links

          Activity

            So far I am unable to reproduce the failed coaddAnalysis issues.  I tried running the current master (just a trivial commit ahead of where you ran it) on just a handful of patches of HSC-G:

            coaddAnalysis.py /datasets/hsc/repo --rerun=RC/w_2018_28/DM-14988:private/lauren/testing6 --id filter=HSC-G tract=9813 --no-versions -c doWriteParquetTables=False
            

            and matches were successfully found.

            I'm now running the full tract for HSC-G and HSC-I to see what happens.

            lauren Lauren MacArthur added a comment - So far I am unable to reproduce the failed coaddAnalysis issues.  I tried running the current master (just a trivial commit ahead of where you ran it) on just a handful of patches of HSC-G: coaddAnalysis.py /datasets/hsc/repo --rerun=RC/w_2018_28/DM-14988:private/lauren/testing6 --id filter=HSC-G tract=9813 --no-versions -c doWriteParquetTables=False and matches were successfully found. I'm now running the full tract for HSC-G and HSC-I to see what happens.

            Huh...ok, that one did bomb...the mystery deepens!

            lauren Lauren MacArthur added a comment - Huh...ok, that one did bomb...the mystery deepens!

            Got it...just a few troublesome patches at the edges needed an extra check for zero good matches after I added some extra culling on flags.  I can confirm with this fix those bands now make it all the way through.  Results can be seen at:
            https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15133/w_2018_28/UD_COSMOS_9813/plots/

            lauren Lauren MacArthur added a comment - Got it...just a few troublesome patches at the edges needed an extra check for zero good matches after I added some extra culling on flags.  I can confirm with this fix those bands now make it all the way through.  Results can be seen at: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15133/w_2018_28/UD_COSMOS_9813/plots/

            Would you mind giving this a look?  PR is here.

            lauren Lauren MacArthur added a comment - Would you mind giving this a look?  PR is here .

            LGTM. Thanks for solving the mystery!

            hchiang2 Hsin-Fang Chiang added a comment - LGTM. Thanks for solving the mystery!

            Merged per conversations with lauren when she's away from the computer.

            hchiang2 Hsin-Fang Chiang added a comment - Merged per conversations with lauren when she's away from the computer.

            People

              lauren Lauren MacArthur
              hchiang2 Hsin-Fang Chiang
              Hsin-Fang Chiang
              Hsin-Fang Chiang, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.