[DM-15133] Empty matches in coaddAnalysis.py at COSMOS field? - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: pipe_analysis
Labels:
None

Story Points:
2
Epic Link:
f18-drp-qa
Sprint:
DRP F18-2
Team:
Data Release Production

Description

Using w_2018_28 and pipe_analysis at commit ebd48cf, coaddAnalysis.py failed in the COSMOS tract=9813 filters HSC-G, HSC-R, HSC-Z, and HSC-Y:

Traceback (most recent call last):

  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-2-g852da13+6/python/lsst/pipe/base/cmdLineTask.py", line 392, in __call__

    result = task.run(dataRef, **kwargs)

  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 348, in run

    hscRun=repoInfo.hscRun, wcs=repoInfo.wcs)

  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 450, in readSrcMatches

    if not hasattr(matches[0].first, "schema"):

IndexError: list index out of range

The command I ran was
coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_28/DM-14988 --id tract=9813 filter=HSC-G -c doWriteParquetTables=True
(and 3 other filters)

Attachments

Issue Links

relates to

DM-15119 Run pipe_analysis and validate_drp with w_2018_28 HSC RC2 outputs

Done

Activity

Ascending order - Click to sort in descending order

No builds found.

Hsin-Fang Chiang created issue - 16/Jul/18 3:13 PM

Hsin-Fang Chiang made changes - 16/Jul/18 3:13 PM

Field	Original Value	New Value
Risk Score		0

Hsin-Fang Chiang made changes - 16/Jul/18 3:14 PM

Description

Using {{w_2018_28}} and {{pipe_analysis}} at commit {{ebd48cf}}, {{coaddAnalysis.py}} failed in filters HSC-G, HSC-R, HSC-Z, and HSC-Y:

{code:java}
Traceback (most recent call last):
  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-2-g852da13+6/python/lsst/pipe/base/cmdLineTask.py", line 392, in __call__
    result = task.run(dataRef, **kwargs)
  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 348, in run
    hscRun=repoInfo.hscRun, wcs=repoInfo.wcs)
  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 450, in readSrcMatches
    if not hasattr(matches[0].first, "schema"):
IndexError: list index out of range
{code}

The command I ran was
{{coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_28/~~DM-14988~~ --id tract=9813 filter=HSC-G -c doWriteParquetTables=True}}
(and 3 other filters)

Using {{w_2018_28}} and {{pipe_analysis}} at commit {{ebd48cf}}, {{coaddAnalysis.py}} failed in the COSMOS tract=9813 filters HSC-G, HSC-R, HSC-Z, and HSC-Y:

{code:java}
Traceback (most recent call last):
  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-2-g852da13+6/python/lsst/pipe/base/cmdLineTask.py", line 392, in __call__
    result = task.run(dataRef, **kwargs)
  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 348, in run
    hscRun=repoInfo.hscRun, wcs=repoInfo.wcs)
  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 450, in readSrcMatches
    if not hasattr(matches[0].first, "schema"):
IndexError: list index out of range
{code}

The command I ran was
{{coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_28/~~DM-14988~~ --id tract=9813 filter=HSC-G -c doWriteParquetTables=True}}
(and 3 other filters)

Hsin-Fang Chiang made changes - 16/Jul/18 6:49 PM

Watchers

Hsin-Fang Chiang [ Hsin-Fang Chiang ]

Hsin-Fang Chiang, Lauren MacArthur [ Hsin-Fang Chiang, Lauren MacArthur ]

Hsin-Fang Chiang made changes - 17/Jul/18 12:07 PM

Link

This issue relates to ~~DM-15119~~ [ ~~DM-15119~~ ]

Lauren MacArthur made changes - 17/Jul/18 1:45 PM

Assignee

Lauren MacArthur [ lauren ]

Hide

Permalink

Lauren MacArthur added a comment - 17/Jul/18 1:45 PM

So far I am unable to reproduce the failed coaddAnalysis issues. I tried running the current master (just a trivial commit ahead of where you ran it) on just a handful of patches of HSC-G:

coaddAnalysis.py /datasets/hsc/repo --rerun=RC/w_2018_28/DM-14988:private/lauren/testing6 --id filter=HSC-G tract=9813 --no-versions -c doWriteParquetTables=False

and matches were successfully found.

I'm now running the full tract for HSC-G and HSC-I to see what happens.

Show

Lauren MacArthur added a comment - 17/Jul/18 1:45 PM So far I am unable to reproduce the failed coaddAnalysis issues. I tried running the current master (just a trivial commit ahead of where you ran it) on just a handful of patches of HSC-G: coaddAnalysis.py /datasets/hsc/repo --rerun=RC/w_2018_28/DM-14988:private/lauren/testing6 --id filter=HSC-G tract=9813 --no-versions -c doWriteParquetTables=False and matches were successfully found. I'm now running the full tract for HSC-G and HSC-I to see what happens.

Hide

Permalink

Lauren MacArthur added a comment - 17/Jul/18 2:19 PM

Huh...ok, that one did bomb...the mystery deepens!

Show

Lauren MacArthur added a comment - 17/Jul/18 2:19 PM Huh...ok, that one did bomb...the mystery deepens!

Lauren MacArthur made changes - 17/Jul/18 4:10 PM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Hide

Permalink

Lauren MacArthur added a comment - 17/Jul/18 5:06 PM

Got it...just a few troublesome patches at the edges needed an extra check for zero good matches after I added some extra culling on flags. I can confirm with this fix those bands now make it all the way through. Results can be seen at:
https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15133/w_2018_28/UD_COSMOS_9813/plots/

Show

Lauren MacArthur added a comment - 17/Jul/18 5:06 PM Got it...just a few troublesome patches at the edges needed an extra check for zero good matches after I added some extra culling on flags. I can confirm with this fix those bands now make it all the way through. Results can be seen at: https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-15133/w_2018_28/UD_COSMOS_9813/plots/

Hide

Permalink

Lauren MacArthur added a comment - 17/Jul/18 5:08 PM

Would you mind giving this a look? PR is here.

Show

Lauren MacArthur added a comment - 17/Jul/18 5:08 PM Would you mind giving this a look? PR is here .

Lauren MacArthur made changes - 17/Jul/18 5:08 PM

Reviewers		Hsin-Fang Chiang [ hchiang2 ]
Status	In Progress [ 3 ]	In Review [ 10004 ]

Hide

Permalink

Hsin-Fang Chiang added a comment - 17/Jul/18 6:23 PM

LGTM. Thanks for solving the mystery!

Show

Hsin-Fang Chiang added a comment - 17/Jul/18 6:23 PM LGTM. Thanks for solving the mystery!

Hsin-Fang Chiang made changes - 17/Jul/18 6:23 PM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Hide

Permalink

Hsin-Fang Chiang added a comment - 18/Jul/18 7:58 AM

Merged per conversations with Lauren MacArthur when she's away from the computer.

Show

Hsin-Fang Chiang added a comment - 18/Jul/18 7:58 AM Merged per conversations with Lauren MacArthur when she's away from the computer.

Hsin-Fang Chiang made changes - 18/Jul/18 7:58 AM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

John Swinbank made changes - 23/Jul/18 7:11 AM

Epic Link

~~DM-14402~~ [ 79665 ]

John Swinbank made changes - 23/Jul/18 7:11 AM

Sprint		DRP F18-2 [ 756 ]
Story Points		2
Team		Data Release Production [ 10301 ]

John Swinbank made changes - 23/Jul/18 7:11 AM

Risk Score

People

Assignee:: Lauren MacArthur

Reporter:: Hsin-Fang Chiang

Reviewers:: Hsin-Fang Chiang

Watchers:: Hsin-Fang Chiang, Lauren MacArthur

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 16/Jul/18 3:13 PM

Updated:: 23/Jul/18 7:11 AM

Resolved:: 18/Jul/18 7:58 AM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins