Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_analysis
-
Labels:None
-
Story Points:3
-
Epic Link:
-
Sprint:DRP S20-6 (May)
-
Team:Data Release Production
-
Urgent?:No
Description
In one edge tract/patch of the HSC-PDR2 DEEP+UDEEP, coaddAnalysis gave this error:
coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/calib/20200115/ --rerun DM-23243/MULTIBAND/DEEP:private/hchiang2/testCoaddAna20200316 --doraise --config doWriteParquetTables=True --id tract=09572 filter=HSC-G
|
root INFO: Running: /home/hchiang2/PDR2/pa_stack_2/pipe_analysis/bin/coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/calib/20200115/ --rerun DM-23243/MULTIBAND/DEEP:private/hchiang2/testCoaddAna20200316 --doraise --config doWriteParquetTables=True --id tract=09572 filter=HSC-G
|
coaddAnalysis INFO: patchList size: 3
|
coaddAnalysis INFO: External calibration(s) used: photoCal: FGCM wcs: JOINTCAL
|
coaddAnalysis INFO: shortName = overlap_modelfit_CModel_forced
|
coaddAnalysis INFO: No data for dataset: unknown
|
coaddAnalysis INFO: Statistics from DataId(initialdata={'tract': 9572, 'filter': 'HSC-G', 'patch': '8,6'}, tag=set()) of Overlap mag difference (CModel) (mmag): {'galaxy': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0), 'split': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0), 'star': Stats(mean=0.0000; stdev=127.7816; num=2; total=2; median=0.0000; clip=378.2334; forcedMean=None; thresholdType=S/N; thresholdValue=300.0), 'unknown': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0)}
|
coaddAnalysis WARN: Overlap mag difference (CModel) (mmag) stdev = 127.78 exceeds maximum limit of 3.00: DataId(initialdata={'tract': 9572, 'filter': 'HSC-G', 'patch': '8,6'}, tag=set())
|
coaddAnalysis INFO: shortName = overlap_distance_forced
|
coaddAnalysis INFO: No data for dataset: unknown
|
coaddAnalysis INFO: Statistics from DataId(initialdata={'tract': 9572, 'filter': 'HSC-G', 'patch': '8,6'}, tag=set()) of Distance (mas): {'galaxy': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0), 'split': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0), 'star': Stats(mean=458.1745; stdev=458.1745; num=2; total=2; median=458.1745; clip=0.0000; forcedMean=0.0; thresholdType=S/N; thresholdValue=300.0), 'unknown': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0)}
|
coaddAnalysis WARN: Distance (mas) stdev = 458.17 exceeds maximum limit of 5.00: DataId(initialdata={'tract': 9572, 'filter': 'HSC-G', 'patch': '8,6'}, tag=set())
|
coaddAnalysis INFO: Number of forced overlap objects matched = 170
|
coaddAnalysis INFO: shortName = overlap_modelfit_CModel_unforced
|
Traceback (most recent call last):
|
File "/home/hchiang2/PDR2/pa_stack_2/pipe_analysis/bin/coaddAnalysis.py", line 3, in <module>
|
CoaddAnalysisTask.parseAndRun()
|
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+2/python/lsst/pipe/base/cmdLineTask.py", line 610, in parseAndRun
|
resultList = taskRunner.run(parsedCmd)
|
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+2/python/lsst/pipe/base/cmdLineTask.py", line 221, in run
|
resultList = list(mapFunc(self, targetList))
|
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+2/python/lsst/pipe/base/cmdLineTask.py", line 385, in __call__
|
result = self.runTask(task, dataRef, kwargs)
|
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+2/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
|
return task.runDataRef(dataRef, **kwargs)
|
File "/home/hchiang2/PDR2/pa_stack_2/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 318, in runDataRef
|
highlightList=highlightList, **plotKwargs)
|
File "/home/hchiang2/PDR2/pa_stack_2/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 1051, in plotOverlaps
|
unitScale=self.unitScale,
|
File "/home/hchiang2/PDR2/pa_stack_2/pipe_analysis/python/lsst/pipe/analysis/analysis.py", line 229, in __init__
|
format(labeller.plot[0]))
|
RuntimeError: No good data points to plot for sample labelled: star
|
Could this edge case be handled more gracefully, such as a softer warning and skipping plotting the patch rather than an error, or other approaches?
The pipe__analysis version is at commit c11be5b with stack w_2020_08.
Attachments
Issue Links
- relates to
-
- Done
Activity
I'm finally getting around to fixing the edge-case bugs you encountered (and so helpfully reported) when doing the HSC-SSP PDR2 run. Would you mind giving this on a look over?
LGTM
Hsin-Fang Chiang
added a comment - LGTM Thanks! I looked into your suggestion about logging names on the PR, but I couldn't find a simple solution, so left it as is. Merged to master.
This is now handled more gracefully by just skipping over the particular plot for which no good data points were found and with a log message along the lines of:
I ran the following:
which now produces 142 plots (many more than the previous count of zero!). See https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-23894/plots/HSC-G/tract-9572/
This also fixes a bug I encountered when no good data points were found for a particular dataset in the sky plots where the function returned without closing the opened matplotlib plt object. This resulted in the contents already added in the plt object to the "next" (supposed to be independent) one, resulting in the overlap mess à la:
vs. the fixed:
(gonna need to do something about those lengthy LATISS filter names...but that's for another ticket!).