Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23894

coaddAnalysis: RuntimeError: No good data points to plot for sample labelled: star

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • pipe_analysis
    • None
    • 3
    • DRP S20-6 (May)
    • Data Release Production
    • No

    Description

      In one edge tract/patch of the HSC-PDR2 DEEP+UDEEP, coaddAnalysis gave this error:

      coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/calib/20200115/ --rerun DM-23243/MULTIBAND/DEEP:private/hchiang2/testCoaddAna20200316 --doraise --config doWriteParquetTables=True --id tract=09572 filter=HSC-G
      

      root INFO: Running: /home/hchiang2/PDR2/pa_stack_2/pipe_analysis/bin/coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/calib/20200115/ --rerun DM-23243/MULTIBAND/DEEP:private/hchiang2/testCoaddAna20200316 --doraise --config doWriteParquetTables=True --id tract=09572 filter=HSC-G
      coaddAnalysis INFO: patchList size: 3
      coaddAnalysis INFO: External calibration(s) used: photoCal: FGCM  wcs: JOINTCAL
      coaddAnalysis INFO: shortName = overlap_modelfit_CModel_forced
      coaddAnalysis INFO: No data for dataset: unknown
      coaddAnalysis INFO: Statistics from DataId(initialdata={'tract': 9572, 'filter': 'HSC-G', 'patch': '8,6'}, tag=set()) of   Overlap mag difference (CModel) (mmag): {'galaxy': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0), 'split': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0), 'star': Stats(mean=0.0000; stdev=127.7816; num=2; total=2; median=0.0000; clip=378.2334; forcedMean=None; thresholdType=S/N; thresholdValue=300.0), 'unknown': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0)}
      coaddAnalysis WARN:   Overlap mag difference (CModel) (mmag) stdev = 127.78 exceeds maximum limit of 3.00: DataId(initialdata={'tract': 9572, 'filter': 'HSC-G', 'patch': '8,6'}, tag=set())
      coaddAnalysis INFO: shortName = overlap_distance_forced
      coaddAnalysis INFO: No data for dataset: unknown
      coaddAnalysis INFO: Statistics from DataId(initialdata={'tract': 9572, 'filter': 'HSC-G', 'patch': '8,6'}, tag=set()) of Distance (mas): {'galaxy': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0), 'split': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0), 'star': Stats(mean=458.1745; stdev=458.1745; num=2; total=2; median=458.1745; clip=0.0000; forcedMean=0.0; thresholdType=S/N; thresholdValue=300.0), 'unknown': Stats(mean=nan; stdev=nan; num=0; total=0; median=nan; clip=nan; forcedMean=nan; thresholdType=S/N; thresholdValue=300.0)}
      coaddAnalysis WARN: Distance (mas) stdev = 458.17 exceeds maximum limit of 5.00: DataId(initialdata={'tract': 9572, 'filter': 'HSC-G', 'patch': '8,6'}, tag=set())
      coaddAnalysis INFO: Number of forced overlap objects matched = 170
      coaddAnalysis INFO: shortName = overlap_modelfit_CModel_unforced
      Traceback (most recent call last):
        File "/home/hchiang2/PDR2/pa_stack_2/pipe_analysis/bin/coaddAnalysis.py", line 3, in <module>
          CoaddAnalysisTask.parseAndRun()
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+2/python/lsst/pipe/base/cmdLineTask.py", line 610, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+2/python/lsst/pipe/base/cmdLineTask.py", line 221, in run
          resultList = list(mapFunc(self, targetList))
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+2/python/lsst/pipe/base/cmdLineTask.py", line 385, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+2/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/home/hchiang2/PDR2/pa_stack_2/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 318, in runDataRef
          highlightList=highlightList, **plotKwargs)
        File "/home/hchiang2/PDR2/pa_stack_2/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 1051, in plotOverlaps
          unitScale=self.unitScale,
        File "/home/hchiang2/PDR2/pa_stack_2/pipe_analysis/python/lsst/pipe/analysis/analysis.py", line 229, in __init__
          format(labeller.plot[0]))
      RuntimeError: No good data points to plot for sample labelled: star
      

      Could this edge case be handled more gracefully, such as a softer warning and skipping plotting the patch rather than an error, or other approaches?

      The pipe__analysis version is at commit c11be5b with stack w_2020_08.

      Attachments

        Issue Links

          Activity

            lauren Lauren MacArthur added a comment - - edited

            This is now handled more gracefully by just skipping over the particular plot for which no good data points were found and with a log message along the lines of:

            coaddAnalysis WARN: No good data points to plot for: overlap_modelfit_CModel_forced galaxy.  Skipping sky-gals plot.
            

            I ran the following:

            coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/calib/20200115/ --rerun DM-23243/MULTIBAND/DEEP:private/lauren/DM-23894 --doraise --config doWriteParquetTables=False --id tract=09572 filter=HSC-G
            

            which now produces 142 plots (many more than the previous count of zero!). See https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-23894/plots/HSC-G/tract-9572/

            This also fixes a bug I encountered when no good data points were found for a particular dataset in the sky plots where the function returned without closing the opened matplotlib plt object. This resulted in the contents already added in the plt object to the "next" (supposed to be independent) one, resulting in the overlap mess à la:

            vs. the fixed:


            (gonna need to do something about those lengthy LATISS filter names...but that's for another ticket!).

            lauren Lauren MacArthur added a comment - - edited This is now handled more gracefully by just skipping over the particular plot for which no good data points were found and with a log message along the lines of: coaddAnalysis WARN: No good data points to plot for : overlap_modelfit_CModel_forced galaxy. Skipping sky - gals plot. I ran the following: coaddAnalysis.py / datasets / hsc / repo / - - calib / datasets / hsc / calib / 20200115 / - - rerun DM - 23243 / MULTIBAND / DEEP:private / lauren / DM - 23894 - - doraise - - config doWriteParquetTables = False - - id tract = 09572 filter = HSC - G which now produces 142 plots (many more than the previous count of zero!). See https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-23894/plots/HSC-G/tract-9572/ This also fixes a bug I encountered when no good data points were found for a particular dataset in the sky plots where the function returned without closing the opened matplotlib plt object. This resulted in the contents already added in the plt object to the "next" (supposed to be independent) one, resulting in the overlap mess à la: vs. the fixed: (gonna need to do something about those lengthy LATISS filter names...but that's for another ticket!).

            I'm finally getting around to fixing the edge-case bugs you encountered (and so helpfully reported) when doing the HSC-SSP PDR2 run.  Would you mind giving this on a look over?

            lauren Lauren MacArthur added a comment - I'm finally getting around to fixing the edge-case bugs you encountered (and so helpfully reported) when doing the HSC-SSP PDR2 run.  Would you mind giving this on a look over?

            LGTM

            hchiang2 Hsin-Fang Chiang added a comment - LGTM

            Thanks! I looked into your suggestion about logging names on the PR, but I couldn't find a simple solution, so left it as is. Merged to master.

            lauren Lauren MacArthur added a comment - Thanks! I looked into your suggestion about logging names on the PR, but I couldn't find a simple solution, so left it as is. Merged to master.

            People

              lauren Lauren MacArthur
              hchiang2 Hsin-Fang Chiang
              Hsin-Fang Chiang
              Hsin-Fang Chiang, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.