Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20472

Empty matches in measureCoaddSources results in unnecessary error

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
    • Team:
      External

      Description

      Hiroyuki Ikeda reports an error when running multiband:

      [73] Traceback (most recent call last):
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py", line 113, in wrapper
      [73]     return func(*args, **kwargs)
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py", line 1070, in run
      [73]     while not menu[command]():
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py", line 240, in wrapper
      [73]     return func(*args, **kwargs)
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py", line 1088, in reduce
      [73]     result = self._processQueue(context, func, [(index, data)], *args, **kwargs)[0]
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py", line 547, in _processQueue
      [73]     return self._reduceQueue(context, None, func, queue, *args, **kwargs)
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py", line 573, in _reduceQueue
      [73]     resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py", line 573, in <listcomp>
      [73]     resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.5-hsc+1/python/lsst/pipe/drivers/multiBandDriver.py", line 428, in runMeasurements
      [73]     self.measureCoaddSources.runDataRef(dataRef)
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.0-hsc+1/python/lsst/pipe/tasks/multiBand.py", line 996, in runDataRef
      [73]     exposureId=self.getExposureId(patchRef))
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.0-hsc+1/python/lsst/pipe/tasks/multiBand.py", line 1073, in run
      [73]     matchResult.matchMeta)
      [73]   File "/ana/products.7.7.2/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/7.0-hsc+1/python/lsst/meas/astrom/denormalizeMatches.py", line 65, in denormalizeMatches
      [73]     raise RuntimeError("No matches provided.")
      [73] RuntimeError: No matches provided.
      [73] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 73
      

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          I'd be surprised and if concerned the schema changes with the data ID, but if it just isn't easily accessible given the code flow just write one with a minimal schema and add a comment.

           

          Show
          jbosch Jim Bosch added a comment - I'd be surprised and if concerned the schema changes with the data ID, but if it just isn't easily accessible given the code flow just write one with a minimal schema and add a comment.  
          Hide
          price Paul Price added a comment -

          afwTable.BaseCatalog(afwTable.Schema()) works.

          Show
          price Paul Price added a comment - afwTable.BaseCatalog(afwTable.Schema()) works.
          Hide
          price Paul Price added a comment -

          Jim Bosch, please let me know if this solution is suitable.

          price@price-laptop:~/LSST/pipe_tasks (tickets/DM-20472=) $ git sub-patch
          commit b8b873864a72598837a307c9d066e5a1e70a633f (HEAD -> tickets/DM-20472, origin/tickets/DM-20472)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Jul 3 13:48:48 2019 -0400
           
              measureCoaddSources: fix error from empty matches
              
              We can't generate a proper denormalised match list from no matches,
              because we don't have the schemas (which come from the matches);
              so lsst.meas.astrom.denormalizeMatches raises RuntimeError if
              there are no matches. We need to catch this case before calling
              denormalizeMatches. We generate a dummy denormalized matches file
              to keep the Gen3 butler happy (it's expecting an output of some
              kind).
           
          diff --git a/python/lsst/pipe/tasks/multiBand.py b/python/lsst/pipe/tasks/multiBand.py
          index 83d5303a..ca1e95a5 100644
          --- a/python/lsst/pipe/tasks/multiBand.py
          +++ b/python/lsst/pipe/tasks/multiBand.py
          @@ -1100,8 +1100,16 @@ class MeasureMergedCoaddSourcesTask(PipelineTask, CmdLineTask):
                       matches.table.setMetadata(matchResult.matchMeta)
                       results.matchResult = matches
                       if self.config.doWriteMatchesDenormalized:
          -                results.denormMatches = denormalizeMatches(matchResult.matches,
          -                                                           matchResult.matchMeta)
          +                if matchResult.matches:
          +                    denormMatches = denormalizeMatches(matchResult.matches, matchResult.matchMeta)
          +                else:
          +                    self.log.warn("No matches, so generating dummy denormalized matches file")
          +                    denormMatches = afwTable.BaseCatalog(afwTable.Schema())
          +                    denormMatches.setMetadata(PropertyList())
          +                    denormMatches.getMetadata().add("COMMENT",
          +                                                    "This catalog is empty because no matches were found.")
          +                    results.denormMatches = denormMatches
          +                results.denormMatches = denormMatches
           
                   results.outputSources = sources
                   return results
          

          Show
          price Paul Price added a comment - Jim Bosch , please let me know if this solution is suitable. price@price-laptop:~/LSST/pipe_tasks (tickets/DM-20472=) $ git sub-patch commit b8b873864a72598837a307c9d066e5a1e70a633f (HEAD -> tickets/DM-20472, origin/tickets/DM-20472) Author: Paul Price <price@astro.princeton.edu> Date: Wed Jul 3 13:48:48 2019 -0400   measureCoaddSources: fix error from empty matches We can't generate a proper denormalised match list from no matches, because we don't have the schemas (which come from the matches); so lsst.meas.astrom.denormalizeMatches raises RuntimeError if there are no matches. We need to catch this case before calling denormalizeMatches. We generate a dummy denormalized matches file to keep the Gen3 butler happy (it's expecting an output of some kind).   diff --git a/python/lsst/pipe/tasks/multiBand.py b/python/lsst/pipe/tasks/multiBand.py index 83d5303a..ca1e95a5 100644 --- a/python/lsst/pipe/tasks/multiBand.py +++ b/python/lsst/pipe/tasks/multiBand.py @@ -1100,8 +1100,16 @@ class MeasureMergedCoaddSourcesTask(PipelineTask, CmdLineTask): matches.table.setMetadata(matchResult.matchMeta) results.matchResult = matches if self.config.doWriteMatchesDenormalized: - results.denormMatches = denormalizeMatches(matchResult.matches, - matchResult.matchMeta) + if matchResult.matches: + denormMatches = denormalizeMatches(matchResult.matches, matchResult.matchMeta) + else: + self.log.warn("No matches, so generating dummy denormalized matches file") + denormMatches = afwTable.BaseCatalog(afwTable.Schema()) + denormMatches.setMetadata(PropertyList()) + denormMatches.getMetadata().add("COMMENT", + "This catalog is empty because no matches were found.") + results.denormMatches = denormMatches + results.denormMatches = denormMatches results.outputSources = sources return results
          Hide
          jbosch Jim Bosch added a comment -

          Fine with me.

           

          Show
          jbosch Jim Bosch added a comment - Fine with me.  
          Hide
          price Paul Price added a comment -

          Thanks Jim.

          Merged to master.

          Show
          price Paul Price added a comment - Thanks Jim. Merged to master.

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Jim Bosch
              Watchers:
              Hiroyuki Ikeda, Hisanori Furusawa, Jim Bosch, John Swinbank, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel