Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26274

Image differencing fatal error using DCR templates

    Details

    • Type: Bug
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_diffim
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      AP F20-3 (August)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Image differencing fails with the following error if DCR coadds are used for the template:

      apPipe FATAL: Failed on dataId={'ccdnum': 23, 'visit': 427943, 'date': '2015-04-03', 'filter': 'g', 'hdu': 21, 'object': 'B2'}: TypeError: sequence item 0: expected str instance, NoneType found
      /software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/astropy/table/row.py:76: FutureWarning: elementwise == comparison failed and returning scalar instead; this will raise an error o
      r perform elementwise comparison in the future.
        return self.as_void() == other
      /software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/astropy/units/function/logarithmic.py:46: RuntimeWarning: invalid value encountered in log10
        return dex.to(self._function_unit, np.log10(x))
      Traceback (most recent call last):
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/pipe_base/20.0.0-7-gb92c176+1/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/pipe_base/20.0.0-7-gb92c176+1/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/pipe_base/20.0.0-7-gb92c176+1/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/ap_pipe/20.0.0-2-gd4c8807+22/python/lsst/ap/pipe/ap_pipe.py", line 170, in runDataRef
          diffImResults = self.runDiffIm(calexpRef, templateIds)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/pipe_base/20.0.0-7-gb92c176+1/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/ap_pipe/20.0.0-2-gd4c8807+22/python/lsst/ap/pipe/ap_pipe.py", line 243, in runDiffIm
          return self.differencer.runDataRef(sensorRef, templateIdList=templateIds)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/pipe_base/20.0.0-7-gb92c176+1/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/pipe_tasks/20.0.0-13-g198ee8df/python/lsst/pipe/tasks/imageDifference.py", line 497, in runDataRef
          template = self.getTemplate.runDataRef(exposure, sensorRef, templateIdList=templateIdList)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/ip_diffim/20.0.0-1-g596936a+26/python/lsst/ip/diffim/getTemplate.py", line 114, in runDataRef
          if sensorRef.datasetExists(**patchArgDict):
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_persistence/20.0.0+5/python/lsst/daf/persistence/butlerSubset.py", line 314, in datasetExists
          datasetType, self.dataId, write=write, **rest)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_persistence/20.0.0+5/python/lsst/daf/persistence/butler.py", line 1259, in datasetExists
          locations = self._locate(datasetType, dataId, write=write)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_persistence/20.0.0+5/python/lsst/daf/persistence/butler.py", line 1316, in _locate
          location = repoData.repo.map(datasetType, dataId, write=write)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_persistence/20.0.0+5/python/lsst/daf/persistence/repository.py", line 239, in map
          loc = self._mapper.map(*args, **kwargs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_persistence/20.0.0+5/python/lsst/daf/persistence/mapper.py", line 163, in map
          return func(self.validate(dataId), write)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-24-g0eb5a41+1/python/lsst/obs/base/cameraMapper.py", line 510, in mapSubClosure
          loc = mapping.map(mapper, subId, write)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-24-g0eb5a41+1/python/lsst/obs/base/mapping.py", line 152, in map
          actualId = self.need(iter(self.keyDict.keys()), dataId)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-24-g0eb5a41+1/python/lsst/obs/base/mapping.py", line 316, in need
          lookups = self.lookup(newProps, newId)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-24-g0eb5a41+1/python/lsst/obs/base/mapping.py", line 261, in lookup
          result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_persistence/20.0.0+5/python/lsst/daf/persistence/registries.py", line 350, in lookup
          cmd += " FROM " + " NATURAL JOIN ".join(reference)
      TypeError: sequence item 0: expected str instance, NoneType found
      

      The issue is caused by line 114 of ip_diffim.getTemplate:
      if sensorRef.datasetExists(**patchArgDict):

      Prior to DM-22952 this was:
      if self.config.coaddName != 'dcr' and sensorRef.datasetExists(**patchArgDict):
      but it can't simply be reverted because new code added in DM-22541 depends on the datasetExists test. That test should be fixed for DcrCoadds.

        Attachments

          Activity

          Hide
          sullivan Ian Sullivan added a comment -

          Accidentally duplicated (and fixed) with DM-26567

          Show
          sullivan Ian Sullivan added a comment - Accidentally duplicated (and fixed) with DM-26567

            People

            • Assignee:
              sullivan Ian Sullivan
              Reporter:
              sullivan Ian Sullivan
              Watchers:
              Ian Sullivan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel