Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17922

Unhelpful "cannot iterate over NoneType" ImageDifferenceTask error

    Details

    • Type: Bug
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: ip_diffim
    • Labels:
      None

      Description

      If you do not specify the exact correct dataId (e.g., visit and ccdnum and filter, for decam), ImageDifferenceTask will fail in a very confusing way. An example traceback thanks to Hayden Smotherman is pasted below.

      Instead, the Task should prompt the user to specify whatever is missing from the dataId, and/or behave more like processCcd where it just figures it out.

      imageDifference INFO: Processing DataId(initialdata=\{'visit': 433932, 'ccdnum': 12}, tag=set())
      imageDifference.getTemplate INFO: Using skyMap tract 0
      imageDifference.getTemplate INFO: Assembling 2 coadd patches
      imageDifference.getTemplate INFO: exposure dimensions=(2046, 4094); coadd dimensions=(3247, 1644)
      imageDifference FATAL: Failed on dataId=DataId(initialdata=\{'visit': 433932, 'ccdnum': 12}, tag=set()): TypeError: argument of type 'NoneType' is not iterable
      Traceback (most recent call last):
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-30-g6787e8a+1/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
       result = self.runTask(task, dataRef, kwargs)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-30-g6787e8a+1/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
       return task.runDataRef(dataRef, **kwargs)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-30-g6787e8a+1/python/lsst/pipe/base/timer.py", line 149, in wrapper
       res = func(self, *args, **keyArgs)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_tasks/16.0-61-gb2b2650a/python/lsst/pipe/tasks/imageDifference.py", line 353, in runDataRef
       template = self.getTemplate.run(exposure, sensorRef, templateIdList=templateIdList)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/ip_diffim/16.0-18-geff7d6b+5/python/lsst/ip/diffim/getTemplate.py", line 134, in run
       if not sensorRef.datasetExists(**patchArgDict):
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-15-g7af1f30/python/lsst/daf/persistence/butlerSubset.py", line 317, in datasetExists
       datasetType, self.dataId, write=write, **rest)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-15-g7af1f30/python/lsst/daf/persistence/butler.py", line 1240, in datasetExists
       locations = self._locate(datasetType, dataId, write=write)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-15-g7af1f30/python/lsst/daf/persistence/butler.py", line 1297, in _locate
       location = repoData.repo.map(datasetType, dataId, write=write)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-15-g7af1f30/python/lsst/daf/persistence/repository.py", line 242, in map
       loc = self._mapper.map(*args, **kwargs)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-15-g7af1f30/python/lsst/daf/persistence/mapper.py", line 166, in map
       return func(self.validate(dataId), write)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_base/16.0-30-g216c9f1/python/lsst/obs/base/cameraMapper.py", line 466, in mapSubClosure
       loc = mapping.map(mapper, subId, write)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_base/16.0-30-g216c9f1/python/lsst/obs/base/mapping.py", line 151, in map
       actualId = self.need(iter(self.keyDict.keys()), dataId)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_base/16.0-30-g216c9f1/python/lsst/obs/base/mapping.py", line 315, in need
       lookups = self.lookup(newProps, newId)
       File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_base/16.0-30-g216c9f1/python/lsst/obs/base/mapping.py", line 241, in lookup
       if fastPath and 'visit' in dataId and "raw" in self.tables:
      TypeError: argument of type 'NoneType' is not iterable
      

       

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Butler in “confusing” shocker!

            More seriously, I agree that this is not good behaviour, but I suggest it's not something we should address separately — all of the data access in ip_diffim will need to be reworked over the next few months for BG3, and which is (...should be) intrinsically less user-hostile anyway. Let's keep this ticket open so we can check that the new version really does do better.

            Show
            swinbank John Swinbank added a comment - Butler in “confusing” shocker! More seriously, I agree that this is not good behaviour, but I suggest it's not something we should address separately — all of the data access in ip_diffim will need to be reworked over the next few months for BG3, and which is (...should be) intrinsically less user-hostile anyway. Let's keep this ticket open so we can check that the new version really does do better.

              People

              • Assignee:
                gkovacs Gabor Kovacs
                Reporter:
                mrawls Meredith Rawls
                Watchers:
                Gabor Kovacs, John Swinbank, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel