Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29581

runISR.py can't find defect files

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      When trying to run ISR on recent ComCam files, the ISR task can't find the dfect files.  I've had this problem before, but I thought it was fixed.  The files are there in the CALIB directory.  I know the problem is the defect files, because if I change doDefect=True to doDefect=False, then the ISR runs and completes. 

      As I've said in the Slack messages, if the best fix is to move to Gen3, I'm open to doing this, but will need smoe guidance.

      Here's the command line:

      runIsr.py /project/shared/comCam --rerun /project/shared/comCam/rerun/cslage/PTC_2021-04-02 --calib /project/shared/comCam/rerun/cslage/PTC_2021-04-02/CALIB --id detector=4 expId=2021040200025 -c isr.doBias=True isr.doDark=True isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False     isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1

      and here's the error:

      runIsr FATAL: Failed on dataId={'detector': 4, 'expId': 2021040200025, 'dayObs': '2021-04-02', 'raftName': 'R22', 'detectorName': 'S11'}: RuntimeError: No registry for lookup
      Traceback (most recent call last):
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-16-g0fb55c1+b60e2d390c/python/lsst/pipe/base/cmdLineTask.py", line 431, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-16-g0fb55c1+b60e2d390c/python/lsst/pipe/base/cmdLineTask.py", line 495, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-11-g2b59f77+a9c1acf22d/python/lsst/ip/isr/isrTask.py", line 2657, in runDataRef
          return self.isr.runDataRef(dataRef)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-16-g0fb55c1+b60e2d390c/python/lsst/pipe/base/timer.py", line 157, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-11-g2b59f77+a9c1acf22d/python/lsst/ip/isr/isrTask.py", line 1719, in runDataRef
          isrData = self.readIsrData(sensorRef, ccdExposure)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-11-g2b59f77+a9c1acf22d/python/lsst/ip/isr/isrTask.py", line 1144, in readIsrData
          defectList = (dataRef.get("defects")
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/daf_persistence/21.0.0-8-g5674e7b+d1bd76f71f/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
          return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/daf_persistence/21.0.0-8-g5674e7b+d1bd76f71f/python/lsst/daf/persistence/butler.py", line 1399, in get
          location = self._locate(datasetType, dataId, write=False)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/daf_persistence/21.0.0-8-g5674e7b+d1bd76f71f/python/lsst/daf/persistence/butler.py", line 1318, in _locate
          location = repoData.repo.map(datasetType, dataId, write=write)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/daf_persistence/21.0.0-8-g5674e7b+d1bd76f71f/python/lsst/daf/persistence/repository.py", line 239, in map
          loc = self._mapper.map(*args, **kwargs)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/daf_persistence/21.0.0-8-g5674e7b+d1bd76f71f/python/lsst/daf/persistence/mapper.py", line 165, in map
          return func(self.validate(dataId), write)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/obs_base/21.0.0-46-g873c13b+6225f1ba97/python/lsst/obs/base/cameraMapper.py", line 392, in mapClosure
          return mapping.map(mapper, dataId, write)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/obs_base/21.0.0-46-g873c13b+6225f1ba97/python/lsst/obs/base/mapping.py", line 460, in map
          location = Mapping.map(self, mapper, dataId, write=write)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/obs_base/21.0.0-46-g873c13b+6225f1ba97/python/lsst/obs/base/mapping.py", line 155, in map
          actualId = self.need(iter(self.keyDict.keys()), dataId)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/obs_base/21.0.0-46-g873c13b+6225f1ba97/python/lsst/obs/base/mapping.py", line 322, in need
          lookups = self.lookup(newProps, newId)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/obs_base/21.0.0-46-g873c13b+6225f1ba97/python/lsst/obs/base/mapping.py", line 519, in lookup
          return Mapping.lookup(self, properties, newId)
        File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/obs_base/21.0.0-46-g873c13b+6225f1ba97/python/lsst/obs/base/mapping.py", line 209, in lookup
          raise RuntimeError("No registry for lookup")
      RuntimeError: No registry for lookup
      

        Attachments

          Issue Links

            Activity

            Hide
            czw Christopher Waters added a comment -

            The previous ticket (DM-28346) got lost.  I've updated it and will put it into review (once I check the changes over to see if they still make sense), but moving to gen3 is the best option for the long term.  DM-27922 has a list of example cpp commands.

            Show
            czw Christopher Waters added a comment - The previous ticket ( DM-28346 ) got lost.  I've updated it and will put it into review (once I check the changes over to see if they still make sense), but moving to gen3 is the best option for the long term.  DM-27922  has a list of example cpp commands.
            Hide
            cslage Craig Lage added a comment -

            Since I successfully ran the PTC with Gen3 (with the defect files included), this is probably of less importance.  It's OK with me if we kill this or call it done.

            Show
            cslage Craig Lage added a comment - Since I successfully ran the PTC with Gen3 (with the defect files included), this is probably of less importance.  It's OK with me if we kill this or call it done.
            Hide
            czw Christopher Waters added a comment -

            I will mark it invalid, and try to get DM-28346 through review in case a gen2 solution is needed in the future.

            Show
            czw Christopher Waters added a comment - I will mark it invalid, and try to get DM-28346 through review in case a gen2 solution is needed in the future.

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              cslage Craig Lage
              Watchers:
              Christopher Waters, Craig Lage
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: