Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33596

"NoneType object is not subscriptable" in PsfWcsSelectImagesTask

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      During DP0.2 step3 we had a visit (1164075) that produced tracebacks in makeWarp like:

       Traceback (most recent call last):

      File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.0/Linux64/ctrl_mpexec/g6727979600+15d2600a0d/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 363, in _executeQuantaInProcess
      self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)

      File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.0/Linux64/ctrl_mpexec/g6727979600+15d2600a0d/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 174, in execute
      self.runQuantum(task, quantum, taskDef, butler)

      File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.0/Linux64/ctrl_mpexec/g6727979600+15d2600a0d/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 464, in runQuantum
      task.runQuantum(butlerQC, inputRefs, outputRefs)

      File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.0/Linux64/pipe_tasks/g50db10430f+6048f86b6d/python/lsst/pipe/tasks/makeCoaddTempExp.py", line 771, in runQuantum
      goodIndices = self.select.run(**inputs, coordList=coordList, dataIds=dataIdList)

      File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.0/Linux64/pipe_tasks/g50db10430f+6048f86b6d/python/lsst/pipe/tasks/selectImages.py", line 411, in run
      if self.isValid(visitSummary, dataId["detector"]):

      File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.0/Linux64/pipe_tasks/g50db10430f+6048f86b6d/python/lsst/pipe/tasks/selectImages.py", line 442, in isValid
      row["visit"], detectorId)

      TypeError: 'NoneType' object is not subscriptable

      This causes makeWarp to error out, when it should be able to continue by ignoring the bad visit.

        Attachments

          Activity

          Hide
          yusra Yusra AlSayyad added a comment -

          Can you add an example tract and patch so that I can reproduce?

          Show
          yusra Yusra AlSayyad added a comment - Can you add an example tract and patch so that I can reproduce?
          Hide
          ctslater Colin Slater added a comment -

          Full data ID is

          {instrument: 'LSSTCam-imSim', skymap: 'DC2', tract: 2897, patch: 39, visit: 1164075, ...}

          , collection "2.2i/runs/DP0.2/v23_0_0_rc5/PREOPS-905" on data-int.

          Show
          ctslater Colin Slater added a comment - Full data ID is {instrument: 'LSSTCam-imSim', skymap: 'DC2', tract: 2897, patch: 39, visit: 1164075, ...} , collection "2.2i/runs/DP0.2/v23_0_0_rc5/PREOPS-905" on data-int.
          Hide
          yusra Yusra AlSayyad added a comment -

          Plot twist.

           pipetask run -b /repo/dc2 -i 2.2i/defaults  -o u/yusra/DC2/DM-33596 -d "visit=1164075 AND patch=39 AND tract=2897 and skymap='DC2'"  -p $OBS_LSST_DIR/pipelines/imsim/DRP.yaml#isr,characterizeImage,calibrate,writeSourceTable,transformSourceTable,consolidateSourceTable,consolidateVisitSummary,makeWarp -j 8 

          succeeds with w_2022_06. Will try again with v23.0.1.rc2, and if that still works, I'll try on data-int.

          Show
          yusra Yusra AlSayyad added a comment - Plot twist. pipetask run -b /repo/dc2 -i 2.2i/defaults -o u/yusra/DC2/DM-33596 -d "visit=1164075 AND patch=39 AND tract=2897 and skymap='DC2'" -p $OBS_LSST_DIR/pipelines/imsim/DRP.yaml#isr,characterizeImage,calibrate,writeSourceTable,transformSourceTable,consolidateSourceTable,consolidateVisitSummary,makeWarp -j 8 succeeds with w_2022_06. Will try again with v23.0.1.rc2, and if that still works, I'll try on data-int.
          Hide
          yusra Yusra AlSayyad added a comment -

          Above pipetask succeeds on lsst-devl with v23_0_1_rc2.
          Over on data-int. All the calexps are there, all the calexpBackrounds are there, the visitSummary is there.

          I could reproduce only by running pipetask on data-int which had the visitSummary table missing detector 140. Easy fix. PS: I'm using this as my example on why pipelines needs to be able to pipetask run on the intermediate data products. PR in a few minutes.

          Show
          yusra Yusra AlSayyad added a comment - Above pipetask succeeds on lsst-devl with v23_0_1_rc2. Over on data-int. All the calexps are there, all the calexpBackrounds are there, the visitSummary is there. I could reproduce only by running pipetask on data-int which had the visitSummary table missing detector 140. Easy fix. PS: I'm using this as my example on why pipelines needs to be able to pipetask run on the intermediate data products. PR in a few minutes.
          Hide
          yusra Yusra AlSayyad added a comment -

          Eli, do you have a sec to review?
          https://github.com/lsst/pipe_tasks/pull/631
          The warning in your PsfWcsSelectImagesTask backport raises its own exception.

          I looked into it with intention to triage and add a how-to-reproduce to the ticket, but the change was so simple I just made the PR. We are considering building a v23_0_1_rc3 just for this, so faster the better.

          Show
          yusra Yusra AlSayyad added a comment - Eli, do you have a sec to review? https://github.com/lsst/pipe_tasks/pull/631 The warning in your PsfWcsSelectImagesTask backport raises its own exception. I looked into it with intention to triage and add a how-to-reproduce to the ticket, but the change was so simple I just made the PR. We are considering building a v23_0_1_rc3 just for this, so faster the better.
          Hide
          yusra Yusra AlSayyad added a comment -

          Log now looks like:

          lsst.makeWarp.select INFO: Selecting calexp {instrument: 'LSSTCam-imSim', detector: 139, visit: 1164075, ...}
          lsst.makeWarp.select INFO: Selecting calexp {instrument: 'LSSTCam-imSim', detector: 140, visit: 1164075, ...}
          lsst.makeWarp.select INFO: Selecting calexp {instrument: 'LSSTCam-imSim', detector: 142, visit: 1164075, ...}
          lsst.makeWarp.select INFO: Selecting calexp {instrument: 'LSSTCam-imSim', detector: 143, visit: 1164075, ...}
          lsst.makeWarp.select WARNING: Removing detector 140 because summary stats not available.
          lsst.makeWarp INFO: Processing calexp 1 of 3 for this Warp: id={instrument: 'LSSTCam-imSim', detector: 139, visit: 1164075, ...}
          ...
          

          No one will miss the "visit"

          Show
          yusra Yusra AlSayyad added a comment - Log now looks like: lsst.makeWarp.select INFO: Selecting calexp {instrument: 'LSSTCam-imSim', detector: 139, visit: 1164075, ...} lsst.makeWarp.select INFO: Selecting calexp {instrument: 'LSSTCam-imSim', detector: 140, visit: 1164075, ...} lsst.makeWarp.select INFO: Selecting calexp {instrument: 'LSSTCam-imSim', detector: 142, visit: 1164075, ...} lsst.makeWarp.select INFO: Selecting calexp {instrument: 'LSSTCam-imSim', detector: 143, visit: 1164075, ...} lsst.makeWarp.select WARNING: Removing detector 140 because summary stats not available. lsst.makeWarp INFO: Processing calexp 1 of 3 for this Warp: id={instrument: 'LSSTCam-imSim', detector: 139, visit: 1164075, ...} ... No one will miss the "visit"
          Hide
          yusra Yusra AlSayyad added a comment -

          Colin Slater Can you find someone on the execution side to look into how the `calibrate_log` could look normal, have succeeded making a calexp and src for (‘visit’: 1164075, ‘detector’:140), but `writeSourceTable_log` says it had nothing to do (and therefore there is no `source` table for it)? I don't understand how retries or the rescue jobs work.
          I don't see NoWorkFound in `calibrate_log`.
          The timestamp of the end of calibrate_log is suspiciously after the writeSourceTable_log.

          Show
          yusra Yusra AlSayyad added a comment - Colin Slater Can you find someone on the execution side to look into how the `calibrate_log` could look normal, have succeeded making a calexp and src for (‘visit’: 1164075, ‘detector’:140), but `writeSourceTable_log` says it had nothing to do (and therefore there is no `source` table for it)? I don't understand how retries or the rescue jobs work. I don't see NoWorkFound in `calibrate_log`. The timestamp of the end of calibrate_log is suspiciously after the writeSourceTable_log.
          Hide
          ctslater Colin Slater added a comment -

          Filed PREOPS-1037 to investigate if these missing detectors are signs of a bigger problem.

          Show
          ctslater Colin Slater added a comment - Filed PREOPS-1037 to investigate if these missing detectors are signs of a bigger problem.

            People

            Assignee:
            yusra Yusra AlSayyad
            Reporter:
            ctslater Colin Slater
            Reviewers:
            Eli Rykoff
            Watchers:
            Colin Slater, Eli Rykoff, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.