Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31777

Determine and use correct position for PSF shape determination in ImageDifferenceTask

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As discussed on this thread on Slack, while processing a small subset of the RC2 dataset on DM-30927, six instances (out of 3811 total) of the following error were encountered during imageDifference

        File "/home/lauren/LSST/pipe_tasks/python/lsst/pipe/tasks/imageDifference.py", line 736, in run
          templateSigma = templateExposure.getPsf().computeShape().getDeterminantRadius()
      lsst.pex.exceptions.wrappers.InvalidParameterError: 
        File "src/CoaddPsf.cc", line 250, in virtual std::shared_ptr<lsst::afw::image::Image<double> > lsst::meas::al
      gorithms::CoaddPsf::doComputeKernelImage(const Point2D&, const lsst::afw::image::Color&) const
          Cannot compute CoaddPsf at point (31923.8, 32218); no input images at that point. {0}
      lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (31923.8, 32218); no input imag
      es at that point.'
      

      This stems from a call to getPsf().computeShape() without specifying a position, so the default and highly dangerous position is used, see RFC-796 (I believe this is happening near the edge of the coadd footprint, so there are many “template” images that will have very few actual good science pixels and whatever is being returned by the default position is an area where there is no real data). It seems we have gotten lucky so far with the full RC2/DC2 runs, but this subset has revealed that this is an issue that warrants fixing.

      There are a few offenders in this file:
      here and here (and, with the deprecation warning added on DM-31535, a recent processing run will reveal just how many such cases we have with the message:
      FutureWarning: Default position argument overload is deprecated and will be removed in version 24.0. Please explicitly specify a position.)

        Attachments

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment -

            The fix ended up to be to switch to load in the component coadd CoaddPsfs into template's CoaddPsf with the tractWcs. Previously, I was loading in the post-warped PSFs in the detector-WCS frame, and I now see I was abusing CoaddPsf. Sorry CoaddPsf

            While I was in there I also gave the computeShape call in imageDifferenceTask a position.

            Passes Jenkins: https://ci.lsst.codes/job/stack-os-matrix/35370/display/redirect

            I ran a 3828/3829
            $OBS_LSST_DIR/pipelines/imsim/DRP.yaml#getTemplate,imageDifference,transformDiaSourceCat,consolidateDiaSourceTable into u/yusra/DC2/DM-31777-bps.

            • All detectors successfully complete
            • Looked at the diaSource number counts for w44 rerun and this rerun. And total raw diaSources for 3828/3829 are reduced by 0.06%.
              (498966[w44] - 498851DM-31777 + 174[new in DM-31777 because previously failed )/ 498966[w44] = 0.000579
            Show
            yusra Yusra AlSayyad added a comment - The fix ended up to be to switch to load in the component coadd CoaddPsfs into template's CoaddPsf with the tractWcs. Previously, I was loading in the post-warped PSFs in the detector-WCS frame, and I now see I was abusing CoaddPsf. Sorry CoaddPsf While I was in there I also gave the computeShape call in imageDifferenceTask a position. Passes Jenkins: https://ci.lsst.codes/job/stack-os-matrix/35370/display/redirect I ran a 3828/3829 $OBS_LSST_DIR/pipelines/imsim/DRP.yaml#getTemplate,imageDifference,transformDiaSourceCat,consolidateDiaSourceTable into u/yusra/DC2/ DM-31777 -bps . All detectors successfully complete Looked at the diaSource number counts for w44 rerun and this rerun. And total raw diaSources for 3828/3829 are reduced by 0.06%. (498966 [w44] - 498851 DM-31777 + 174[new in DM-31777 because previously failed )/ 498966 [w44] = 0.000579
            Hide
            yusra Yusra AlSayyad added a comment - - edited

            I was planning to create a ticket and fix these, but I could hold off if it was easier to just fix all the instances together on this ticket.

            Please make a new ticket.

            It has nothing to do with RFC-796

            Show
            yusra Yusra AlSayyad added a comment - - edited I was planning to create a ticket and fix these, but I could hold off if it was easier to just fix all the instances together on this ticket. Please make a new ticket. It has nothing to do with RFC-796
            Hide
            jbosch Jim Bosch added a comment -

            Recommended a bit more dead code removal on ip_diffim PR.  Otherwise looks good.

            Show
            jbosch Jim Bosch added a comment - Recommended a bit more dead code removal on ip_diffim PR.  Otherwise looks good.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            This ticket patch  was merged before w_2021_46; is the outcome of step4 failures expected to change?

            From the comments of HSC-RC2 w_2021_46 versus w_2021_42 they had same number of failures in imageDifference. And one of the six dataIds, visit=23718 ccd=30, was in w_2021_46's failure list.

            Just wanted to double check what's expected. 

            Show
            hchiang2 Hsin-Fang Chiang added a comment - This ticket patch   was merged before w_2021_46; is the outcome of step4 failures expected to change? From the comments of HSC-RC2 w_2021_46 versus w_2021_42 they had same number of failures in imageDifference. And one of the six dataIds, visit=23718 ccd=30, was in w_2021_46's failure list. Just wanted to double check what's expected. 
            Hide
            yusra Yusra AlSayyad added a comment -

            Nope. Not what I was expecting. Filing a ticket.

            Show
            yusra Yusra AlSayyad added a comment - Nope. Not what I was expecting. Filing a ticket.

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Jim Bosch
              Watchers:
              Hsin-Fang Chiang, Ian Sullivan, Jim Bosch, Lauren MacArthur, Meredith Rawls, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.