# Determine and use correct position for PSF shape determination in ImageDifferenceTask

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
3
• Team:
Data Release Production
• Urgent?:
No

#### Description

As discussed on this thread on Slack, while processing a small subset of the RC2 dataset on DM-30927, six instances (out of 3811 total) of the following error were encountered during imageDifference

  File "/home/lauren/LSST/pipe_tasks/python/lsst/pipe/tasks/imageDifference.py", line 736, in run  templateSigma = templateExposure.getPsf().computeShape().getDeterminantRadius() lsst.pex.exceptions.wrappers.InvalidParameterError:   File "src/CoaddPsf.cc", line 250, in virtual std::shared_ptr > lsst::meas::al gorithms::CoaddPsf::doComputeKernelImage(const Point2D&, const lsst::afw::image::Color&) const  Cannot compute CoaddPsf at point (31923.8, 32218); no input images at that point. {0} lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (31923.8, 32218); no input imag es at that point.' 

This stems from a call to getPsf().computeShape() without specifying a position, so the default and highly dangerous position is used, see RFC-796 (I believe this is happening near the edge of the coadd footprint, so there are many “template” images that will have very few actual good science pixels and whatever is being returned by the default position is an area where there is no real data). It seems we have gotten lucky so far with the full RC2/DC2 runs, but this subset has revealed that this is an issue that warrants fixing.

There are a few offenders in this file:
here and here (and, with the deprecation warning added on DM-31535, a recent processing run will reveal just how many such cases we have with the message:
FutureWarning: Default position argument overload is deprecated and will be removed in version 24.0. Please explicitly specify a position.)

#### Attachments

1. calexp.png
463 kB
2. goodSeeingDiff_templateExp.png
487 kB

#### Activity

Hide

While I was in there I also gave the computeShape call in imageDifferenceTask a position.

I ran a 3828/3829
$OBS_LSST_DIR/pipelines/imsim/DRP.yaml#getTemplate,imageDifference,transformDiaSourceCat,consolidateDiaSourceTable into u/yusra/DC2/DM-31777-bps. • All detectors successfully complete • Looked at the diaSource number counts for w44 rerun and this rerun. And total raw diaSources for 3828/3829 are reduced by 0.06%. (498966[w44] - 498851DM-31777 + 174[new in DM-31777 because previously failed )/ 498966[w44] = 0.000579 Show Yusra AlSayyad added a comment - The fix ended up to be to switch to load in the component coadd CoaddPsfs into template's CoaddPsf with the tractWcs. Previously, I was loading in the post-warped PSFs in the detector-WCS frame, and I now see I was abusing CoaddPsf. Sorry CoaddPsf While I was in there I also gave the computeShape call in imageDifferenceTask a position. Passes Jenkins: https://ci.lsst.codes/job/stack-os-matrix/35370/display/redirect I ran a 3828/3829$OBS_LSST_DIR/pipelines/imsim/DRP.yaml#getTemplate,imageDifference,transformDiaSourceCat,consolidateDiaSourceTable into u/yusra/DC2/ DM-31777 -bps . All detectors successfully complete Looked at the diaSource number counts for w44 rerun and this rerun. And total raw diaSources for 3828/3829 are reduced by 0.06%. (498966 [w44] - 498851 DM-31777 + 174[new in DM-31777 because previously failed )/ 498966 [w44] = 0.000579
Hide

I was planning to create a ticket and fix these, but I could hold off if it was easier to just fix all the instances together on this ticket.

It has nothing to do with RFC-796

Show
Yusra AlSayyad added a comment - - edited I was planning to create a ticket and fix these, but I could hold off if it was easier to just fix all the instances together on this ticket. Please make a new ticket. It has nothing to do with RFC-796
Hide
Jim Bosch added a comment -

Recommended a bit more dead code removal on ip_diffim PR.  Otherwise looks good.

Show
Jim Bosch added a comment - Recommended a bit more dead code removal on ip_diffim PR.  Otherwise looks good.
Hide
Hsin-Fang Chiang added a comment -

This ticket patch  was merged before w_2021_46; is the outcome of step4 failures expected to change?

From the comments of HSC-RC2 w_2021_46 versus w_2021_42 they had same number of failures in imageDifference. And one of the six dataIds, visit=23718 ccd=30, was in w_2021_46's failure list.

Just wanted to double check what's expected.

Show
Hsin-Fang Chiang added a comment - This ticket patch   was merged before w_2021_46; is the outcome of step4 failures expected to change? From the comments of HSC-RC2 w_2021_46 versus w_2021_42 they had same number of failures in imageDifference. And one of the six dataIds, visit=23718 ccd=30, was in w_2021_46's failure list. Just wanted to double check what's expected.
Hide

Nope. Not what I was expecting. Filing a ticket.

Show
Yusra AlSayyad added a comment - Nope. Not what I was expecting. Filing a ticket.

#### People

Assignee:
Reporter:
Lauren MacArthur
Reviewers:
Jim Bosch
Watchers:
Hsin-Fang Chiang, Ian Sullivan, Jim Bosch, Lauren MacArthur, Meredith Rawls, Yusra AlSayyad