Only do minimal necessary processing in characterizeImage by default

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
8
• Sprint:
AP F22-2 (July)
• Team:
• Urgent?:
No

Description

This ticket is part of the implementation of RFC-857, reducing the amount of extra processing that is happening in a default configuration of characterizeImage.

1. Restrict plugins used in characterize to only a minimal list that is either required or is our recommended algorithm (e.g. shape HSM over SdssShape).
2. Default measurePsf to psfex instead of piff, and ensure piff is used where it is needed (possibly just FinalizeCharacterizeTask, which already defaults to piff).
3. Default to psfIterations=1.
4. Cleanup ap_pipe's characterizeImage plugin list/configs to reflect that the default plugins are now appropriate for AP (this should remove the additions that were made on DM-34826).

Activity

Hide
Jim Bosch added a comment - - edited

Definitely not (3).

I think for DRP purposes we should switch any analysis of PSF properties to the finalize measurements, but I can imagine ways in which that's nontrivial and best punted. So let's do (2), especially since I think my past self may be wrong in thinking that AP won't want plots/metrics based on that slot, and it should be negligible in terms of computational performance.

Show
Jim Bosch added a comment - - edited Definitely not (3). I think for DRP purposes we should switch any analysis of PSF properties to the finalize measurements, but I can imagine ways in which that's nontrivial and best punted. So let's do (2), especially since I think my past self may be wrong in thinking that AP won't want plots/metrics based on that slot, and it should be negligible in terms of computational performance.
Hide
John Parejko added a comment -

Jim Bosch: this should be a short review, but I'm not sure I've considered all the DRP implications.

pex_config is failing the GitHub action checks, but I can't see why, and drp_pipe is failing on lines I didn't change; ideas on what's going on with those?

PRs:

Show
John Parejko added a comment - Jim Bosch : this should be a short review, but I'm not sure I've considered all the DRP implications. pex_config is failing the GitHub action checks, but I can't see why, and drp_pipe is failing on lines I didn't change; ideas on what's going on with those? PRs: https://github.com/lsst/pipe_tasks/pull/701 https://github.com/lsst/drp_pipe/pull/32 https://github.com/lsst/pex_config/pull/84 https://github.com/lsst/ap_pipe/pull/118
Hide
Jim Bosch added a comment -

Looks good!

I've commented about both GitHub Actions issues on the PRs.

Were the DM-35538 problems the only exceptions you saw in the new logging in pipe_tasks when running ci_hsc or ci_imsim? If so, great; if there were other exceptions that looked like they should be caught and silently squashed, we should be a little more nuanced on the change here before merging it.

Show
Jim Bosch added a comment - Looks good! I've commented about both GitHub Actions issues on the PRs. Were the DM-35538 problems the only exceptions you saw in the new logging in pipe_tasks when running ci_hsc or ci_imsim? If so, great; if there were other exceptions that looked like they should be caught and silently squashed, we should be a little more nuanced on the change here before merging it.
Hide
John Parejko added a comment -
Show
John Parejko added a comment - Post-rebase Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36965/pipeline
Hide
John Parejko added a comment -

Re: DM-35538, there's a bunch of errors related to what Yusra AlSayyad told me were a known problem in transformForcedSourceOnDiaObjectTable (apparently the DIA sources or objects have a bunch of known failures right now and someone is planning to fix it?). I don't know if there's a ticket for that, and I don't recall any of the details. That's the only other thing I saw though (although it did send me down a bit of rabbit hole while debugging my own problem); it's going to be hard to filter that out, since I think it is an actual error (it's a KeyError on most of the fields).

Show
John Parejko added a comment - Re: DM-35538 , there's a bunch of errors related to what Yusra AlSayyad told me were a known problem in transformForcedSourceOnDiaObjectTable (apparently the DIA sources or objects have a bunch of known failures right now and someone is planning to fix it?). I don't know if there's a ticket for that, and I don't recall any of the details. That's the only other thing I saw though (although it did send me down a bit of rabbit hole while debugging my own problem); it's going to be hard to filter that out, since I think it is an actual error (it's a KeyError on most of the fields).

People

Assignee:
John Parejko
Reporter:
John Parejko
Reviewers:
Jim Bosch
Watchers:
Eli Rykoff, Eric Bellm, Jim Bosch, John Parejko, Yusra AlSayyad