# Identify unnecessary configs and Tasks and test removal from ap_pipe

XMLWordPrintable

#### Details

• Type: Technical task
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Sprint:
AP S22-6 (May)
• Team:
Alert Production

#### Description

Likely in pair-coding, identify unnecessary-to-AP processing steps and test removing them from our pipeline.

#### Activity

Hide
John Parejko added a comment -

In discussion with Jim Bosch, he said that the HSM moments are necessary (at minimum the HsmPsfMoments plugin, possibly all of the ones in meas_extensions_shapeHSM/config/enable.py): they're the default shape measurements, having superceded SdssShape for years. I will file a ticket to make the HSM measurements a required part of characterize (it would be nice to move that code out of meas_extensions_shapeHSM since it's not really an "extensions" thing any more, but that's a bigger question), and also re-insert the HSM stuff on this ticket and re-run.

As part of this investigation, we also discovered that characterize is almost certainly running all its plugins too many times: it only should run them after psf determination is complete, not during the determinePsf loop. This might explain why cmodel is such a large part of the runtime, if it's being run repeatedly.

Show
John Parejko added a comment - In discussion with Jim Bosch , he said that the HSM moments are necessary (at minimum the HsmPsfMoments plugin, possibly all of the ones in meas_extensions_shapeHSM/config/enable.py ): they're the default shape measurements, having superceded SdssShape for years. I will file a ticket to make the HSM measurements a required part of characterize (it would be nice to move that code out of meas_extensions_shapeHSM since it's not really an "extensions" thing any more, but that's a bigger question), and also re-insert the HSM stuff on this ticket and re-run. As part of this investigation, we also discovered that characterize is almost certainly running all its plugins too many times: it only should run them after psf determination is complete, not during the determinePsf loop. This might explain why cmodel is such a large part of the runtime, if it's being run repeatedly.
Hide
Jim Bosch added a comment -

"ext_shapeHSM_HsmSourceMoments" is the critical one that I expect to be used as slot_Shape. "ext_shapeHSM_HsmPsfMoments" is also probably useful for diagnostic purposes (both by our team and science users), and you don't need the others, which are there for weak lensing.

Show
Jim Bosch added a comment - "ext_shapeHSM_HsmSourceMoments" is the critical one that I expect to be used as slot_Shape . "ext_shapeHSM_HsmPsfMoments" is also probably useful for diagnostic purposes (both by our team and science users), and you don't need the others, which are there for weak lensing.
Hide
John Parejko added a comment -

Adding HsmSourceMoments and HsmPsfMoments back in to the plugin list results in identical metrics for ci_cosmos_pdr2, and a only these metrics being different for hits2015:

 Difference between ci_hits2015-main/repo/ and DM-34826-hits2015/repo/: 10 metricvalue_ap_association_fracUpdatedDiaObjects 0.0019291053773812517 10 metricvalue_ap_association_numUnassociatedDiaObjects -4.0 ct 56 metricvalue_ap_association_numUnassociatedDiaObjects 24.0 ct 

I've pushed a new ap_pipe commit that adds back those two HSM plugins. I think we can go forward with that.

Show
John Parejko added a comment - Adding HsmSourceMoments and HsmPsfMoments back in to the plugin list results in identical metrics for ci_cosmos_pdr2, and a only these metrics being different for hits2015: Difference between ci_hits2015-main/repo/ and DM-34826-hits2015/repo/: 10 metricvalue_ap_association_fracUpdatedDiaObjects 0.0019291053773812517 10 metricvalue_ap_association_numUnassociatedDiaObjects -4.0 ct 56 metricvalue_ap_association_numUnassociatedDiaObjects 24.0 ct I've pushed a new ap_pipe commit that adds back those two HSM plugins. I think we can go forward with that.
Hide
John Parejko added a comment -

And related to the performance sprint: I don't see HSM taking up a substantial part of the runtime (though it's hard to tell because it's an sfm plugin, so C++, so it doesn't show up in the profile in an obvious manner).

Show
John Parejko added a comment - And related to the performance sprint: I don't see HSM taking up a substantial part of the runtime (though it's hard to tell because it's an sfm plugin, so C++, so it doesn't show up in the profile in an obvious manner).
Hide
John Parejko added a comment -

After discussion with Krzysztof Findeisen, we believe the above difference is due to the order of processing detectors within a visit.

Merged and done! ap_pipe is now quite a bit faster.

Show
John Parejko added a comment - After discussion with Krzysztof Findeisen , we believe the above difference is due to the order of processing detectors within a visit. Merged and done! ap_pipe is now quite a bit faster.

#### People

Assignee:
John Parejko
Reporter:
Eric Bellm
Reviewers:
Krzysztof Findeisen
Watchers:
Eric Bellm, Jim Bosch, John Parejko, Krzysztof Findeisen, Meredith Rawls
Votes:
0 Vote for this issue
Watchers:
5 Start watching this issue

#### Dates

Created:
Updated:
Resolved:

#### Jenkins

No builds found.