Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34623 AP Performance sprint
  3. DM-34826

Identify unnecessary configs and Tasks and test removal from ap_pipe

    XMLWordPrintable

Details

    • Technical task
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • AP S22-6 (May)
    • Alert Production

    Description

      Likely in pair-coding, identify unnecessary-to-AP processing steps and test removing them from our pipeline.

      Attachments

        Issue Links

          Activity

            Parejkoj John Parejko added a comment -

            In discussion with jbosch, he said that the HSM moments are necessary (at minimum the HsmPsfMoments plugin, possibly all of the ones in meas_extensions_shapeHSM/config/enable.py): they're the default shape measurements, having superceded SdssShape for years. I will file a ticket to make the HSM measurements a required part of characterize (it would be nice to move that code out of meas_extensions_shapeHSM since it's not really an "extensions" thing any more, but that's a bigger question), and also re-insert the HSM stuff on this ticket and re-run.

            As part of this investigation, we also discovered that characterize is almost certainly running all its plugins too many times: it only should run them after psf determination is complete, not during the determinePsf loop. This might explain why cmodel is such a large part of the runtime, if it's being run repeatedly.

            Parejkoj John Parejko added a comment - In discussion with jbosch , he said that the HSM moments are necessary (at minimum the HsmPsfMoments plugin, possibly all of the ones in meas_extensions_shapeHSM/config/enable.py ): they're the default shape measurements, having superceded SdssShape for years. I will file a ticket to make the HSM measurements a required part of characterize (it would be nice to move that code out of meas_extensions_shapeHSM since it's not really an "extensions" thing any more, but that's a bigger question), and also re-insert the HSM stuff on this ticket and re-run. As part of this investigation, we also discovered that characterize is almost certainly running all its plugins too many times: it only should run them after psf determination is complete, not during the determinePsf loop. This might explain why cmodel is such a large part of the runtime, if it's being run repeatedly.
            jbosch Jim Bosch added a comment -

            "ext_shapeHSM_HsmSourceMoments" is the critical one that I expect to be used as slot_Shape. "ext_shapeHSM_HsmPsfMoments" is also probably useful for diagnostic purposes (both by our team and science users), and you don't need the others, which are there for weak lensing.

            jbosch Jim Bosch added a comment - "ext_shapeHSM_HsmSourceMoments" is the critical one that I expect to be used as slot_Shape . "ext_shapeHSM_HsmPsfMoments" is also probably useful for diagnostic purposes (both by our team and science users), and you don't need the others, which are there for weak lensing.

            Adding HsmSourceMoments and HsmPsfMoments back in to the plugin list results in identical metrics for ci_cosmos_pdr2, and a only these metrics being different for hits2015:

            Difference between ci_hits2015-main/repo/ and DM-34826-hits2015/repo/:
            10 metricvalue_ap_association_fracUpdatedDiaObjects 0.0019291053773812517
            10 metricvalue_ap_association_numUnassociatedDiaObjects -4.0 ct
            56 metricvalue_ap_association_numUnassociatedDiaObjects 24.0 ct
            

            I've pushed a new ap_pipe commit that adds back those two HSM plugins. I think we can go forward with that.

            Parejkoj John Parejko added a comment - Adding HsmSourceMoments and HsmPsfMoments back in to the plugin list results in identical metrics for ci_cosmos_pdr2, and a only these metrics being different for hits2015: Difference between ci_hits2015-main/repo/ and DM-34826-hits2015/repo/: 10 metricvalue_ap_association_fracUpdatedDiaObjects 0.0019291053773812517 10 metricvalue_ap_association_numUnassociatedDiaObjects -4.0 ct 56 metricvalue_ap_association_numUnassociatedDiaObjects 24.0 ct I've pushed a new ap_pipe commit that adds back those two HSM plugins. I think we can go forward with that.

            And related to the performance sprint: I don't see HSM taking up a substantial part of the runtime (though it's hard to tell because it's an sfm plugin, so C++, so it doesn't show up in the profile in an obvious manner).

            Parejkoj John Parejko added a comment - And related to the performance sprint: I don't see HSM taking up a substantial part of the runtime (though it's hard to tell because it's an sfm plugin, so C++, so it doesn't show up in the profile in an obvious manner).

            After discussion with krzys, we believe the above difference is due to the order of processing detectors within a visit.

            Merged and done! ap_pipe is now quite a bit faster.

            Parejkoj John Parejko added a comment - After discussion with krzys , we believe the above difference is due to the order of processing detectors within a visit. Merged and done! ap_pipe is now quite a bit faster.

            People

              Parejkoj John Parejko
              ebellm Eric Bellm
              Krzysztof Findeisen
              Eric Bellm, Jim Bosch, John Parejko, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.