Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32838

AP pipeline performance characterization

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe
    • Labels:
    • Story Points:
      6
    • Epic Link:
    • Sprint:
      AP S22-6 (May)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Run ap_pipe on DC2 datasets to characterize the current performance of the pipeline, both CPU time possibly memory usage. Configure the minimal AP pipeline and compare, and look for tasks and functions that take excessive resources.

        Attachments

          Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. DM-34699 Patch the ap_pipe config reset hack

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31063 Copy configs from obs_* packages to ap_pipe

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34623 AP Performance sprint

          • Done

          Technical task - Created by JIRA Agile - do not edit or delete. Issue type for a technical task. DM-34825 Perform initial profiling run on test dataset

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31652 Process a subset of DC2 through ap_pipe

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31653 Reprocess a subset of DC2 with profiling

          • Done

            Activity

            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            For when I get back to this (waiting on DM-31063 so we can not have to worry about DRP-only measurement tasks sneaking in): I have some working scripts and pipeline configurations in /project/parejkoj/ap_profile. Run a set of detectors via profile_script.sh (the DM-33001 version is to profile Ian's work on the new ImageDifferenceTask). average_profile.py will combine the output from multiple detectors into one average profile. It's probably worth doing some more detailed statistics than just a simple average, but this is a start. The --stripped yaml file was a start at cutting out unnecessary measurement plugins, I'm just going to wait a little longer on that now, for the decoupling of obs packages and DRP-focused configs.

            Show
            Parejkoj John Parejko added a comment - For when I get back to this (waiting on DM-31063 so we can not have to worry about DRP-only measurement tasks sneaking in): I have some working scripts and pipeline configurations in /project/parejkoj/ap_profile . Run a set of detectors via profile_script.sh (the DM-33001 version is to profile Ian's work on the new ImageDifferenceTask). average_profile.py will combine the output from multiple detectors into one average profile. It's probably worth doing some more detailed statistics than just a simple average, but this is a start. The --stripped yaml file was a start at cutting out unnecessary measurement plugins, I'm just going to wait a little longer on that now, for the decoupling of obs packages and DRP-focused configs.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              sullivan Ian Sullivan
              Watchers:
              Ian Sullivan, John Parejko, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.