Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22299

Speed up specific diaCalculation plugins using fast pandas functionality

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_association
    • Labels:
      None
    • Templates:
    • Story Points:
      4
    • Team:
      Alert Production

      Description

      While working on DM-21688, it was discovered that certainly plugins in diaCalculationPlugins could be sped up by a factor of ~ 100 by using some of the pandas build in functions.

      This ticket will implement these simple changes and compare the speed of using these plugins that of the current stack. The plug insto change are:

      • NumDiaSourcesDiaPlugin
      • SimpleSourceFlagDiaPlugin
      • SigmaDiaPsFlux
      • MinMaxDiaPsFlux
      • ErrMeanDiaPsFlux
      • SigmaDiaTotFlux

      This is around 30% of the plugins current implemented. One would naively expect a roughly a 30% decrease in the speed of the diaCalculation step. Functions for skew and percentiles are implemented in Pandas GroupBy/Dataframe processing but don't seem to be highly optimized like the above calculations are.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cmorrison Chris Morrison
                Reporter:
                cmorrison Chris Morrison
                Reviewers:
                Siegfried Eggl
                Watchers:
                Chris Morrison, Eric Bellm, John Swinbank, Siegfried Eggl
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel