# Create multi plugins for DiaObject summary statistics

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
10
• Team:

## Description

Some of the current plugins created for DiaObject summary statistics may benefit from more clever pandas usage and conversion to multi style plugins. This ticket will either be done at a later date of if during conversion to the plugin system in ap_association, the plugins are found to be too slow.

## Attachments

1. image-2019-11-21-13-58-19-661.png
48 kB
2. image-2019-11-21-14-13-37-240.png
31 kB

## Activity

Hide
Chris Morrison added a comment -
Show
Chris Morrison added a comment - Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30799/pipeline
Hide
Chris Morrison added a comment -

This ticket ended up exploring a the timing of the diaCalculation plugin system using the full HiTS2015 dataset. Runs of the current stack we compared to the updated diaCalculation with multi plugins. The timing plotted is the amount of time taken to update and create new diaObject summary statistics. This time does include some database access, though looking at a finer grained timing of the new multi-plugins, ~95% of this time is taken up by plugins on average.

First is the timing for the current stack with a linear fit to data of timing versus total number of diaObjects that were processed (i.e. updated or created). Errors on the fit are estimated by bootstrapping the datapoints and re-fitting.

And using the multi-plugins:

The multi plugins are extremely linear compared to the single plugins.

Show
Chris Morrison added a comment - This ticket ended up exploring a the timing of the diaCalculation plugin system using the full HiTS2015 dataset. Runs of the current stack we compared to the updated diaCalculation with multi plugins. The timing plotted is the amount of time taken to update and create new diaObject summary statistics. This time does include some database access, though looking at a finer grained timing of the new multi-plugins, ~95% of this time is taken up by plugins on average. First is the timing for the current stack with a linear fit to data of timing versus total number of diaObjects that were processed (i.e. updated or created). Errors on the fit are estimated by bootstrapping the datapoints and re-fitting. And using the multi-plugins: The multi plugins are extremely linear compared to the single plugins.
Hide
Eric Bellm added a comment -

A couple of small items on Github.

Show
Eric Bellm added a comment - A couple of small items on Github.
Hide
Chris Morrison added a comment -

Comments finished on github. Added gesture toward DM-21267 to resolve hard coded broad band filter definitions in ap_association.

Show
Chris Morrison added a comment - Comments finished on github. Added gesture toward DM-21267 to resolve hard coded broad band filter definitions in ap_association.
Show
Chris Morrison added a comment - Final Jenkins run after review changes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47
Show
Chris Morrison added a comment - Final Jenkins run after review changes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47

## People

• Assignee:
Chris Morrison
Reporter:
Chris Morrison
Reviewers:
Eric Bellm
Watchers:
Chris Morrison, Eric Bellm, John Swinbank