Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21688

Create multi plugins for DiaObject summary statistics

    Details

      Description

      Some of the current plugins created for DiaObject summary statistics may benefit from more clever pandas usage and conversion to `multi` style plugins. This ticket will either be done at a later date of if during conversion to the plugin system in ap_association, the plugins are found to be too slow.

        Attachments

          Issue Links

            Activity

            Show
            cmorrison Chris Morrison added a comment - Final Jenkins run after review changes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47
            Show
            cmorrison Chris Morrison added a comment - Final Jenkins run after review changes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47
            Hide
            cmorrison Chris Morrison added a comment -

            Comments finished on github. Added gesture toward DM-21267 to resolve hard coded broad band filter definitions in ap_association.

            Show
            cmorrison Chris Morrison added a comment - Comments finished on github. Added gesture toward DM-21267 to resolve hard coded broad band filter definitions in ap_association.
            Hide
            ebellm Eric Bellm added a comment -

            A couple of small items on Github.

            Show
            ebellm Eric Bellm added a comment - A couple of small items on Github.
            Hide
            cmorrison Chris Morrison added a comment -

            This ticket ended up exploring a the timing of the diaCalculation plugin system using the full HiTS2015 dataset. Runs of the current stack we compared to the updated diaCalculation with multi plugins. The timing plotted is the amount of time taken to update and create new diaObject summary statistics. This time does include some database access, though looking at a finer grained timing of the new multi-plugins, ~95% of this time is taken up by plugins on average.

            First is the timing for the current stack with a linear fit to data of timing versus total number of diaObjects that were processed (i.e. updated or created). Errors on the fit are estimated by bootstrapping the datapoints and re-fitting.

            And using the multi-plugins:

            The multi plugins are extremely linear compared to the single plugins. 

            Show
            cmorrison Chris Morrison added a comment - This ticket ended up exploring a the timing of the diaCalculation plugin system using the full HiTS2015 dataset. Runs of the current stack we compared to the updated diaCalculation with multi plugins. The timing plotted is the amount of time taken to update and create new diaObject summary statistics. This time does include some database access, though looking at a finer grained timing of the new multi-plugins, ~95% of this time is taken up by plugins on average. First is the timing for the current stack with a linear fit to data of timing versus total number of diaObjects that were processed (i.e. updated or created). Errors on the fit are estimated by bootstrapping the datapoints and re-fitting. And using the multi-plugins: The multi plugins are extremely linear compared to the single plugins. 
            Show
            cmorrison Chris Morrison added a comment - Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30799/pipeline

              People

              • Assignee:
                cmorrison Chris Morrison
                Reporter:
                cmorrison Chris Morrison
                Reviewers:
                Eric Bellm
                Watchers:
                Chris Morrison, Eric Bellm, John Swinbank
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel