Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21688

Create multi plugins for DiaObject summary statistics

    Details

      Description

      Some of the current plugins created for DiaObject summary statistics may benefit from more clever pandas usage and conversion to `multi` style plugins. This ticket will either be done at a later date of if during conversion to the plugin system in ap_association, the plugins are found to be too slow.

        Attachments

          Issue Links

            Activity

            cmorrison Chris Morrison created issue -
            cmorrison Chris Morrison made changes -
            Field Original Value New Value
            Link This issue relates to DM-21167 [ DM-21167 ]
            cmorrison Chris Morrison made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Show
            cmorrison Chris Morrison added a comment - Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30799/pipeline
            cmorrison Chris Morrison made changes -
            Reviewers Eric Bellm [ ebellm ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            cmorrison Chris Morrison made changes -
            Story Points 6 10
            cmorrison Chris Morrison made changes -
            Attachment image-2019-11-21-13-58-19-661.png [ 41329 ]
            cmorrison Chris Morrison made changes -
            Attachment image-2019-11-21-14-00-16-900.png [ 41330 ]
            cmorrison Chris Morrison made changes -
            Attachment image-2019-11-21-14-13-37-240.png [ 41331 ]
            cmorrison Chris Morrison made changes -
            Attachment image-2019-11-21-14-00-16-900.png [ 41330 ]
            Hide
            cmorrison Chris Morrison added a comment -

            This ticket ended up exploring a the timing of the diaCalculation plugin system using the full HiTS2015 dataset. Runs of the current stack we compared to the updated diaCalculation with multi plugins. The timing plotted is the amount of time taken to update and create new diaObject summary statistics. This time does include some database access, though looking at a finer grained timing of the new multi-plugins, ~95% of this time is taken up by plugins on average.

            First is the timing for the current stack with a linear fit to data of timing versus total number of diaObjects that were processed (i.e. updated or created). Errors on the fit are estimated by bootstrapping the datapoints and re-fitting.

            And using the multi-plugins:

            The multi plugins are extremely linear compared to the single plugins. 

            Show
            cmorrison Chris Morrison added a comment - This ticket ended up exploring a the timing of the diaCalculation plugin system using the full HiTS2015 dataset. Runs of the current stack we compared to the updated diaCalculation with multi plugins. The timing plotted is the amount of time taken to update and create new diaObject summary statistics. This time does include some database access, though looking at a finer grained timing of the new multi-plugins, ~95% of this time is taken up by plugins on average. First is the timing for the current stack with a linear fit to data of timing versus total number of diaObjects that were processed (i.e. updated or created). Errors on the fit are estimated by bootstrapping the datapoints and re-fitting. And using the multi-plugins: The multi plugins are extremely linear compared to the single plugins. 
            cmorrison Chris Morrison made changes -
            Link This issue is triggering DM-22299 [ DM-22299 ]
            Hide
            ebellm Eric Bellm added a comment -

            A couple of small items on Github.

            Show
            ebellm Eric Bellm added a comment - A couple of small items on Github.
            ebellm Eric Bellm made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            cmorrison Chris Morrison added a comment -

            Comments finished on github. Added gesture toward DM-21267 to resolve hard coded broad band filter definitions in ap_association.

            Show
            cmorrison Chris Morrison added a comment - Comments finished on github. Added gesture toward DM-21267 to resolve hard coded broad band filter definitions in ap_association.
            Show
            cmorrison Chris Morrison added a comment - Final Jenkins run after review changes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47
            Show
            cmorrison Chris Morrison added a comment - Final Jenkins run after review changes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47
            cmorrison Chris Morrison made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            ebellm Eric Bellm made changes -
            Remote Link This issue links to "Page (Confluence)" [ 22609 ]
            swinbank John Swinbank made changes -
            Epic Link DM-20346 [ 338002 ]
            swinbank John Swinbank made changes -
            Epic Link DM-20346 [ 338002 ] DM-21445 [ 423052 ]

              People

              • Assignee:
                cmorrison Chris Morrison
                Reporter:
                cmorrison Chris Morrison
                Reviewers:
                Eric Bellm
                Watchers:
                Chris Morrison, Eric Bellm, John Swinbank
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel