Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21688

Create multi plugins for DiaObject summary statistics

    XMLWordPrintable

    Details

      Description

      Some of the current plugins created for DiaObject summary statistics may benefit from more clever pandas usage and conversion to `multi` style plugins. This ticket will either be done at a later date of if during conversion to the plugin system in ap_association, the plugins are found to be too slow.

        Attachments

          Issue Links

          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-22299 Speed up specific diaCalculation plugins using fast pandas functionality

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21167 Migrate ap_association fully to new DiaCalculation plugin system for time-series features.

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            cmorrison Chris Morrison created issue -
            cmorrison Chris Morrison made changes -
            Field Original Value New Value
            Link This issue relates to DM-21167 [ DM-21167 ]
            cmorrison Chris Morrison made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            cmorrison Chris Morrison added a comment -

            Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30799/pipeline

            Show
            cmorrison Chris Morrison added a comment - Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30799/pipeline
            cmorrison Chris Morrison made changes -
            Reviewers Eric Bellm [ ebellm ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            cmorrison Chris Morrison made changes -
            Story Points 6 10
            cmorrison Chris Morrison made changes -
            Attachment image-2019-11-21-13-58-19-661.png [ 41329 ]
            cmorrison Chris Morrison made changes -
            Attachment image-2019-11-21-14-00-16-900.png [ 41330 ]
            cmorrison Chris Morrison made changes -
            Attachment image-2019-11-21-14-13-37-240.png [ 41331 ]
            cmorrison Chris Morrison made changes -
            Attachment image-2019-11-21-14-00-16-900.png [ 41330 ]
            Hide
            Permalink
            cmorrison Chris Morrison added a comment -

            This ticket ended up exploring a the timing of the diaCalculation plugin system using the full HiTS2015 dataset. Runs of the current stack we compared to the updated diaCalculation with multi plugins. The timing plotted is the amount of time taken to update and create new diaObject summary statistics. This time does include some database access, though looking at a finer grained timing of the new multi-plugins, ~95% of this time is taken up by plugins on average.

            First is the timing for the current stack with a linear fit to data of timing versus total number of diaObjects that were processed (i.e. updated or created). Errors on the fit are estimated by bootstrapping the datapoints and re-fitting.

            And using the multi-plugins:

            The multi plugins are extremely linear compared to the single plugins. 

            Show
            cmorrison Chris Morrison added a comment - This ticket ended up exploring a the timing of the diaCalculation plugin system using the full HiTS2015 dataset. Runs of the current stack we compared to the updated diaCalculation with multi plugins. The timing plotted is the amount of time taken to update and create new diaObject summary statistics. This time does include some database access, though looking at a finer grained timing of the new multi-plugins, ~95% of this time is taken up by plugins on average. First is the timing for the current stack with a linear fit to data of timing versus total number of diaObjects that were processed (i.e. updated or created). Errors on the fit are estimated by bootstrapping the datapoints and re-fitting. And using the multi-plugins: The multi plugins are extremely linear compared to the single plugins. 
            cmorrison Chris Morrison made changes -
            Link This issue is triggering DM-22299 [ DM-22299 ]
            Hide
            Permalink
            ebellm Eric Bellm added a comment -

            A couple of small items on Github.

            Show
            ebellm Eric Bellm added a comment - A couple of small items on Github.
            ebellm Eric Bellm made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            Permalink
            cmorrison Chris Morrison added a comment -

            Comments finished on github. Added gesture toward DM-21267 to resolve hard coded broad band filter definitions in ap_association.

            Show
            cmorrison Chris Morrison added a comment - Comments finished on github. Added gesture toward DM-21267 to resolve hard coded broad band filter definitions in ap_association.
            Hide
            Permalink
            cmorrison Chris Morrison added a comment -

            Final Jenkins run after review changes: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47

            Show
            cmorrison Chris Morrison added a comment - Final Jenkins run after review changes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47
            Hide
            Permalink
            cmorrison Chris Morrison added a comment -

            Final Jenkins run after review changes: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47

            Show
            cmorrison Chris Morrison added a comment - Final Jenkins run after review changes:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30826/pipeline/47
            cmorrison Chris Morrison made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            ebellm Eric Bellm made changes -
            Remote Link This issue links to "Page (Confluence)" [ 22609 ]
            swinbank John Swinbank made changes -
            Epic Link DM-20346 [ 338002 ]
            swinbank John Swinbank made changes -
            Epic Link DM-20346 [ 338002 ] DM-21445 [ 423052 ]

              People

              Assignee:
              cmorrison Chris Morrison
              Reporter:
              cmorrison Chris Morrison
              Reviewers:
              Eric Bellm
              Watchers:
              Chris Morrison, Eric Bellm, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: