Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24862

Update with latest scarlet and proxmin and test for performance

    Details

    • Story Points:
      5
    • Sprint:
      DRP S20-6 (May)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The API for scarlet changed slightly, with some improvements in performance and in memory usage. So this ticket is to update meas_extensions_scarlet to properly wrap and propagate those changes and run on the fake patches to measure the gains in performance from the previously tested version.

        Attachments

          Activity

          Show
          fred3m Fred Moolekamp added a comment - Jenkins build passed: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31947/pipeline/
          Hide
          fred3m Fred Moolekamp added a comment -

          Note to the reviewer: the proxmin and scarlet code is external and has already been reviewed, they are just updated along with meas_extensions_scarlet to keep it in working order.

          There are two modules in meas_extensions_scarlet, observation and blend, that were removed because they only contained classes that just passed through their scarlet equivalents without any modifications. There are a few other changes that you'll notice:

          • We decided a while back to limit the logic that goes into source initialization since each user will have their own specific initialization needs. So I have my own 3rd party library scarlet_extensions that contains my initialization scripts for use outside of the stack and for the past year or so I have maintained a second version of that code in meas_extensions_scarlet.source that uses stack primitives. This has been cumbersome to maintain, so instead I modified meas_extensions_scarlet so that I can just copy and paste my initialization script into meas_extensions_scarlet and execute it without any stack dependencies. In the future this can be made even more robust by including scarlet_extensions in the stack conda package but I'd like to wait until the end of the next Epic when (hopefully) our initialization method has stabilized.
          • A few extra testing config parameters have been added to make it easier to identify broken code
          • A few minor changes were made in the code and the tests due to small changes in the scarlet API.
          Show
          fred3m Fred Moolekamp added a comment - Note to the reviewer: the proxmin and scarlet code is external and has already been reviewed, they are just updated along with meas_extensions_scarlet to keep it in working order. There are two modules in meas_extensions_scarlet , observation and blend , that were removed because they only contained classes that just passed through their scarlet equivalents without any modifications. There are a few other changes that you'll notice: We decided a while back to limit the logic that goes into source initialization since each user will have their own specific initialization needs. So I have my own 3rd party library scarlet_extensions that contains my initialization scripts for use outside of the stack and for the past year or so I have maintained a second version of that code in meas_extensions_scarlet.source that uses stack primitives. This has been cumbersome to maintain, so instead I modified meas_extensions_scarlet so that I can just copy and paste my initialization script into meas_extensions_scarlet and execute it without any stack dependencies. In the future this can be made even more robust by including scarlet_extensions in the stack conda package but I'd like to wait until the end of the next Epic when (hopefully) our initialization method has stabilized. A few extra testing config parameters have been added to make it easier to identify broken code A few minor changes were made in the code and the tests due to small changes in the scarlet API.
          Hide
          fred3m Fred Moolekamp added a comment -

          Hi Dan, would you mind reviewing this ticket for me?

          Thank you,
          -Fred

          Show
          fred3m Fred Moolekamp added a comment - Hi Dan, would you mind reviewing this ticket for me? Thank you, -Fred
          Hide
          dtaranu Dan Taranu added a comment -

          I only had minor suggestions/fixes.

          Show
          dtaranu Dan Taranu added a comment - I only had minor suggestions/fixes.

            People

            • Assignee:
              fred3m Fred Moolekamp
              Reporter:
              fred3m Fred Moolekamp
              Reviewers:
              Dan Taranu
              Watchers:
              Dan Taranu, Fred Moolekamp
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel