Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14030

Move transformation filters to C++

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_deblender
    • Labels:
      None

      Description

      After DM-9584, the current bottleneck in the deblender is the translation and PSF operators, where (when PSF convolution is used) nearly all of the processing time is spent in a single function:

      def apply_filter(X, weights, slices, inv_slices):
          """Apply a filter to a 2D image X
       
          Parameters
          ----------
          X: 2D numpy array
              The image to apply the filter to
          weights: 1D array
              Weights corresponding to each slice in `slices`
          slices: list of `slice` objects
              Slices in the new `X` to store the filtered X
          inv_slices: list of `slice` objects
              Slices of `X` to apply each weight
          
          Returns
          -------
          new_X: 2D numpy array
              The result of applying the filter to `X`
          """
          result = np.zeros(X.shape, dtype=X.dtype)
          for n, weight in enumerate(weights):
              result[slices[n]] += weight * X[inv_slices[n]]
          return result
      

      Moving this to C++ and using the Eigen package is likely to improve the performance, and is the last tall pole remaining in optimizing the deblender. After this any performance increases will have to come from more clever implementations of certain algorithms and small optimizations throughout the code.

        Attachments

          Activity

          Hide
          fred3m Fred Moolekamp added a comment -

          There is an extra attribute LinearFilter.cpp that is used to turn the C++ methods on/off for testing. Before this ticket is merged that will be removed, but I'm leaving it in until this passes the rest of the review.

          Show
          fred3m Fred Moolekamp added a comment - There is an extra attribute LinearFilter.cpp that is used to turn the C++ methods on/off for testing. Before this ticket is merged that will be removed, but I'm leaving it in until this passes the rest of the review.
          Hide
          fred3m Fred Moolekamp added a comment -

          The changes are made in https://github.com/fred3m/scarlet/pull/43. Jim Bosch, the main parts I'd like you to check are that I made the best use of Eigen in operators_pybind11.cc and my implementation to download the Eigen headers if needed. I tested that this works on my machine, but it would be useful to have both Peter Melchior and Jim Bosch verify that this works for them too.

          Show
          fred3m Fred Moolekamp added a comment - The changes are made in https://github.com/fred3m/scarlet/pull/43. Jim Bosch , the main parts I'd like you to check are that I made the best use of Eigen in  operators_pybind11.cc  and my implementation  to download the Eigen headers if needed. I tested that this works on my machine, but it would be useful to have both Peter Melchior and Jim Bosch verify that this works for them too.
          Hide
          pmelchior Peter Melchior added a comment -

          It works fine for me.

          Show
          pmelchior Peter Melchior added a comment - It works fine for me.
          Hide
          fred3m Fred Moolekamp added a comment -

          Peter Melchior I updated the branch and it should be ready to merge with your approval.

          Show
          fred3m Fred Moolekamp added a comment - Peter Melchior I updated the branch and it should be ready to merge with your approval.
          Hide
          pmelchior Peter Melchior added a comment -

          This is great work and a substantial speed-up, which allows us to routinely do the full PSF treatment.

          Show
          pmelchior Peter Melchior added a comment - This is great work and a substantial speed-up, which allows us to routinely do the full PSF treatment.

            People

            Assignee:
            fred3m Fred Moolekamp
            Reporter:
            fred3m Fred Moolekamp
            Reviewers:
            Jim Bosch, Peter Melchior
            Watchers:
            Fred Moolekamp, Jim Bosch, Peter Melchior
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: