Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8959

Create NMF deblender plugin

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      16
    • Sprint:
      DRP S17-2, DRP S17-3
    • Team:
      Data Release Production

      Description

      DM-8621, which will re-organize the deblender tasks, requires some sort of multi-color deblender. The best candidate so far is using non-negative matrix factorization to ensure that templates have the same footprint in each band.

      This ticket involves writing and testing an NMF plugin for the deblender and comparing its performance to the current deblender. NMF may be robust enough to replace the current SDSS deblender or it might just be another step in the deblending process.

        Attachments

          Issue Links

            Activity

            Hide
            fred3m Fred Moolekamp added a comment -

            This ticket was marked DONE already, but accidentally re-opened this morning.

            Show
            fred3m Fred Moolekamp added a comment - This ticket was marked DONE already, but accidentally re-opened this morning.
            Hide
            fred3m Fred Moolekamp added a comment -

            Oops, I accidentally marked this as in progress. It should still be in review.

            Show
            fred3m Fred Moolekamp added a comment - Oops, I accidentally marked this as in progress. It should still be in review.
            Hide
            rearmstr Bob Armstrong added a comment -

            Code looks like a good start to implementing the new deblender.

            Show
            rearmstr Bob Armstrong added a comment - Code looks like a good start to implementing the new deblender.
            Hide
            fred3m Fred Moolekamp added a comment -

            Hi Bob,
            Can you review this ticket for me? It doesn't require a code review, since nothing is being merged to the stack, but John Swinbank wanted to have you look it over and give your approval that I'm not doing anything crazy.

            Show
            fred3m Fred Moolekamp added a comment - Hi Bob, Can you review this ticket for me? It doesn't require a code review, since nothing is being merged to the stack, but John Swinbank wanted to have you look it over and give your approval that I'm not doing anything crazy.
            Hide
            fred3m Fred Moolekamp added a comment - - edited

            The Jupyter notebooks https://github.com/lsst/meas_deblender/blob/u/fred3m/deblender/examples/testNmfLeeSeung.ipynb and https://github.com/lsst/meas_deblender/blob/u/fred3m/deblender/examples/testNmfHSC.ipynb illustrate the current status of the NMF deblender when running on simulated and HSC data respectively. The simulated data was created by Bob Armstrong as part of DM-8624.

            Until we have good PSF homogenized data to run through the NMF deblender it is difficult to say how the deblender performs on real data, however it's performance on the simulated data is encouraging. Compared to the current deblender, which the NMF deblender uses as initial conditions, the NMF algorithm shows significant improvements, although more thorough analysis needs to be performed to verify that this holds for the general case. It does, however, perform poorly on the current calibrated HSC images due to color gradients created by their differing PSF's.

            There are still many details that need to be worked out, including fine tuning of the constraints and a reasonable stopping criteria for each blend, but the biggest point of failure is the inability to deblend sources with nearly the same SED (as we expected). One of the first tasks in the next sprint will be to develop a constraint that enforces monotonicity (DM-9143), which ideally will solve the problem of degenerate SED's. This will require the creation of a linear monotonicity operator to act on the intensity matrix H that will penalize pixels that are not monotonically decreasing from its center.

            Show
            fred3m Fred Moolekamp added a comment - - edited The Jupyter notebooks https://github.com/lsst/meas_deblender/blob/u/fred3m/deblender/examples/testNmfLeeSeung.ipynb and https://github.com/lsst/meas_deblender/blob/u/fred3m/deblender/examples/testNmfHSC.ipynb illustrate the current status of the NMF deblender when running on simulated and HSC data respectively. The simulated data was created by Bob Armstrong as part of DM-8624 . Until we have good PSF homogenized data to run through the NMF deblender it is difficult to say how the deblender performs on real data, however it's performance on the simulated data is encouraging. Compared to the current deblender, which the NMF deblender uses as initial conditions, the NMF algorithm shows significant improvements, although more thorough analysis needs to be performed to verify that this holds for the general case. It does, however, perform poorly on the current calibrated HSC images due to color gradients created by their differing PSF's. There are still many details that need to be worked out, including fine tuning of the constraints and a reasonable stopping criteria for each blend, but the biggest point of failure is the inability to deblend sources with nearly the same SED (as we expected). One of the first tasks in the next sprint will be to develop a constraint that enforces monotonicity ( DM-9143 ), which ideally will solve the problem of degenerate SED's. This will require the creation of a linear monotonicity operator to act on the intensity matrix H that will penalize pixels that are not monotonically decreasing from its center.
            Hide
            fred3m Fred Moolekamp added a comment -

            After a conversation with John Swinbank the goal of this ticket became to create a user branch (https://github.com/lsst/meas_deblender/tree/u/fred3m/deblender) to contain work on the NMF deblender. A big part of this included evaluating whether or not using NMF is a valid option and identifying parts of the algorithm that can be improved in future tickets.

            For a good overview of NMF in general see https://www.semanticscholar.org/paper/Algorithms-and-applications-for-approximate-Berry-Browne/153adef6b1ee85833fb4261b3c5f616fc06c5a1e. Attached to this ticket is a more detailed look at how the NMF algorithm has been implemented in this ticket. Peter Melchior is currently investigating other improvements.

            Show
            fred3m Fred Moolekamp added a comment - After a conversation with John Swinbank the goal of this ticket became to create a user branch ( https://github.com/lsst/meas_deblender/tree/u/fred3m/deblender ) to contain work on the NMF deblender. A big part of this included evaluating whether or not using NMF is a valid option and identifying parts of the algorithm that can be improved in future tickets. For a good overview of NMF in general see https://www.semanticscholar.org/paper/Algorithms-and-applications-for-approximate-Berry-Browne/153adef6b1ee85833fb4261b3c5f616fc06c5a1e . Attached to this ticket is a more detailed look at how the NMF algorithm has been implemented in this ticket. Peter Melchior is currently investigating other improvements.

              People

              • Assignee:
                fred3m Fred Moolekamp
                Reporter:
                fred3m Fred Moolekamp
                Reviewers:
                Bob Armstrong
                Watchers:
                Bob Armstrong, Fred Moolekamp
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel