Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4550

Add SDSS anti-shredding algorithm to deblender

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      8
    • Sprint:
      DRP F16-5, DRP F16-6
    • Team:
      Data Release Production

      Description

      Robert Lupton identifies this as a priority on HipChat (HSC room, 2015-12-04, 18:17 Eastern).

        Attachments

        1. example1.png
          example1.png
          500 kB
        2. example2.png
          example2.png
          444 kB
        3. example3.png
          example3.png
          390 kB
        4. example4.png
          example4.png
          340 kB
        5. example5.png
          example5.png
          470 kB

          Issue Links

            Activity

            swinbank John Swinbank created issue -
            swinbank John Swinbank made changes -
            Field Original Value New Value
            Epic Link DM-3622 [ 19720 ]
            Hide
            swinbank John Swinbank added a comment -

            Robert Lupton – could you please add pointers to code, documentation or other relevant material on the SDSS algorithm?

            Show
            swinbank John Swinbank added a comment - Robert Lupton – could you please add pointers to code, documentation or other relevant material on the SDSS algorithm?
            swinbank John Swinbank made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            swinbank John Swinbank made changes -
            Story Points 8
            swinbank John Swinbank made changes -
            Link This issue blocks DM-7821 [ DM-7821 ]
            Hide
            rearmstr Bob Armstrong added a comment -

            Lauren MacArthur would you be able to review this ticket?

            Show
            rearmstr Bob Armstrong added a comment - Lauren MacArthur would you be able to review this ticket?
            rearmstr Bob Armstrong made changes -
            Reviewers Lauren MacArthur [ lauren ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            rearmstr Bob Armstrong made changes -
            Sprint DRP F16-5 [ 252 ]
            Hide
            swinbank John Swinbank added a comment -

            I've not done a code review on this (that's Lauren MacArthur's job) but I did quickly glance at it and I was a bit concerned about the lack of documentation. This is difficult because the deblender as a whole isn't well documented so there's nowhere obvious to add an explanation of what this new algorithm does, but I do think we need to capture more information — even if just in code comments — than has been done so far. Are there published papers (or unpublished proto-papers) we could refer to, for example?

            Show
            swinbank John Swinbank added a comment - I've not done a code review on this (that's Lauren MacArthur 's job) but I did quickly glance at it and I was a bit concerned about the lack of documentation. This is difficult because the deblender as a whole isn't well documented so there's nowhere obvious to add an explanation of what this new algorithm does, but I do think we need to capture more information — even if just in code comments — than has been done so far. Are there published papers (or unpublished proto-papers) we could refer to, for example?
            Hide
            lauren Lauren MacArthur added a comment -

            John Swinbank beat me to it! +1 on more details (wherever they end up) about the algorithm. Perhaps a figure or two demonstrating it in action would be illuminating as well.

            Show
            lauren Lauren MacArthur added a comment - John Swinbank beat me to it! +1 on more details (wherever they end up) about the algorithm. Perhaps a figure or two demonstrating it in action would be illuminating as well.
            Hide
            rearmstr Bob Armstrong added a comment -

            I was also not quite sure how to document this. There is no reference except that this procedure helped in SDSS. I am happy to put more comments in the code, but given that the deblender is to going to be completely rewritten (with documentation!) I don't know how useful these will be.

            Lauren MacArthur Is there a place to put documentation with figures?

            Show
            rearmstr Bob Armstrong added a comment - I was also not quite sure how to document this. There is no reference except that this procedure helped in SDSS. I am happy to put more comments in the code, but given that the deblender is to going to be completely rewritten (with documentation!) I don't know how useful these will be. Lauren MacArthur Is there a place to put documentation with figures?
            Hide
            lauren Lauren MacArthur added a comment -

            I'd be fine with a few "before" and "after" figures attached to the ticket.

            Show
            lauren Lauren MacArthur added a comment - I'd be fine with a few "before" and "after" figures attached to the ticket.
            Hide
            swinbank John Swinbank added a comment - - edited

            I'd be fine with a few "before" and "after" figures attached to the ticket.

            Hmm, I'm not sure I would. At least, I'd like to discuss in more detail what our longer term plans for the deblender and its documentation will be before agreeing to that. In particular:

            1. Does the "complete rewrite" literally involve throwing away everything and starting again with completely new algorithms? If not, it seems like anything we can capture on this new algorithm now will continue to be useful;
            2. While I think Lauren is asking for documentation on this ticket that helps her understand the impact of the work that has been carried out, I'd actually like to see documentation we can present to folks trying to use the deblender to get work done in the future;
            3. There is some documentation in the doc/design.tex file written by Dustin. That hasn't been touched in a couple of years; I've not read it properly, I don't know if it reflects reality, and I'm not sure if it's actually in a format that we'd find useful, but we should at least consider updating it to reflect this new functionality.
            Show
            swinbank John Swinbank added a comment - - edited I'd be fine with a few "before" and "after" figures attached to the ticket. Hmm, I'm not sure I would. At least, I'd like to discuss in more detail what our longer term plans for the deblender and its documentation will be before agreeing to that. In particular: Does the "complete rewrite" literally involve throwing away everything and starting again with completely new algorithms? If not, it seems like anything we can capture on this new algorithm now will continue to be useful; While I think Lauren is asking for documentation on this ticket that helps her understand the impact of the work that has been carried out, I'd actually like to see documentation we can present to folks trying to use the deblender to get work done in the future; There is some documentation in the doc/design.tex file written by Dustin. That hasn't been touched in a couple of years; I've not read it properly, I don't know if it reflects reality, and I'm not sure if it's actually in a format that we'd find useful, but we should at least consider updating it to reflect this new functionality.
            Hide
            lauren Lauren MacArthur added a comment -

            Oh, I didn't mean to imply the documentation should be attached to the ticket, just that if the location of it (whether temporary or not) doesn't easily accommodate demonstrative figures, a few of the latter attached to the ticket would nice.

            Show
            lauren Lauren MacArthur added a comment - Oh, I didn't mean to imply the documentation should be attached to the ticket, just that if the location of it (whether temporary or not) doesn't easily accommodate demonstrative figures, a few of the latter attached to the ticket would nice.
            Hide
            swinbank John Swinbank added a comment -

            That sounds fine to me.

            Show
            swinbank John Swinbank added a comment - That sounds fine to me.
            Hide
            rearmstr Bob Armstrong added a comment -

            Here are a couple of examples of what the algorithm does from a list of merging galaxies in HSC that are known to have their flux underestimated as compared to SDSS. (You can find a list of these in DM-8139).

            In the following pictures I have labeled

            • blue "x" = peak the deblender thought was a PSF
            • green "+" = peak that is not a PSF
            • purple "o" = objects identified as having their template degenerate with another one.
            • red "x" = object that had underestimated flux compared to SDSS

            Here are a few examples of cases that look like this is helping to not shred galaxies


            Here are some examples where objects were identified as having a degenerate template, but perhaps should not have been.

            Show
            rearmstr Bob Armstrong added a comment - Here are a couple of examples of what the algorithm does from a list of merging galaxies in HSC that are known to have their flux underestimated as compared to SDSS. (You can find a list of these in DM-8139 ). In the following pictures I have labeled blue "x" = peak the deblender thought was a PSF green "+" = peak that is not a PSF purple "o" = objects identified as having their template degenerate with another one. red "x" = object that had underestimated flux compared to SDSS Here are a few examples of cases that look like this is helping to not shred galaxies Here are some examples where objects were identified as having a degenerate template, but perhaps should not have been.
            rearmstr Bob Armstrong made changes -
            Attachment example1.png [ 28588 ]
            Attachment example3.png [ 28589 ]
            Attachment example2.png [ 28590 ]
            Attachment example4.png [ 28591 ]
            Attachment example5.png [ 28592 ]
            swinbank John Swinbank made changes -
            Sprint DRP F16-5 [ 252 ] DRP F16-5, DRP F16-6 [ 252, 287 ]
            swinbank John Swinbank made changes -
            Rank Ranked higher
            Hide
            lauren Lauren MacArthur added a comment -

            Looks great Bob Armstrong. A few minor comments on the PR and a request for more test coverage (which can be pushed to another ticket if you'd like).

            Show
            lauren Lauren MacArthur added a comment - Looks great Bob Armstrong . A few minor comments on the PR and a request for more test coverage (which can be pushed to another ticket if you'd like).
            lauren Lauren MacArthur made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            rearmstr Bob Armstrong made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14411 ]

              People

              • Assignee:
                rearmstr Bob Armstrong
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Lauren MacArthur
                Watchers:
                Bob Armstrong, John Swinbank, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel