Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4550

Add SDSS anti-shredding algorithm to deblender

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 8
    • DRP F16-5, DRP F16-6
    • Data Release Production

    Description

      rhl identifies this as a priority on HipChat (HSC room, 2015-12-04, 18:17 Eastern).

      Attachments

        1. example1.png
          example1.png
          500 kB
        2. example2.png
          example2.png
          444 kB
        3. example3.png
          example3.png
          390 kB
        4. example4.png
          example4.png
          340 kB
        5. example5.png
          example5.png
          470 kB

        Issue Links

          Activity

            rhl – could you please add pointers to code, documentation or other relevant material on the SDSS algorithm?

            swinbank John Swinbank added a comment - rhl – could you please add pointers to code, documentation or other relevant material on the SDSS algorithm?

            lauren would you be able to review this ticket?

            rearmstr Bob Armstrong added a comment - lauren would you be able to review this ticket?

            I've not done a code review on this (that's lauren's job) but I did quickly glance at it and I was a bit concerned about the lack of documentation. This is difficult because the deblender as a whole isn't well documented so there's nowhere obvious to add an explanation of what this new algorithm does, but I do think we need to capture more information — even if just in code comments — than has been done so far. Are there published papers (or unpublished proto-papers) we could refer to, for example?

            swinbank John Swinbank added a comment - I've not done a code review on this (that's lauren 's job) but I did quickly glance at it and I was a bit concerned about the lack of documentation. This is difficult because the deblender as a whole isn't well documented so there's nowhere obvious to add an explanation of what this new algorithm does, but I do think we need to capture more information — even if just in code comments — than has been done so far. Are there published papers (or unpublished proto-papers) we could refer to, for example?

            swinbank beat me to it! +1 on more details (wherever they end up) about the algorithm. Perhaps a figure or two demonstrating it in action would be illuminating as well.

            lauren Lauren MacArthur added a comment - swinbank beat me to it! +1 on more details (wherever they end up) about the algorithm. Perhaps a figure or two demonstrating it in action would be illuminating as well.

            I was also not quite sure how to document this. There is no reference except that this procedure helped in SDSS. I am happy to put more comments in the code, but given that the deblender is to going to be completely rewritten (with documentation!) I don't know how useful these will be.

            lauren Is there a place to put documentation with figures?

            rearmstr Bob Armstrong added a comment - I was also not quite sure how to document this. There is no reference except that this procedure helped in SDSS. I am happy to put more comments in the code, but given that the deblender is to going to be completely rewritten (with documentation!) I don't know how useful these will be. lauren Is there a place to put documentation with figures?

            I'd be fine with a few "before" and "after" figures attached to the ticket.

            lauren Lauren MacArthur added a comment - I'd be fine with a few "before" and "after" figures attached to the ticket.
            swinbank John Swinbank added a comment - - edited

            I'd be fine with a few "before" and "after" figures attached to the ticket.

            Hmm, I'm not sure I would. At least, I'd like to discuss in more detail what our longer term plans for the deblender and its documentation will be before agreeing to that. In particular:

            1. Does the "complete rewrite" literally involve throwing away everything and starting again with completely new algorithms? If not, it seems like anything we can capture on this new algorithm now will continue to be useful;
            2. While I think Lauren is asking for documentation on this ticket that helps her understand the impact of the work that has been carried out, I'd actually like to see documentation we can present to folks trying to use the deblender to get work done in the future;
            3. There is some documentation in the doc/design.tex file written by Dustin. That hasn't been touched in a couple of years; I've not read it properly, I don't know if it reflects reality, and I'm not sure if it's actually in a format that we'd find useful, but we should at least consider updating it to reflect this new functionality.
            swinbank John Swinbank added a comment - - edited I'd be fine with a few "before" and "after" figures attached to the ticket. Hmm, I'm not sure I would. At least, I'd like to discuss in more detail what our longer term plans for the deblender and its documentation will be before agreeing to that. In particular: Does the "complete rewrite" literally involve throwing away everything and starting again with completely new algorithms? If not, it seems like anything we can capture on this new algorithm now will continue to be useful; While I think Lauren is asking for documentation on this ticket that helps her understand the impact of the work that has been carried out, I'd actually like to see documentation we can present to folks trying to use the deblender to get work done in the future; There is some documentation in the doc/design.tex file written by Dustin. That hasn't been touched in a couple of years; I've not read it properly, I don't know if it reflects reality, and I'm not sure if it's actually in a format that we'd find useful, but we should at least consider updating it to reflect this new functionality.

            Oh, I didn't mean to imply the documentation should be attached to the ticket, just that if the location of it (whether temporary or not) doesn't easily accommodate demonstrative figures, a few of the latter attached to the ticket would nice.

            lauren Lauren MacArthur added a comment - Oh, I didn't mean to imply the documentation should be attached to the ticket, just that if the location of it (whether temporary or not) doesn't easily accommodate demonstrative figures, a few of the latter attached to the ticket would nice.

            That sounds fine to me.

            swinbank John Swinbank added a comment - That sounds fine to me.

            Here are a couple of examples of what the algorithm does from a list of merging galaxies in HSC that are known to have their flux underestimated as compared to SDSS. (You can find a list of these in DM-8139).

            In the following pictures I have labeled

            • blue "x" = peak the deblender thought was a PSF
            • green "+" = peak that is not a PSF
            • purple "o" = objects identified as having their template degenerate with another one.
            • red "x" = object that had underestimated flux compared to SDSS

            Here are a few examples of cases that look like this is helping to not shred galaxies


            Here are some examples where objects were identified as having a degenerate template, but perhaps should not have been.

            rearmstr Bob Armstrong added a comment - Here are a couple of examples of what the algorithm does from a list of merging galaxies in HSC that are known to have their flux underestimated as compared to SDSS. (You can find a list of these in DM-8139 ). In the following pictures I have labeled blue "x" = peak the deblender thought was a PSF green "+" = peak that is not a PSF purple "o" = objects identified as having their template degenerate with another one. red "x" = object that had underestimated flux compared to SDSS Here are a few examples of cases that look like this is helping to not shred galaxies Here are some examples where objects were identified as having a degenerate template, but perhaps should not have been.

            Looks great rearmstr. A few minor comments on the PR and a request for more test coverage (which can be pushed to another ticket if you'd like).

            lauren Lauren MacArthur added a comment - Looks great rearmstr . A few minor comments on the PR and a request for more test coverage (which can be pushed to another ticket if you'd like).

            People

              rearmstr Bob Armstrong
              swinbank John Swinbank
              Lauren MacArthur
              Bob Armstrong, John Swinbank, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.