Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12776

Fix problem in deblender interface with the stack

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_deblender
    • Labels:
      None

      Description

      There appears to be a problem with the interface between the stack and the deblender. In the attached images, stack_output shows the data and the footprints created from the stack results, while deblender_output shows the results when the deblender is run using the exact same settings. This shows that somehow the results aren't being persisted properly in the pipeline.

      This ticket is to investigate includes work the last few days to diagnose the problem and upcoming work to find and fix the bug.

        Attachments

        1. deblender_output.png
          deblender_output.png
          132 kB
        2. flux_comparison.png
          flux_comparison.png
          504 kB
        3. stack_output.png
          stack_output.png
          460 kB
        4. variance.png
          variance.png
          989 kB

          Issue Links

            Activity

            Hide
            fred3m Fred Moolekamp added a comment -

            Variance map for the entire patch

            Show
            fred3m Fred Moolekamp added a comment - Variance map for the entire patch
            Hide
            fred3m Fred Moolekamp added a comment -

            Comparison of the total flux of each source in the old and new deblenders.

            Show
            fred3m Fred Moolekamp added a comment - Comparison of the total flux of each source in the old and new deblenders.
            Hide
            fred3m Fred Moolekamp added a comment -

            It appears that the problem is a result of the way that the inverse variance map is used to weight pixels. I added two plots to illustrate what I think the problem is, but I confirmed that when we don't use weights in the deblender we get the correct result (in this field).

            flux_comparison.png shows the total flux for each source, in each band, compared between the old and new deblenders. We see an obvious underestimate of flux in the HSC-Z and HSC-Y bands in the new deblender in the comparison plots, which is also what we see in the colored image generated by the new deblender when the inverse variance is used as a weight map (stack_output, right panel).

            variance.png show the variance map in each band for the entire patch. The mean variance in each band is:
            HSC-G: 0.0017263
            HSC-R: 0.00255741
            HSC-I: 0.0041927
            HSC-Z: 0.94245
            HSC-Y: 0.0668132

            So we see that there is more than an order of magnitude difference between GRI and Y, and the variance in Z is more than 100 times greater.

            It's possible that the deblender is not handling weights correctly, which is causing a failure when the variance between bands is very different.

            Show
            fred3m Fred Moolekamp added a comment - It appears that the problem is a result of the way that the inverse variance map is used to weight pixels. I added two plots to illustrate what I think the problem is, but I confirmed that when we don't use weights in the deblender we get the correct result (in this field). flux_comparison.png shows the total flux for each source, in each band, compared between the old and new deblenders. We see an obvious underestimate of flux in the HSC-Z and HSC-Y bands in the new deblender in the comparison plots, which is also what we see in the colored image generated by the new deblender when the inverse variance is used as a weight map (stack_output, right panel). variance.png show the variance map in each band for the entire patch. The mean variance in each band is: HSC-G: 0.0017263 HSC-R: 0.00255741 HSC-I: 0.0041927 HSC-Z: 0.94245 HSC-Y: 0.0668132 So we see that there is more than an order of magnitude difference between GRI and Y, and the variance in Z is more than 100 times greater. It's possible that the deblender is not handling weights correctly, which is causing a failure when the variance between bands is very different.
            Hide
            fred3m Fred Moolekamp added a comment -

            It is also worth mentioning that there were a small number (8) of NaN values in the variance map, which probably shouldn't be there and had to be modified.

            Show
            fred3m Fred Moolekamp added a comment - It is also worth mentioning that there were a small number (8) of NaN values in the variance map, which probably shouldn't be there and had to be modified.
            Hide
            fred3m Fred Moolekamp added a comment -

            DM-12783 resolved this issue.

            Show
            fred3m Fred Moolekamp added a comment - DM-12783 resolved this issue.

              People

              • Assignee:
                fred3m Fred Moolekamp
                Reporter:
                fred3m Fred Moolekamp
                Watchers:
                Fred Moolekamp
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel