Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16151

Diagnose potential issue with setting of NOT_DEBLENDED mask

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      4
    • Epic Link:
    • Sprint:
      DRP F18-5
    • Team:
      Data Release Production

      Description

      In working on some new diagnostics plots, I noted a large area in the COSMOS field that was not getting any detections.  In an effort to make sure I understood the cause and consequences, I looked more deeply into the images, masks, catalogs, and logs from the processing and became suspicious that something was amiss.  This ticket is to document the further exploration and observations in order to diagnose/confirm any issues.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Some context: The dithering pattern for the DEEP COSMOS field is tight, so artifact rejection is hard/impossible. Accordingly, there are many regions showing artifacts that bleed into the coadd and typically I see no source detections around these areas. This is not surprising as the footprints around these areas would understandably end up quite large and would be skipped according to the config parameter:  

            maxFootprintArea = pexConfig.Field(dtype=int, default=1000000, doc=("Maximum area for footprints before they are ignored as large; non-positive means no threshold applied"))
            

            However, the size and extension of these areas looked a bit extreme to me, so I wanted to make sure I was interpreting this correctly.  If this was indeed the case, the NOT_DEBLENDED mask should have been set for all pixels in the affected area.

            I became suspicious/puzzled when I looked into the logs and did not find any warnings about footprints getting skipped. It turns out, the message from the deblender is at the log.trace level, so does not get printed to the logs. So, I set it to manually to WARN and reran multiband. Indeed I saw one instance:

            multiBandDriver.deblendCoaddSources.singleBandDeblend WARN: Parent 43159138515025921: skipping large footprint (area: 2018438)
            

            But only the one…and there seemed to be many “isolated” areas that I thought would also be affected. So I put a break in the deblender to display this one footprint before it skips over it:
            The following shows the (large indeed!) footprint for this skipped source:

            This monster parent source is indeed the catalog and can be identified by the deblend_skipped flag being set. So this should also turn up in the NOT_DEBLENDED mask plane. However, this is a look at some of the mask planes (along with all detections in the upper-right "ALL MASKS") for that image:

            Doh! Looks like the NOT_DEBLENDED mask is empty.

            Conclusions:
            1. The NOT_DEBLENDED mask plane does not seem to be getting set correctly.
            2. We push the log level up to WARN for the footprint skipping so that someone looking in the logs (as I first did...) will see it easily.

            I will make tickets to address these two issues.

            Show
            lauren Lauren MacArthur added a comment - Some context: The dithering pattern for the DEEP COSMOS field is tight, so artifact rejection is hard/impossible. Accordingly, there are many regions showing artifacts that bleed into the coadd and typically I see no source detections around these areas. This is not surprising as the footprints around these areas would understandably end up quite large and would be skipped according to the config parameter:   maxFootprintArea = pexConfig.Field(dtype = int , default = 1000000 , doc = ( "Maximum area for footprints before they are ignored as large; non-positive means no threshold applied" )) However, the size and extension of these areas looked a bit extreme to me, so I wanted to make sure I was interpreting this correctly.  If this was indeed the case, the NOT_DEBLENDED mask should have been set for all pixels in the affected area. I became suspicious/puzzled when I looked into the logs and did not find any warnings about footprints getting skipped. It turns out, the message from the deblender is at the log.trace level, so does not get printed to the logs. So, I set it to manually to WARN and reran multiband. Indeed I saw one instance: multiBandDriver.deblendCoaddSources.singleBandDeblend WARN: Parent 43159138515025921 : skipping large footprint (area: 2018438 ) But only the one…and there seemed to be many “isolated” areas that I thought would also be affected. So I put a break in the deblender to display this one footprint before it skips over it: The following shows the (large indeed!) footprint for this skipped source: This monster parent source is indeed the catalog and can be identified by the deblend_skipped flag being set. So this should also turn up in the NOT_DEBLENDED mask plane. However, this is a look at some of the mask planes (along with all detections in the upper-right "ALL MASKS") for that image: Doh! Looks like the NOT_DEBLENDED mask is empty. Conclusions: 1. The NOT_DEBLENDED mask plane does not seem to be getting set correctly. 2. We push the log level up to WARN for the footprint skipping so that someone looking in the logs (as I first did...) will see it easily. I will make tickets to address these two issues.
            Hide
            lauren Lauren MacArthur added a comment -

            Tickets are DM-16178 and DM-16179.

            Show
            lauren Lauren MacArthur added a comment - Tickets are DM-16178 and DM-16179 .
            Hide
            lauren Lauren MacArthur added a comment -

            Can you have a look and sign off on this if/when you're satisfied.  It's really just a report of an investigation that spawned two tickets, so there's no code to review.

            Show
            lauren Lauren MacArthur added a comment - Can you have a look and sign off on this if/when you're satisfied.  It's really just a report of an investigation that spawned two tickets, so there's no code to review.
            Hide
            yusra Yusra AlSayyad added a comment -

            Nice work solving the case

            Show
            yusra Yusra AlSayyad added a comment - Nice work solving the case

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                Yusra AlSayyad
                Watchers:
                Lauren MacArthur, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel