Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17431

Peak culling is removing real sources

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      5
    • Sprint:
      DRP S19-5
    • Team:
      Data Release Production

      Description

      Peak culling was introduced to remove false detections around bright stars. Now it's been observed (with better sky subtraction and deeper images) that real sources are being culled. Ikeda-san is trying different parameter combinations.

        Attachments

        1. AnnotatedFootprintMerge.cc
          18 kB
        2. culled_peaks.jpg
          culled_peaks.jpg
          237 kB
        3. mergeDet-9813-3,3.fits
          6.14 MB
        4. mergeDet-9813-3,3-DM-17431.fits
          6.14 MB
        5. peakcull_v1.pdf
          221 kB
        6. peakculled_mpdetected.png
          peakculled_mpdetected.png
          14 kB
        7. peakCulling.log
          79 kB
        8. peaksWG.png
          peaksWG.png
          36 kB
        9. peaksWGzoom.png
          peaksWGzoom.png
          190 kB
        10. peaksWoG_zoom.png
          peaksWoG_zoom.png
          34 kB
        11. peaksWoG.png
          peaksWoG.png
          184 kB
        12. Screen Shot 2019-04-01 at 5.45.33 PM.png
          Screen Shot 2019-04-01 at 5.45.33 PM.png
          588 kB
        13. Screen Shot 2019-04-11 at 12.32.48 AM.png
          Screen Shot 2019-04-11 at 12.32.48 AM.png
          60 kB
        14. Screen Shot 2019-04-11 at 12.48.55 AM.png
          Screen Shot 2019-04-11 at 12.48.55 AM.png
          80 kB

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment -

            There are two real sources in Ikeda-san's green box that we are concerned about: Let's call them (14008, 14160) and (14013, 14159).

            Here is the behavior we EXPECT:
            1) i-band: add (14008, 14160)(i) to the PeakCatalog
            2) r-band:

            • merge (14009, 14159)-->(14008,14160)(ir).
            • throw away (14013, 14159) because it is < 1" from (14008,14160)

            3) z-band: merge (14008, 14160)-->(14008,14160)(irz).
            4) y-band: merge (14009, 14160)-->(14008,14160)(irzy)
            5) g-band: throw away (14013,14159) because it it < 1" from (14008,14160)

            Therefore, we expect just (14008,14160)(irzy) to be in the catalog.

            The bug:
            actual step 5) g-band:

            • add (14013,14159) into a bigger/different overlapping footprint.
            • Throw away (14008,14160)(irzy) because it is < 1" from (14013,14159)(g)!
              And only (14013,14159)(g) is in the catalog. It only has one band, and gets culled, leaving us with zero.
            Show
            yusra Yusra AlSayyad added a comment - There are two real sources in Ikeda-san's green box that we are concerned about: Let's call them (14008, 14160) and (14013, 14159). Here is the behavior we EXPECT: 1) i-band: add (14008, 14160)(i) to the PeakCatalog 2) r-band: merge (14009, 14159)-->(14008,14160)(ir). throw away (14013, 14159) because it is < 1" from (14008,14160) 3) z-band: merge (14008, 14160)-->(14008,14160)(irz). 4) y-band: merge (14009, 14160)-->(14008,14160)(irzy) 5) g-band: throw away (14013,14159) because it it < 1" from (14008,14160) Therefore, we expect just (14008,14160)(irzy) to be in the catalog. The bug: actual step 5) g-band: add (14013,14159) into a bigger/different overlapping footprint. Throw away (14008,14160)(irzy) because it is < 1" from (14013,14159)(g)! And only (14013,14159)(g) is in the catalog. It only has one band, and gets culled, leaving us with zero.
            Hide
            furusawa.hisanori Hisanori Furusawa added a comment -

            Thank you! I agree with the expected behavior, and the current step 5) is doing something bad.

            Show
            furusawa.hisanori Hisanori Furusawa added a comment - Thank you! I agree with the expected behavior, and the current step 5) is doing something bad.
            Hide
            yusra Yusra AlSayyad added a comment -

            Attaching the before and after for this patch (based on the S18A input detection catalogs)

            mergeDet-9813-3,3.fits

            mergeDet-9813-3,3-DM-17431.fits

             

            Of  35471 total peaks, 372 are new and 173 peaks were the old that aren't in the new.

            Green + are new, red X are not in new (when they were in old). Red X's are peaks from the low-priority band which are now superseded by the neighboring peak in a high priority band.  This behavior is expected and welcome. 


             

             

            Show
            yusra Yusra AlSayyad added a comment - Attaching the before and after for this patch (based on the S18A input detection catalogs) mergeDet-9813-3,3.fits mergeDet-9813-3,3-DM-17431.fits   Of  35471 total peaks, 372 are new and 173 peaks were the old that aren't in the new. Green + are new, red X are not in new (when they were in old). Red X's are peaks from the low-priority band which are now superseded by the neighboring peak in a high priority band.  This behavior is expected and welcome.     
            Hide
            yusra Yusra AlSayyad added a comment -

            Jim Bosch, would you review? PR in afw: https://github.com/lsst/afw/pull/451

            https://ci.lsst.codes/job/stack-os-matrix/29660/

            I went for the minimal lines of code change. It'd be more readable IMO if I made _addSpans public or moved the doMerge check out of each if/else in the loop. 

            Show
            yusra Yusra AlSayyad added a comment - Jim Bosch , would you review? PR in afw:  https://github.com/lsst/afw/pull/451 https://ci.lsst.codes/job/stack-os-matrix/29660/ I went for the minimal lines of code change. It'd be more readable IMO if I made _addSpans public or moved the doMerge  check out of each if/else in the loop. 
            Hide
            yusra Yusra AlSayyad added a comment -

            Merged. Thanks for the review Jim.

            Show
            yusra Yusra AlSayyad added a comment - Merged. Thanks for the review Jim.

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              price Paul Price
              Reviewers:
              Jim Bosch
              Watchers:
              Hiroyuki Ikeda, Hisanori Furusawa, Jim Bosch, Paul Price, Robert Lupton, Sogo Mineo, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.