Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4821

HSC backport: Remove interpolated background before detection to reduce junk sources

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_algorithms
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      Science Pipelines DM-W16-5
    • Team:
      Data Release Production

      Description

      This is a port of HSC-1353 and HSC-1360.

      Descriptions from HSC:

      HSC-1353

      We typically get a large number of junk detections around bright objects due to noise fluctuations in the elevated background. We can try to reduce the number of junk detections by adding an additional local background subtraction before object detection. We can then add this back in after detection of footprints and peaks.

      HSC-1360

      I forgot to set the useApprox=True for the background subtraction that runs before footprint and peak detection. This will then use the Chebyshev instead of the spline.

        Attachments

          Issue Links

            Activity

            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Epic Link DM-3628 [ 19726 ]
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lauren Lauren MacArthur made changes -
            Sprint Science Pipelines DM-W16-5 [ 193 ]
            swinbank John Swinbank made changes -
            Epic Link DM-3628 [ 19726 ] DM-3568 [ 19662 ]
            lauren Lauren MacArthur made changes -
            Attachment junkDetections2.png [ 27420 ]
            Attachment junkDetections1.png [ 27421 ]
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Bob Armstrong would you mind having a look at this. I've attached two figures demonstrating the difference between running with this feature off (red crosses) and on (green circles). FYI: This was for processCcd.py runs of visit=1228 ccd=71 of the UDEEP COSMOS field.

            The decision of whether to turn it on by default (or in config overrides in camera packages) will be left until the ongoing testing is completed on the HSC side of things.

            A Jenkins build was successful.

            Show
            lauren Lauren MacArthur added a comment - - edited Bob Armstrong would you mind having a look at this. I've attached two figures demonstrating the difference between running with this feature off (red crosses) and on (green circles). FYI: This was for processCcd.py runs of visit=1228 ccd=71 of the UDEEP COSMOS field. The decision of whether to turn it on by default (or in config overrides in camera packages) will be left until the ongoing testing is completed on the HSC side of things. A Jenkins build was successful.
            lauren Lauren MacArthur made changes -
            Reviewers Bob Armstrong [ rearmstr ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            rearmstr Bob Armstrong added a comment -

            Your plots seem to indicate that turning it on makes things worse in that there are more junk detections. You ran your test on an individual frame, but I only looked at things on the coadds. This probably makes sense in that you would expect more junk for higher S/N. We probably need to test this more before we decide how/where to turn this on.

            Show
            rearmstr Bob Armstrong added a comment - Your plots seem to indicate that turning it on makes things worse in that there are more junk detections. You ran your test on an individual frame, but I only looked at things on the coadds. This probably makes sense in that you would expect more junk for higher S/N. We probably need to test this more before we decide how/where to turn this on.
            Hide
            lauren Lauren MacArthur added a comment -

            I totally agree. I was pretty sure you hadn't tried running this on single frames. It looks to me like there are trade offs to both modes, so further testing is certainly needed before (if/where) turning this feature on.

            Show
            lauren Lauren MacArthur added a comment - I totally agree. I was pretty sure you hadn't tried running this on single frames. It looks to me like there are trade offs to both modes, so further testing is certainly needed before (if/where) turning this feature on.
            rearmstr Bob Armstrong made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            lauren Lauren MacArthur added a comment -

            I made the footprintBackground --> tempLocalBackground name change, pushed a tickets/DM-4821 branch, reran Jenkins (successful) and pushed to master.

            Show
            lauren Lauren MacArthur added a comment - I made the footprintBackground --> tempLocalBackground name change, pushed a tickets/ DM-4821 branch, reran Jenkins (successful) and pushed to master.
            lauren Lauren MacArthur made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ]
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            price Paul Price made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            Hide
            swinbank John Swinbank added a comment -

            And release notes have been updated: thanks!

            Show
            swinbank John Swinbank added a comment - And release notes have been updated: thanks!
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            rearmstr Bob Armstrong made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            rowen Russell Owen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            nlust Nate Lust made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            nlust Nate Lust made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            nlust Nate Lust made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            nlust Nate Lust made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            nlust Nate Lust made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            price Paul Price made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            pschella Pim Schellart [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            nlust Nate Lust made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13313 ] This issue links to "Page (Confluence)" [ 13313 ]

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                Bob Armstrong
                Watchers:
                Bob Armstrong, John Swinbank, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel