Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9953

Pixels rejected from coaddition and CCD are not masked on coadds

    XMLWordPrintable

    Details

      Description

      Rejecting an input pixel during coaddition does not result in the CLIPPED bit being set, even though can yield bad CoaddPsf.

      We similarly do not flag objects that lie on the edges of CCDs in the coadd. This could probably be fixed by propagating the EDGE flag through to the coadds.

        Attachments

          Issue Links

          blocks

          Verification - An activity relating to planned verification LVV-106 DMS-REQ-0275-V-01: Object Catalog

          • 1b - Must be demonstrated to be working before the start of the observing.
          • Not Covered

          Verification - An activity relating to planned verification LVV-162 DMS-REQ-0331-V-01: Computing Derived Quantities

          • 1b - Must be demonstrated to be working before the start of the observing.
          • Not Covered

          Verification - An activity relating to planned verification LVV-165 DMS-REQ-0334-V-01: Persisting Data Products

          • 1b - Must be demonstrated to be working before the start of the observing.
          • Not Covered

          Verification - An activity relating to planned verification LVV-178 DMS-REQ-0347-V-01: Measurements in catalogs

          • 1b - Must be demonstrated to be working before the start of the observing.
          • Not Covered
          is duplicated by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-5528 Remove background matching from pipe_tasks

          • Invalid
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12881 coaddDriver tries to unset doMatchBackgrounds, which does not exist

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-9979 Redesign afw::math::Statistics

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12931 Ignore EDGE pixels in coaddition

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            yusra Yusra AlSayyad added a comment -

            CLIPPED bit is being manually set in SafeClip and now CompareWarp. Remove these blocks when DM-9953 is complete:

            https://github.com/lsst/pipe_tasks/commit/a1b2df812ae7b91836085ebea70598a41e9ca156

            Show
            yusra Yusra AlSayyad added a comment - CLIPPED bit is being manually set in SafeClip and now CompareWarp. Remove these blocks when DM-9953 is complete: https://github.com/lsst/pipe_tasks/commit/a1b2df812ae7b91836085ebea70598a41e9ca156
            Hide
            Permalink
            price Paul Price added a comment -

            I'm adding some statistics (NCLIPPED, NMASKED), which Pim Schellart [X] may be interested in incorporating into his work.

            Show
            price Paul Price added a comment - I'm adding some statistics ( NCLIPPED , NMASKED ), which Pim Schellart [X] may be interested in incorporating into his work.
            Hide
            Permalink
            price Paul Price added a comment -

            Thanks for volunteering to look at this, Jim Bosch. There are commits in afw, obs_subaru and pipe_tasks. This includes stripping out background matching from assembleCoadd (because it was easy to do and has long been hanging over my head and it makes it easier to follow the code). I'm about to run this through Jenkins, but I've verified that a ci_hsc coadd's mask is what I expect.

            Show
            price Paul Price added a comment - Thanks for volunteering to look at this, Jim Bosch . There are commits in afw, obs_subaru and pipe_tasks. This includes stripping out background matching from assembleCoadd (because it was easy to do and has long been hanging over my head and it makes it easier to follow the code). I'm about to run this through Jenkins, but I've verified that a ci_hsc coadd's mask is what I expect.
            Hide
            Permalink
            price Paul Price added a comment -

            Here's the i-band coadd from ci_hsc, with the CLIPPED pixels in white. The edges of the CCDs are showing up, along with pixels rejected in the coadd.

            Show
            price Paul Price added a comment - Here's the i-band coadd from ci_hsc, with the CLIPPED pixels in white. The edges of the CCDs are showing up, along with pixels rejected in the coadd.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Looks good. One minor comment on how you may be able to slightly improve some test code.

            But while this is all functional, I wonder if it's an abuse of CLIPPED for it to be used for EDGE pixels (and, to lesser extent, mask-rejected pixels). Since we can now add new mask planes without much cost, it might be a good idea to do so; something like:

            • CLIPPED: one or more pixels going into the coadd was rejected by safe-clip or WarpCompare.
            • REJECTED: one or more pixels going into the coadd was rejected by mask values set in processCcd.
            • EDGE: one or more pixels going into the coadd was on the edge of a CCD
            • INEXACT_PSF: CLIPPED | REJECTED | EDGE
            Show
            jbosch Jim Bosch added a comment - Looks good. One minor comment on how you may be able to slightly improve some test code. But while this is all functional, I wonder if it's an abuse of CLIPPED for it to be used for EDGE pixels (and, to lesser extent, mask-rejected pixels). Since we can now add new mask planes without much cost, it might be a good idea to do so; something like: CLIPPED: one or more pixels going into the coadd was rejected by safe-clip or WarpCompare. REJECTED: one or more pixels going into the coadd was rejected by mask values set in processCcd. EDGE: one or more pixels going into the coadd was on the edge of a CCD INEXACT_PSF: CLIPPED | REJECTED | EDGE
            Hide
            Permalink
            price Paul Price added a comment -

            One additional change in obs_sdss.

            Show
            price Paul Price added a comment - One additional change in obs_sdss.
            Hide
            Permalink
            price Paul Price added a comment -

            The modified version is starting to go green on Jenkins.

            Show
            price Paul Price added a comment - The modified version is starting to go green on Jenkins .
            Hide
            Permalink
            price Paul Price added a comment -

            Merged to master.

            I should explain the changes made after offline consultation with Jim:

            • Pixels that are masked EDGE on the individual CCDs get masked SENSOR_EDGE on the coadd.
            • Pixels that are clipped/rejected (there's no point distinguishing between the two because we don't do any rejection so as to avoid messing up the PSF) are masked CLIPPED on the coadd.
            • Any pixel masked either SENSOR_EDGE or CLIPPED are also masked INEXACT_PSF.
            • Added each of these to the PixelFlags measurement in both measureCoaddSources and forcedPhotCoadd.
            Show
            price Paul Price added a comment - Merged to master. I should explain the changes made after offline consultation with Jim: Pixels that are masked EDGE on the individual CCDs get masked SENSOR_EDGE on the coadd. Pixels that are clipped/rejected (there's no point distinguishing between the two because we don't do any rejection so as to avoid messing up the PSF) are masked CLIPPED on the coadd. Any pixel masked either SENSOR_EDGE or CLIPPED are also masked INEXACT_PSF . Added each of these to the PixelFlags measurement in both measureCoaddSources and forcedPhotCoadd.

              People

              Assignee:
              price Paul Price
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Paul Price, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.