Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13396

Fix coadd mask propagation

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Sprint:
      DRP S18-2
    • Team:
      Data Release Production

      Description

      DM-9953 created the SENSOR_EDGE mask to mark coadd pixels that were on or near the boundary of an input CCD (and hence should have INEXACT_PSF set as well). It also started the propagation of those EDGE regions to the coadd.

      Propagating the EDGE regions to the coadd caused problems, however, because many of them were affected by otherwise-unmasked bad pixels (leaks from amps or somesuch), so that was disabled on DM-12931. Contrary to comments on that ticket, this seems to have broken the propagation of SENSOR_EDGE, or perhaps something more recent broke it.

      We should also consider whether to split the current CLIPPED flag into both CLIPPED and REJECTED, with the latter being used for pixels rejected due to mask values from the calexps rather than an explicit smart-clip algorithm (e.g. SafeClip or CompareWarps).

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            The error in coaddDriver is my fault from not adapting it to the new API following DM-12995. I've fixed this on a ticket branch:

            price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-13396=) $ git sub-patch
            commit f99c95d48896431c128506e3890c3ac9b1ca84f3 (HEAD -> tickets/DM-13396, origin/tickets/DM-13396)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Jan 30 09:13:13 2018 -0500
             
                coaddDriver: adapt to API change in detection
                
                DetectCoaddSourcesTask.runDetection now requires an exposure ID as a
                RNG seed.
             
            diff --git a/python/lsst/pipe/drivers/coaddDriver.py b/python/lsst/pipe/drivers/coaddDriver.py
            index b9c8d4d..c7d8bd0 100644
            --- a/python/lsst/pipe/drivers/coaddDriver.py
            +++ b/python/lsst/pipe/drivers/coaddDriver.py
            @@ -310,9 +310,10 @@ class CoaddDriverTask(BatchPoolTask):
                         with self.logOperation("detection on {}".format(patchRef.dataId),
                                                catch=True):
                             idFactory = self.detectCoaddSources.makeIdFactory(patchRef)
            +                expId = int(patchRef.get(self.config.coaddName + "CoaddId"))
                             # This includes background subtraction, so do it before writing
                             # the coadd
            -                detResults = self.detectCoaddSources.runDetection(coadd, idFactory)
            +                detResults = self.detectCoaddSources.runDetection(coadd, idFactory, expId=expId)
                             self.detectCoaddSources.write(coadd, detResults, patchRef)
                     else:
                         patchRef.put(coadd, self.assembleCoadd.config.coaddName+"Coadd")
            

            Show
            price Paul Price added a comment - The error in coaddDriver is my fault from not adapting it to the new API following DM-12995 . I've fixed this on a ticket branch: price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-13396=) $ git sub-patch commit f99c95d48896431c128506e3890c3ac9b1ca84f3 (HEAD -> tickets/DM-13396, origin/tickets/DM-13396) Author: Paul Price <price@astro.princeton.edu> Date: Tue Jan 30 09:13:13 2018 -0500   coaddDriver: adapt to API change in detection DetectCoaddSourcesTask.runDetection now requires an exposure ID as a RNG seed.   diff --git a/python/lsst/pipe/drivers/coaddDriver.py b/python/lsst/pipe/drivers/coaddDriver.py index b9c8d4d..c7d8bd0 100644 --- a/python/lsst/pipe/drivers/coaddDriver.py +++ b/python/lsst/pipe/drivers/coaddDriver.py @@ -310,9 +310,10 @@ class CoaddDriverTask(BatchPoolTask): with self.logOperation("detection on {}".format(patchRef.dataId), catch=True): idFactory = self.detectCoaddSources.makeIdFactory(patchRef) + expId = int(patchRef.get(self.config.coaddName + "CoaddId")) # This includes background subtraction, so do it before writing # the coadd - detResults = self.detectCoaddSources.runDetection(coadd, idFactory) + detResults = self.detectCoaddSources.runDetection(coadd, idFactory, expId=expId) self.detectCoaddSources.write(coadd, detResults, patchRef) else: patchRef.put(coadd, self.assembleCoadd.config.coaddName+"Coadd")
            Hide
            yusra Yusra AlSayyad added a comment -

            Jim's edit appears to have done trick: 

            Show
            yusra Yusra AlSayyad added a comment - Jim's edit appears to have done trick: 
            Hide
            price Paul Price added a comment -

            I noticed that INTRP is now printing through to the coadd. Jim Bosch points out that this is desired, but we should remove them where they are assumed to mean bad. HSM appears to be the only place:

            price@pap-laptop:~/LSST/meas_extensions_shapeHSM (tickets/DM-13396=) $ git sub
            commit 27f6ff060995481f4c51b0b363ae21a250180c66
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Jan 30 13:19:04 2018 -0500
             
                remove policy file
                
                Not used any more, since the measurement framework was upgraded years ago.
             
             policy/hsmShape.paf | 22 ----------------------
             1 file changed, 22 deletions(-)
             
            commit eb15b086b19ac2d39bb1fdf06402293857674929 (HEAD -> tickets/DM-13396, origin/tickets/DM-13396)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Jan 30 13:23:52 2018 -0500
             
                remove INTRP from badMaskPlanes
                
                INTRP is now allowed to print through to the coadd (meaning that one of
                the inputs was interpolated), so it can appear in coadd sources that are
                otherwise fine.
             
             include/lsst/meas/extensions/shapeHSM/HsmMomentsControl.h | 1 -
             include/lsst/meas/extensions/shapeHSM/HsmShapeControl.h   | 2 --
             2 files changed, 3 deletions(-)
            

            Show
            price Paul Price added a comment - I noticed that INTRP is now printing through to the coadd. Jim Bosch points out that this is desired, but we should remove them where they are assumed to mean bad. HSM appears to be the only place: price@pap-laptop:~/LSST/meas_extensions_shapeHSM (tickets/DM-13396=) $ git sub commit 27f6ff060995481f4c51b0b363ae21a250180c66 Author: Paul Price <price@astro.princeton.edu> Date: Tue Jan 30 13:19:04 2018 -0500   remove policy file Not used any more, since the measurement framework was upgraded years ago.   policy/hsmShape.paf | 22 ---------------------- 1 file changed, 22 deletions(-)   commit eb15b086b19ac2d39bb1fdf06402293857674929 (HEAD -> tickets/DM-13396, origin/tickets/DM-13396) Author: Paul Price <price@astro.princeton.edu> Date: Tue Jan 30 13:23:52 2018 -0500   remove INTRP from badMaskPlanes INTRP is now allowed to print through to the coadd (meaning that one of the inputs was interpolated), so it can appear in coadd sources that are otherwise fine.   include/lsst/meas/extensions/shapeHSM/HsmMomentsControl.h | 1 - include/lsst/meas/extensions/shapeHSM/HsmShapeControl.h | 2 -- 2 files changed, 3 deletions(-)
            Hide
            jbosch Jim Bosch added a comment -

            I have reviewed and approve of Paul Price's (trivial) changes to pipe_drivers and meas_extensions_shapeHSM; no comments.

            Show
            jbosch Jim Bosch added a comment - I have reviewed and approve of Paul Price 's (trivial) changes to pipe_drivers and meas_extensions_shapeHSM; no comments.
            Hide
            jbosch Jim Bosch added a comment -

            Merged to master.

            Show
            jbosch Jim Bosch added a comment - Merged to master.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Paul Price
              Watchers:
              Jim Bosch, Paul Price, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.