Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19841

Look into unmasked defects on edge of bad amps leaking into coadds

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      2
    • Sprint:
      DRP S19-6b
    • Team:
      Data Release Production

      Description

      Dear Paul-san, Jim-san, and all,

      As we discussed before, I am reviewing the defect masks which are set
      in singleFrameDriver.py. I realized the singleFrameDriver.py produces
      very high values in area masked by defect list, especially for CCD=33
      that has a large defect area. Such high values around the CCD edges
      sometimes cause undesired streaks surviving on coadd images.

      See the attached figure (stripes_z_101092_101096.jpg) (left: coadd in
      a patch, middle: a warp in the same coordinates, right: another warp).
      In the warp figures, light green areas corresponds to the defects. We
      see that streaks on coadd presumably caused by the outlying values
      around the CCD edges due to defects mask. Particularly, the largest
      rectangular green is for the defect mask of CCD=33, which covers
      neighboring 2 channels. For reference, I also attached snapshots of
      CORR images for CCD=33 and 0, where CCD=33 has a large defect area on
      the entire left-handed side of the pixels, and CCD=0 has a 1-channel
      defects located between the healthy ch-1 and 3.
      The defect on CCD=0 seems to be somehow interpolated, but CCD=33's
      seems to be disturbed.

      These resultant (high) values are undesired themselves, but if the
      mask bit (BAD) works perfect, those would be eliminated on coadds, and
      that should be ok. However, the actual coadd shows some fraction of
      streaks.
      (CCD gaps are filled by NaN on warps, and those seem to be decently
      clipped by coadds).
      Note that CCD=33 is the most prominent CCD that causes the present
      problem, but the other CCDs are also set defect masks around the edges
      and middle of the CCD.

      I think we want to gently ignore pixels masked by defect list, rather
      than having such high and regular values, or reasonably interpolated
      or filled with more harmless value.
      Also mask bit should cover these defect area on coadds more carefully?

      I think this is a relatively highly demanded fix, but understand time
      is pressing.
      Do you have ideas of something we could try to track down this issue
      before going into the upcoming production?

      Hisanori

        Attachments

          Issue Links

            Activity

            Hide
            furusawa.hisanori Hisanori Furusawa added a comment -

            On Slack:

            Furusawa:
            And we would like to address the defect issue in coaddDriver (DM-19841) if possible, too. The best solution might be reprocessing isr by setting NaN for BAD pixels per Mineo-san, but it may be too late. Can we expand the mask for CCD=33 upon warping, instead?

            Yusra AlSayyad:
            Yes. Let’s just special case the defect for ccd=33 and apply it while reading in the calexp right before it gets warped. I can’t do this today, but will on Monday.

            Yusra AlSayyad:
            Were there any other ccds that you noticed this on? (編集済み)

            Hisanori Furusawa:
            @yusra Thank you very much for your attention. I have checked pixel counts around the amp edges for CORR files (visit=130334). The most distinct CCDs that may cause an immediate problem are ccd=33 and 9, the count levels of which are 23000, 3800adu, but usually ccd=9 is dropped in coadd stage and can be ignored. Besides, ccd=0, 43, 94 (count levels are 300-350adu) may contribute some, so we might want to treat them for safety, too, but they are two order of magnitude lower than ccd=33.

            Show
            furusawa.hisanori Hisanori Furusawa added a comment - On Slack: Furusawa: And we would like to address the defect issue in coaddDriver ( DM-19841 ) if possible, too. The best solution might be reprocessing isr by setting NaN for BAD pixels per Mineo-san, but it may be too late. Can we expand the mask for CCD=33 upon warping, instead? Yusra AlSayyad: Yes. Let’s just special case the defect for ccd=33 and apply it while reading in the calexp right before it gets warped. I can’t do this today, but will on Monday. Yusra AlSayyad: Were there any other ccds that you noticed this on? (編集済み) Hisanori Furusawa: @yusra Thank you very much for your attention. I have checked pixel counts around the amp edges for CORR files (visit=130334). The most distinct CCDs that may cause an immediate problem are ccd=33 and 9, the count levels of which are 23000, 3800adu, but usually ccd=9 is dropped in coadd stage and can be ignored. Besides, ccd=0, 43, 94 (count levels are 300-350adu) may contribute some, so we might want to treat them for safety, too, but they are two order of magnitude lower than ccd=33.
            Hide
            yusra Yusra AlSayyad added a comment -

            Alright. I masked the entire ccd=9 and added 2 pixels to ccd 33:

                33   1023      0      2   4176
                33        0   4156   2048     20
            

            to

                33   1023      0      4  4176
             33      0   4154   2048     22
            

            Jenkins running. https://ci.lsst.codes/job/stack-os-matrix/29956/

            And have a N=33 RC2 cosmos patch 5,3 running now as a test. Will hunt down a reviewer.

            Show
            yusra Yusra AlSayyad added a comment - Alright. I masked the entire ccd=9 and added 2 pixels to ccd 33: 33 1023 0 2 4176 33 0 4156 2048 20 to 33 1023 0 4 4176 33 0 4154 2048 22 Jenkins running. https://ci.lsst.codes/job/stack-os-matrix/29956/ And have a N=33 RC2 cosmos patch 5,3 running now as a test. Will hunt down a reviewer.
            Hide
            price Paul Price added a comment -

            Looks good. The current design of MakeCoaddTempExpTask makes for much more copied code than we would ordinarily like, but that's unavoidable, and this is only temporary.

            Will DM-20074 also update the defects?

            Show
            price Paul Price added a comment - Looks good. The current design of MakeCoaddTempExpTask makes for much more copied code than we would ordinarily like, but that's unavoidable, and this is only temporary. Will DM-20074 also update the defects?
            Hide
            yusra Yusra AlSayyad added a comment -

            Merged to master. Note that this override to mask ccd=33 is only automatic when using CoaddDriver. A plain makeCoaddTempExp.py from the commandline runs the original Task.

            Longterm fix (either update to the defect list or replace bad amps with nan) will go on DM-20090.

            Show
            yusra Yusra AlSayyad added a comment - Merged to master. Note that this override to mask ccd=33 is only automatic when using CoaddDriver. A plain makeCoaddTempExp.py from the commandline runs the original Task. Longterm fix (either update to the defect list or replace bad amps with nan) will go on DM-20090 .
            Hide
            furusawa.hisanori Hisanori Furusawa added a comment -

            To eliminate the streaks on coadd due to CCD=33 boudaries, I suggest to expand masks a bit further. The below is suggested change.

            --- /ana/products.7.7.1/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.1-hsc/python/lsst/obs/hsc/makeCoaddTempExp.py.org        2019-06-13 14:35:13.509210003 +0900
            +++ /ana/products.7.7.1/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.1-hsc/python/lsst/obs/hsc/makeCoaddTempExp.py    2019-06-13 19:35:28.303562117 +0900
            @@ -108,8 +108,20 @@ class SubaruMakeCoaddTempExpTask(MakeCoa
                     mask = exposure.mask
                     badPlane = mask.getPlaneBitMask('BAD')-        mask[1023:1027, 0:4176] |= badPlane
            -        mask[0:2048, 4154:4176] |= badPlane
            +        # expand mask by further 3 px
            +        mask[1023:1030, 0:4176] |= badPlane
            +        mask[0:2048, 4151:4176] |= badPlane
            +
            +        # mask another long edge
            +        mask[2022:2030, 0:4176] |= badPlane
            +
            +        # mask around mid bottom
            +        mask[1049:1057, 0:25] |= badPlane
            +        mask[1025:1057, 21:29] |= badPlane
            +
            +        # mask around right corner
            +        mask[1993:2001, 4035:4157] |= badPlane
            +        mask[1993:2027, 4030:4038] |= badPlane     def maskWholeCcd(self, exposure):
                     """Mask entire CCD (intended for use with ccd==9)

            Show
            furusawa.hisanori Hisanori Furusawa added a comment - To eliminate the streaks on coadd due to CCD=33 boudaries, I suggest to expand masks a bit further. The below is suggested change. --- /ana/products. 7.7 . 1 /stack/miniconda3- 4.5 . 12 -1172c30/Linux64/obs_subaru/ 7.7 . 1 -hsc/python/lsst/obs/hsc/makeCoaddTempExp.py.org 2019 - 06 - 13 14 : 35 : 13.509210003 + 0900 +++ /ana/products. 7.7 . 1 /stack/miniconda3- 4.5 . 12 -1172c30/Linux64/obs_subaru/ 7.7 . 1 -hsc/python/lsst/obs/hsc/makeCoaddTempExp.py 2019 - 06 - 13 19 : 35 : 28.303562117 + 0900 @@ - 108 , 8 + 108 , 20 @@ class SubaruMakeCoaddTempExpTask(MakeCoa mask = exposure.mask badPlane = mask.getPlaneBitMask( 'BAD' )- mask[ 1023 : 1027 , 0 : 4176 ] |= badPlane - mask[ 0 : 2048 , 4154 : 4176 ] |= badPlane + # expand mask by further 3 px + mask[ 1023 : 1030 , 0 : 4176 ] |= badPlane + mask[ 0 : 2048 , 4151 : 4176 ] |= badPlane + + # mask another long edge + mask[ 2022 : 2030 , 0 : 4176 ] |= badPlane + + # mask around mid bottom + mask[ 1049 : 1057 , 0 : 25 ] |= badPlane + mask[ 1025 : 1057 , 21 : 29 ] |= badPlane + + # mask around right corner + mask[ 1993 : 2001 , 4035 : 4157 ] |= badPlane + mask[ 1993 : 2027 , 4030 : 4038 ] |= badPlane def maskWholeCcd(self, exposure): "" "Mask entire CCD (intended for use with ccd== 9 )

              People

              • Assignee:
                yusra Yusra AlSayyad
                Reporter:
                yusra Yusra AlSayyad
                Reviewers:
                Paul Price
                Watchers:
                Christopher Waters, Hisanori Furusawa, Jim Bosch, Paul Price, Sogo Mineo, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: