Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Sprint:DRP S19-6b
-
Team:Data Release Production
Description
Dear Paul-san, Jim-san, and all,
As we discussed before, I am reviewing the defect masks which are set
in singleFrameDriver.py. I realized the singleFrameDriver.py produces
very high values in area masked by defect list, especially for CCD=33
that has a large defect area. Such high values around the CCD edges
sometimes cause undesired streaks surviving on coadd images.
See the attached figure (stripes_z_101092_101096.jpg) (left: coadd in
a patch, middle: a warp in the same coordinates, right: another warp).
In the warp figures, light green areas corresponds to the defects. We
see that streaks on coadd presumably caused by the outlying values
around the CCD edges due to defects mask. Particularly, the largest
rectangular green is for the defect mask of CCD=33, which covers
neighboring 2 channels. For reference, I also attached snapshots of
CORR images for CCD=33 and 0, where CCD=33 has a large defect area on
the entire left-handed side of the pixels, and CCD=0 has a 1-channel
defects located between the healthy ch-1 and 3.
The defect on CCD=0 seems to be somehow interpolated, but CCD=33's
seems to be disturbed.
These resultant (high) values are undesired themselves, but if the
mask bit (BAD) works perfect, those would be eliminated on coadds, and
that should be ok. However, the actual coadd shows some fraction of
streaks.
(CCD gaps are filled by NaN on warps, and those seem to be decently
clipped by coadds).
Note that CCD=33 is the most prominent CCD that causes the present
problem, but the other CCDs are also set defect masks around the edges
and middle of the CCD.
I think we want to gently ignore pixels masked by defect list, rather
than having such high and regular values, or reasonably interpolated
or filled with more harmless value.
Also mask bit should cover these defect area on coadds more carefully?
I think this is a relatively highly demanded fix, but understand time
is pressing.
Do you have ideas of something we could try to track down this issue
before going into the upcoming production?
Hisanori
Attachments
Issue Links
Activity
Alright. I masked the entire ccd=9 and added 2 pixels to ccd 33:
33 1023 0 2 4176
|
33 0 4156 2048 20
|
to
33 1023 0 4 4176
|
33 0 4154 2048 22
|
Jenkins running. https://ci.lsst.codes/job/stack-os-matrix/29956/
And have a N=33 RC2 cosmos patch 5,3 running now as a test. Will hunt down a reviewer.
Looks good. The current design of MakeCoaddTempExpTask makes for much more copied code than we would ordinarily like, but that's unavoidable, and this is only temporary.
Will DM-20074 also update the defects?
Paul Price
added a comment - Looks good. The current design of MakeCoaddTempExpTask makes for much more copied code than we would ordinarily like, but that's unavoidable, and this is only temporary.
Will DM-20074 also update the defects? Merged to master. Note that this override to mask ccd=33 is only automatic when using CoaddDriver. A plain makeCoaddTempExp.py from the commandline runs the original Task.
Longterm fix (either update to the defect list or replace bad amps with nan) will go on DM-20090.
To eliminate the streaks on coadd due to CCD=33 boudaries, I suggest to expand masks a bit further. The below is suggested change.
--- /ana/products.7.7.1/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.1-hsc/python/lsst/obs/hsc/makeCoaddTempExp.py.org 2019-06-13 14:35:13.509210003 +0900 |
+++ /ana/products.7.7.1/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.1-hsc/python/lsst/obs/hsc/makeCoaddTempExp.py 2019-06-13 19:35:28.303562117 +0900 |
@@ -108,8 +108,20 @@ class SubaruMakeCoaddTempExpTask(MakeCoa |
mask = exposure.mask
|
badPlane = mask.getPlaneBitMask('BAD')- mask[1023:1027, 0:4176] |= badPlane |
- mask[0:2048, 4154:4176] |= badPlane |
+ # expand mask by further 3 px |
+ mask[1023:1030, 0:4176] |= badPlane |
+ mask[0:2048, 4151:4176] |= badPlane |
+
|
+ # mask another long edge |
+ mask[2022:2030, 0:4176] |= badPlane |
+
|
+ # mask around mid bottom
|
+ mask[1049:1057, 0:25] |= badPlane |
+ mask[1025:1057, 21:29] |= badPlane |
+
|
+ # mask around right corner
|
+ mask[1993:2001, 4035:4157] |= badPlane |
+ mask[1993:2027, 4030:4038] |= badPlane def maskWholeCcd(self, exposure): |
"""Mask entire CCD (intended for use with ccd==9) |
Hisanori Furusawa
added a comment - To eliminate the streaks on coadd due to CCD=33 boudaries, I suggest to expand masks a bit further. The below is suggested change.
--- /ana/products. 7.7 . 1 /stack/miniconda3- 4.5 . 12 -1172c30/Linux64/obs_subaru/ 7.7 . 1 -hsc/python/lsst/obs/hsc/makeCoaddTempExp.py.org 2019 - 06 - 13 14 : 35 : 13.509210003 + 0900
+++ /ana/products. 7.7 . 1 /stack/miniconda3- 4.5 . 12 -1172c30/Linux64/obs_subaru/ 7.7 . 1 -hsc/python/lsst/obs/hsc/makeCoaddTempExp.py 2019 - 06 - 13 19 : 35 : 28.303562117 + 0900
@@ - 108 , 8 + 108 , 20 @@ class SubaruMakeCoaddTempExpTask(MakeCoa
mask = exposure.mask
badPlane = mask.getPlaneBitMask( 'BAD' )- mask[ 1023 : 1027 , 0 : 4176 ] |= badPlane
- mask[ 0 : 2048 , 4154 : 4176 ] |= badPlane
+ # expand mask by further 3 px
+ mask[ 1023 : 1030 , 0 : 4176 ] |= badPlane
+ mask[ 0 : 2048 , 4151 : 4176 ] |= badPlane
+
+ # mask another long edge
+ mask[ 2022 : 2030 , 0 : 4176 ] |= badPlane
+
+ # mask around mid bottom
+ mask[ 1049 : 1057 , 0 : 25 ] |= badPlane
+ mask[ 1025 : 1057 , 21 : 29 ] |= badPlane
+
+ # mask around right corner
+ mask[ 1993 : 2001 , 4035 : 4157 ] |= badPlane
+ mask[ 1993 : 2027 , 4030 : 4038 ] |= badPlane def maskWholeCcd(self, exposure):
"" "Mask entire CCD (intended for use with ccd== 9 )
On Slack:
Furusawa:
And we would like to address the defect issue in coaddDriver (
DM-19841) if possible, too. The best solution might be reprocessing isr by setting NaN for BAD pixels per Mineo-san, but it may be too late. Can we expand the mask for CCD=33 upon warping, instead?Yusra AlSayyad:
Yes. Let’s just special case the defect for ccd=33 and apply it while reading in the calexp right before it gets warped. I can’t do this today, but will on Monday.
Yusra AlSayyad:
Were there any other ccds that you noticed this on? （編集済み）
Hisanori Furusawa:
@yusra Thank you very much for your attention. I have checked pixel counts around the amp edges for CORR files (visit=130334). The most distinct CCDs that may cause an immediate problem are ccd=33 and 9, the count levels of which are 23000, 3800adu, but usually ccd=9 is dropped in coadd stage and can be ignored. Besides, ccd=0, 43, 94 (count levels are 300-350adu) may contribute some, so we might want to treat them for safety, too, but they are two order of magnitude lower than ccd=33.