XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
2
• Sprint:
DRP S19-6b
• Team:
Data Release Production

#### Description

Dear Paul-san, Jim-san, and all,

As we discussed before, I am reviewing the defect masks which are set
in singleFrameDriver.py. I realized the singleFrameDriver.py produces
very high values in area masked by defect list, especially for CCD=33
that has a large defect area. Such high values around the CCD edges
sometimes cause undesired streaks surviving on coadd images.

See the attached figure (stripes_z_101092_101096.jpg) (left: coadd in
a patch, middle: a warp in the same coordinates, right: another warp).
In the warp figures, light green areas corresponds to the defects. We
see that streaks on coadd presumably caused by the outlying values
around the CCD edges due to defects mask. Particularly, the largest
rectangular green is for the defect mask of CCD=33, which covers
neighboring 2 channels. For reference, I also attached snapshots of
CORR images for CCD=33 and 0, where CCD=33 has a large defect area on
the entire left-handed side of the pixels, and CCD=0 has a 1-channel
defects located between the healthy ch-1 and 3.
The defect on CCD=0 seems to be somehow interpolated, but CCD=33's
seems to be disturbed.

These resultant (high) values are undesired themselves, but if the
that should be ok. However, the actual coadd shows some fraction of
streaks.
(CCD gaps are filled by NaN on warps, and those seem to be decently
Note that CCD=33 is the most prominent CCD that causes the present
problem, but the other CCDs are also set defect masks around the edges
and middle of the CCD.

I think we want to gently ignore pixels masked by defect list, rather
than having such high and regular values, or reasonably interpolated
or filled with more harmless value.
Also mask bit should cover these defect area on coadds more carefully?

I think this is a relatively highly demanded fix, but understand time
is pressing.
Do you have ideas of something we could try to track down this issue
before going into the upcoming production?

Hisanori

#### Activity

Hide
Hisanori Furusawa added a comment -

On Slack:

Furusawa:
And we would like to address the defect issue in coaddDriver (DM-19841) if possible, too. The best solution might be reprocessing isr by setting NaN for BAD pixels per Mineo-san, but it may be too late. Can we expand the mask for CCD=33 upon warping, instead?

Yes. Let’s just special case the defect for ccd=33 and apply it while reading in the calexp right before it gets warped. I can’t do this today, but will on Monday.

Were there any other ccds that you noticed this on? （編集済み）

Hisanori Furusawa:
@yusra Thank you very much for your attention. I have checked pixel counts around the amp edges for CORR files (visit=130334). The most distinct CCDs that may cause an immediate problem are ccd=33 and 9, the count levels of which are 23000, 3800adu, but usually ccd=9 is dropped in coadd stage and can be ignored. Besides, ccd=0, 43, 94 (count levels are 300-350adu) may contribute some, so we might want to treat them for safety, too, but they are two order of magnitude lower than ccd=33.

Show
Hide

Alright. I masked the entire ccd=9 and added 2 pixels to ccd 33:

  33 1023 0 2 4176  33 0 4156 2048 20 

to

  33 1023 0 4 4176  33 0 4154 2048 22 

Jenkins running. https://ci.lsst.codes/job/stack-os-matrix/29956/

And have a N=33 RC2 cosmos patch 5,3 running now as a test. Will hunt down a reviewer.

Show
Yusra AlSayyad added a comment - Alright. I masked the entire ccd=9 and added 2 pixels to ccd 33: 33 1023 0 2 4176 33 0 4156 2048 20 to 33 1023 0 4 4176 33 0 4154 2048 22 Jenkins running. https://ci.lsst.codes/job/stack-os-matrix/29956/ And have a N=33 RC2 cosmos patch 5,3 running now as a test. Will hunt down a reviewer.
Hide
Paul Price added a comment -

Looks good. The current design of MakeCoaddTempExpTask makes for much more copied code than we would ordinarily like, but that's unavoidable, and this is only temporary.

Will DM-20074 also update the defects?

Show
Paul Price added a comment - Looks good. The current design of MakeCoaddTempExpTask makes for much more copied code than we would ordinarily like, but that's unavoidable, and this is only temporary. Will DM-20074 also update the defects?
Hide

Merged to master. Note that this override to mask ccd=33 is only automatic when using CoaddDriver. A plain makeCoaddTempExp.py from the commandline runs the original Task.

Longterm fix (either update to the defect list or replace bad amps with nan) will go on DM-20090.

Show
Yusra AlSayyad added a comment - Merged to master. Note that this override to mask ccd=33 is only automatic when using CoaddDriver. A plain makeCoaddTempExp.py from the commandline runs the original Task. Longterm fix (either update to the defect list or replace bad amps with nan) will go on DM-20090 .
Hide
Hisanori Furusawa added a comment -

To eliminate the streaks on coadd due to CCD=33 boudaries, I suggest to expand masks a bit further. The below is suggested change.

 --- /ana/products.7.7.1/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.1-hsc/python/lsst/obs/hsc/makeCoaddTempExp.py.org 2019-06-13 14:35:13.509210003 +0900 +++ /ana/products.7.7.1/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.1-hsc/python/lsst/obs/hsc/makeCoaddTempExp.py 2019-06-13 19:35:28.303562117 +0900 @@ -108,8 +108,20 @@ class SubaruMakeCoaddTempExpTask(MakeCoa  mask = exposure.mask  badPlane = mask.getPlaneBitMask('BAD')- mask[1023:1027, 0:4176] |= badPlane - mask[0:2048, 4154:4176] |= badPlane + # expand mask by further 3 px + mask[1023:1030, 0:4176] |= badPlane + mask[0:2048, 4151:4176] |= badPlane + + # mask another long edge + mask[2022:2030, 0:4176] |= badPlane + + # mask around mid bottom + mask[1049:1057, 0:25] |= badPlane + mask[1025:1057, 21:29] |= badPlane + + # mask around right corner + mask[1993:2001, 4035:4157] |= badPlane + mask[1993:2027, 4030:4038] |= badPlane def maskWholeCcd(self, exposure):  """Mask entire CCD (intended for use with ccd==9)

Show

#### People

Assignee:
Reporter:
Reviewers:
Paul Price
Watchers:
Christopher Waters, Hisanori Furusawa, Jim Bosch, Paul Price, Sogo Mineo, Yusra AlSayyad