Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_tasks
-
Labels:
-
Story Points:5
-
Epic Link:
-
Sprint:DRP S18-3, DRP S18-4, DRP S18-5
-
Team:Data Release Production
Description
Overlapping bad amps eat up the temporalThreshold budget.
For example if a local region has 10 visits, and 3 of those visits are not included because of bad amps, then no candidates are clipped.
CLIPPED mask for /datasets/hsc/repo/rerun/private/yusra/RC/DM-13553+DM-134110: 9813, HSC-Z
Seen since DM-12692: Bad amps can be seen in this epochCountImage:
If any detections the warpDiffs are entirely covered with the badPixelMask, then don't contribute to the rolling epochCountImage or clip.
Attachments
Issue Links
- is triggered by
-
- Done
Activity
Note: Holes in CLIPPED regions filled in. This fixes the regions overwhelmed by defects, but NOT the regions overwhelmed by overlapping moustaches/ghosts.
| Before | After |
|---|---|
 |
 |
 |
 |
 |
 |
Case study of HSC-Z, tract=9813, patch=6,7, blinking the following two file shows improvement most everywhere,
/datasets/hsc/repo/rerun/private/yusra/RC2/DM-13637/allBad005/deepCoadd/HSC-Z/9813/6,7.fits
/datasets/hsc/repo/rerun/private/yusra/RC2/DM-14075_v2/deepCoadd/HSC-Z/9813/6,7.fits
with the exception of an edges of a bad amp: 6,7-HSC-Z9813- 883-474.mov
Outlier Epoch count image of the region showed in the video:
And of the whole patch:
Would like to merge tomorrow, but that depends on the results of finishing the QA diligence and Jenkins.
I think this makes sense. Could you please expand the docs to explain the significance of the new parameters?
Where is the temporalThreshold parameter defined? I don't see it anywhere in assembleCoadd.py, except that it's mentioned a few times in the docs for WarpCompareAssembleCoadd.
Paul Price
added a comment - I think this makes sense. Could you please expand the docs to explain the significance of the new parameters?
Where is the temporalThreshold parameter defined? I don't see it anywhere in assembleCoadd.py , except that it's mentioned a few times in the docs for WarpCompareAssembleCoadd . Merged on Tuesday night. There is still room for improvement in this approach but its an improvement for this particular situation of overlapping defects.
Statistically, minimal impact on the stellar locus RMS (e.g. gri Cmodel orig vs ticket) but there are a few more outliers. This could be due to which patches finished running in each of the two runs (gri Cmodel local deviations on sky: orig vs ticket).
Full directories of plots:
OK, I kicked off a few RC2 runs last night with a couple different configs:
/datasets/hsc/repo/rerun/private/yusra/RC2/
DM-13637/allBad005 (baseline change)/datasets/hsc/repo/rerun/private/yusra/RC2/
DM-13637/Bad005/datasets/hsc/repo/rerun/private/yusra/RC2/
DM-14075_v2 (pre ticket work)In all cases that I looked, the clipping is improved except on the edges of bad amps (see case study below for an example.
Multiband run + colorAnalysis QA:
WIDE (no impact on stellar locus):
Still running:
COSMOS: (multiband still running because first couple attempts landed on a node with 67% memory usage while idle)
DUD HSC-Z 3,3 9813 is being rerun on tiger now