Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13637

WarpCompare: Bad amps eat up temporal budget

    Details

    • Story Points:
      5
    • Sprint:
      DRP S18-3, DRP S18-4, DRP S18-5
    • Team:
      Data Release Production

      Description

      Overlapping bad amps eat up the temporalThreshold budget. 

      For example if a local region has 10 visits, and 3 of those visits are not included because of bad amps, then no candidates are clipped.  
      CLIPPED mask for /datasets/hsc/repo/rerun/private/yusra/RC/DM-13553+DM-134110: 9813, HSC-Z

      Seen since DM-12692: Bad amps can be seen in this epochCountImage:

      If any detections the warpDiffs are entirely covered with the badPixelMask, then don't contribute to the rolling epochCountImage or clip.

        Attachments

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment - - edited

            OK, I kicked off a few RC2 runs last night with a couple different configs:

            /datasets/hsc/repo/rerun/private/yusra/RC2/DM-13637/allBad005 (baseline change)

            /datasets/hsc/repo/rerun/private/yusra/RC2/DM-13637/Bad005

            /datasets/hsc/repo/rerun/private/yusra/RC2/DM-14075_v2 (pre ticket work)

             

            In all cases that I looked, the clipping is improved except on the edges of bad amps (see case study below for an example.

            Multiband run + colorAnalysis QA:

            WIDE (no impact on stellar locus):

             
            Still running:

            COSMOS:  (multiband still running because first couple attempts landed on a node with 67% memory usage while idle)
             
            DUD HSC-Z 3,3 9813 is being rerun on tiger now
             

             

            Show
            yusra Yusra AlSayyad added a comment - - edited OK, I kicked off a few RC2 runs last night with a couple different configs: /datasets/hsc/repo/rerun/private/yusra/RC2/ DM-13637 /allBad005 (baseline change) /datasets/hsc/repo/rerun/private/yusra/RC2/ DM-13637 /Bad005 /datasets/hsc/repo/rerun/private/yusra/RC2/ DM-14075 _v2 (pre ticket work)   In all cases that I looked, the clipping is improved except on the edges of bad amps (see case study below for an example. Multiband run + colorAnalysis QA: WIDE (no impact on stellar locus): ticket:  https://lsst-web.ncsa.illinois.edu/~yusra/artifact_rejection/Feb2018/DM-13637_all005/color/tract-9697/ w_2018_14: https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2018_14/pipe_analysis/qaPlots/color/tract-9697/   Still running: COSMOS:  (multiband still running because first couple attempts landed on a node with 67% memory usage while idle)   DUD HSC-Z 3,3 9813 is being rerun on tiger now    
            Hide
            yusra Yusra AlSayyad added a comment - - edited

             Note: Holes in CLIPPED regions filled in. This fixes the regions overwhelmed by defects, but NOT the regions overwhelmed by overlapping moustaches/ghosts. 

             

            Before After
               

             

             

            Show
            yusra Yusra AlSayyad added a comment - - edited  Note: Holes in CLIPPED regions filled in. This fixes the regions overwhelmed by defects, but NOT the regions overwhelmed by overlapping moustaches/ghosts.    Before After        
            Hide
            yusra Yusra AlSayyad added a comment -

            Case study of HSC-Z, tract=9813, patch=6,7, blinking the following two file shows improvement most everywhere,

            /datasets/hsc/repo/rerun/private/yusra/RC2/DM-13637/allBad005/deepCoadd/HSC-Z/9813/6,7.fits

            /datasets/hsc/repo/rerun/private/yusra/RC2/DM-14075_v2/deepCoadd/HSC-Z/9813/6,7.fits

            with the exception of an edges of a bad amp:  6,7-HSC-Z9813- 883-474.mov

            Outlier Epoch count image of the region showed in the video:

             

            And of the whole patch:

             

            Show
            yusra Yusra AlSayyad added a comment - Case study of HSC-Z, tract=9813, patch=6,7, blinking the following two file shows improvement most everywhere, /datasets/hsc/repo/rerun/private/yusra/RC2/ DM-13637 /allBad005/deepCoadd/HSC-Z/9813/6,7.fits /datasets/hsc/repo/rerun/private/yusra/RC2/ DM-14075 _v2/deepCoadd/HSC-Z/9813/6,7.fits with the exception of an edges of a bad amp:   6,7-HSC-Z9813- 883-474.mov Outlier Epoch count image of the region showed in the video:   And of the whole patch:  
            Hide
            yusra Yusra AlSayyad added a comment -

            Would like to merge tomorrow, but that depends on the results of finishing the QA diligence and Jenkins. 

            Show
            yusra Yusra AlSayyad added a comment - Would like to merge tomorrow, but that depends on the results of finishing the QA diligence and Jenkins. 
            Hide
            price Paul Price added a comment -

            I think this makes sense. Could you please expand the docs to explain the significance of the new parameters?

            Where is the temporalThreshold parameter defined? I don't see it anywhere in assembleCoadd.py, except that it's mentioned a few times in the docs for WarpCompareAssembleCoadd.

            Show
            price Paul Price added a comment - I think this makes sense. Could you please expand the docs to explain the significance of the new parameters? Where is the temporalThreshold parameter defined? I don't see it anywhere in assembleCoadd.py , except that it's mentioned a few times in the docs for WarpCompareAssembleCoadd .
            Hide
            yusra Yusra AlSayyad added a comment -

            Merged on Tuesday night. There is still room for improvement in this approach but its an improvement for this particular situation of overlapping defects.

            Statistically, minimal impact on the stellar locus RMS (e.g. gri Cmodel orig vs ticket) but there are a few more outliers. This could be due to which patches finished running in each of the two runs (gri Cmodel local deviations on sky: orig vs ticket).

            Full directories of plots:

            Show
            yusra Yusra AlSayyad added a comment - Merged on Tuesday night. There is still room for improvement in this approach but its an improvement for this particular situation of overlapping defects. Statistically, minimal impact on the stellar locus RMS (e.g. gri Cmodel orig vs ticket) but there are a few more outliers. This could be due to which patches finished running in each of the two runs (gri Cmodel local deviations on sky: orig vs ticket). Full directories of plots: ticket: https://lsst-web.ncsa.illinois.edu/~yusra/artifact_rejection/Feb2018/DM-13637_all005/color/tract-9813 w_2018_14 https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2018_14/pipe_analysis/qaPlots/color/tract-9813

              People

              • Assignee:
                yusra Yusra AlSayyad
                Reporter:
                yusra Yusra AlSayyad
                Reviewers:
                Paul Price
                Watchers:
                Paul Price, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel