Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12692

Improve temporal threshold for CompareWarp

    Details

    • Story Points:
      2
    • Sprint:
      DRP F17-6, DRP S18-1, DRP S18-2
    • Team:
      Data Release Production

      Description

      An integer maxNumEpochs is too crude. The original fraction temporalThreshold of the number warps entering an image is too crude.

      Evaluate and test a fraction with a floor based on the local N_epochs.

        Attachments

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment - - edited

            Ideally the probability that we'll have completely coincident artifacts is a function of N, the local number of epochs. In practice the pointings are not independently distributed and the threshold is actually a function of the number of coincident pointings. Also, the closer the spacing between the epochs in time, the higher the probability that asteroids will overlap.  

             

            For a given local N we want a threshold of: 

            • N = 1, threshold* = 0 (no clipping) *Higher numbers clip more
            • N = 2, threshold = 0 (no clipping)
            • N = 3, threshold = 1 (have enough info to clip the one visit that's not like the others) 
            • N = 4, threshold = 1
            • N = 5, threshold = 2 
            • N = 33, threshold = 2
            • N = 80 local (92 total), with mustaches that overlap in 10: need >=6.
              •   HSC-G 9813/3,3
            • N=170 local. Again mostly the bad amps:
              • HSC-Z 9813/3,3 
            •  N=220 local (241 total), with bad backgrounds that overlap in 13, ghosts in upper left in 12)
              •  HSC-Y 9813/3,3.
            • Next.

            *

            Show
            yusra Yusra AlSayyad added a comment - - edited Ideally the probability that we'll have completely coincident artifacts is a function of N, the local number of epochs. In practice the pointings are not independently distributed and the threshold is actually a function of the number of coincident pointings . Also, the closer the spacing between the epochs in time, the higher the probability that asteroids will overlap.     For a given local N we want a threshold of:  N = 1, threshold* = 0 (no clipping) *Higher numbers clip more N = 2, threshold = 0 (no clipping) N = 3, threshold = 1 (have enough info to clip the one visit that's not like the others)  N = 4, threshold = 1 N = 5, threshold = 2  N = 33, threshold = 2 N = 80 local (92 total), with mustaches that overlap in 10: need >=6.    HSC-G 9813/3,3 N=170 local. Again mostly the bad amps: HSC-Z 9813/3,3   N=220 local (241 total), with bad backgrounds that overlap in 13, ghosts in upper left in 12)  HSC-Y 9813/3,3. Next. *
            Hide
            yusra Yusra AlSayyad added a comment - - edited

            We don't have time to incorporate number of coincident pointings from the metadata.  I'm still generating more examples of epochCount images for N>50, but I'm propose a functional form of a broken line as a function of just local N. Its parametrized by 3 config parameters: low N slope, high N slope and characteristic/turnover which I'm still calling (maxNumEpochs)=2. 

             

            maxFractionEpochsHigh=0.03

            Low end slope and characteristic parameters were already explored with the public HSC dataset. More testing needs to be done to select the best value for the high end slope appropriate for N >50. But I think its appropriate to merge the new configs, and then optimize the value later. 

            Show
            yusra Yusra AlSayyad added a comment - - edited We don't have time to incorporate number of coincident pointings from the metadata.  I'm still generating more examples of epochCount images for N>50, but I'm propose a functional form of a broken line as a function of just local N. Its parametrized by 3 config parameters: low N slope, high N slope and characteristic/turnover which I'm still calling (maxNumEpochs)=2.    maxFractionEpochsHigh=0.03 Low end slope and characteristic parameters were already explored with the public HSC dataset. More testing needs to be done to select the best value for the high end slope appropriate for N >50. But I think its appropriate to merge the new configs, and then optimize the value later. 
            Show
            yusra Yusra AlSayyad added a comment - - edited Sophie, do you have time to review? https://github.com/lsst/pipe_tasks/pull/180   Jenkins passing: https://ci.lsst.codes/job/stack-os-matrix/27405/ https://ci.lsst.codes/job/stack-os-matrix/27406/
            Hide
            yusra Yusra AlSayyad added a comment - - edited

            Before (LEFT) and After (RIGHT)  for HSC-Y, tract=9813, patch=3,3. 

            /tigress/HSC/HSC/rerun/alsayyad/cosmos_test_artifact_reject_dud/testNepochs_orig/deepCoadd/HSC-Y/9813/3,3* /tigress/HSC/HSC/rerun/alsayyad/cosmos_test_artifact_reject_dud/testNepochs/deepCoadd/HSC-Y/9813/3,3*

            Takes care of the satellite trails leaking through on the N~200 coadd. 

             

            Before (LEFT) and After (RIGHT)  for HSC-G, tract=9813, patch=3,3. 

            Minimal change on the N~60 coadd (good).  Extra asteroid clipped. 

             

             

            Before (LEFT) and After (RIGHT)  for HSC-Z, tract=9813, patch=3,3. 

            This ticket care of everything except the disaster of bad amps in the upper right hand corner eating up the maxNumEpochs budget. 

             

            Show
            yusra Yusra AlSayyad added a comment - - edited Before (LEFT) and After (RIGHT)  for HSC-Y, tract=9813, patch=3,3.  /tigress/HSC/HSC/rerun/alsayyad/cosmos_test_artifact_reject_dud/testNepochs_orig/deepCoadd/HSC-Y/9813/3,3* /tigress/HSC/HSC/rerun/alsayyad/cosmos_test_artifact_reject_dud/testNepochs/deepCoadd/HSC-Y/9813/3,3* Takes care of the satellite trails leaking through on the N~200 coadd.    Before (LEFT) and After (RIGHT)  for HSC-G, tract=9813, patch=3,3.  Minimal change on the N~60 coadd (good).  Extra asteroid clipped.      Before (LEFT) and After (RIGHT)  for HSC-Z, tract=9813, patch=3,3.  This ticket care of everything except the disaster of bad amps in the upper right hand corner eating up the maxNumEpochs budget.   
            Hide
            yusra Yusra AlSayyad added a comment -

            Thanks for the review.  Merged with maxNumFractionEpochs=0.03. 

            Show
            yusra Yusra AlSayyad added a comment - Thanks for the review.  Merged with maxNumFractionEpochs=0.03. 

              People

              • Assignee:
                yusra Yusra AlSayyad
                Reporter:
                yusra Yusra AlSayyad
                Reviewers:
                Sophie Reed
                Watchers:
                Paul Price, Sophie Reed, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel