# FULLCOVARIANCE in PTC task is rejecting more points than it should for some BOT data detectors

XMLWordPrintable

#### Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
3
• Team:
Data Release Production

#### Description

When running (w_45)

 measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=183 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c ptcFitType=FULLCOVARIANCE doPhotodiode=False maxIterFullFitCovariancesAstier=10 sigmaClipFullFitCovariancesAstier=15 --clobber-config --clobber-version -j 1 

Using the EXPAPPROXIMATION model, the data seems fine:

#### Attachments

90 kB
94 kB
94 kB
4. Gain_Differences_GT_5Pct_12Nov20.pdf
77 kB
5. Gain_Histograms_12673_NewCode_12Nov20.pdf
18 kB
6. Gain_Summary_12673_FullCov_12Nov20.pdf
50 kB
7. Gain_Summary_12673_NewCode_12Nov20.pdf
48 kB
8. image-2020-11-06-13-51-39-148.png
357 kB
9. image-2020-11-06-13-53-13-765.png
514 kB
10. PTC_det183_80K_S10.pdf
227 kB
11. PTC_det183_80K_S20.pdf
224 kB
228 kB
13. PTC_det94.pdf
222 kB
14. PTC_Eotest_Gains_12673_FullCov_12Nov20.pdf
62 kB
15. PTC_Eotest_Gains_12673_NewCode_12Nov20.pdf
64 kB
16. Rejection.png
13 kB
17. screenshot-1.png
320 kB
18. screenshot-2.png
440 kB
19. screenshot-3.png
845 kB
20. screenshot-4.png
836 kB

#### Activity

Hide
Christopher Waters added a comment -

I'm concerned that the PTC code is becoming increasingly difficult to understand. I'm hopeful that the gen3 rewrite will help somewhat.
I'd also suggest rebasing the existing commits into a simpler set that clearly define what changes have been made. It might also be good to move the travis/lint change to either the beginning or end of the commit chain so the PTC commits are together.

Show
Christopher Waters added a comment - I'm concerned that the PTC code is becoming increasingly difficult to understand. I'm hopeful that the gen3 rewrite will help somewhat. I'd also suggest rebasing the existing commits into a simpler set that clearly define what changes have been made. It might also be good to move the travis/lint change to either the beginning or end of the commit chain so the PTC commits are together.
Hide
Craig Lage added a comment -

Chris,  I share your concerns, but I think functionality has to take precedence over simplicity.  There are over 3000 amplifiers in the focal plane, and each one is a little bit different.  I certainly applaud any efforts to simplify the code, but the fact is that it has to return sensible results for the entire focal plane, which it doesn't do yet.  I think we need to keep fixing the things that fail and get it all working before we try to simplify it.

Show
Craig Lage added a comment - Chris,  I share your concerns, but I think functionality has to take precedence over simplicity.  There are over 3000 amplifiers in the focal plane, and each one is a little bit different.  I certainly applaud any efforts to simplify the code, but the fact is that it has to return sensible results for the entire focal plane, which it doesn't do yet.  I think we need to keep fixing the things that fail and get it all working before we try to simplify it.
Hide
Christopher Waters added a comment -

I think we agree.  Once we have a fully functioning PTC, we'll have a defined set of inputs and a known target output, and can then refactor the code to be more maintainable.

Show
Christopher Waters added a comment - I think we agree.  Once we have a fully functioning PTC, we'll have a defined set of inputs and a known target output, and can then refactor the code to be more maintainable.
Hide
Craig Lage added a comment -

I was able to get good results on basically the whole focal plane with FULLCOVARIANCE.  All CCDs passed and only 3 amps fell out, the same three that failed with EXPAPPROX, including the two known dead amps.  These results and plots are in /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_FullCov_12673A  . Below is a list of proposed changes.  Some of these I think should go in, but some others we need to discuss:

(1) Most of the failures were caused by the following error.  We were already testing for np.sum(w) == 0, which is basically the number of good pixels in the flat pair, at this step: https://github.com/lsst/cp_pipe/blob/f4cdebacd2b778bffd6c5ba5ba4fb438b4052ce2/python/lsst/cp/pipe/ptc.py#L683

But it turns out that if the number of good pixels is small (less than a few hundred or maybe a few thousand), then the FFT routine that calculates the covariances fails because some of the nPix values come out zero.  So I changed the np.sum(w) test to be np.sum(w) < 10000.  This number could perhaps be smaller, but if a flat pair has less than 10,000 good pixels, we probably don't want to include it anyway.  This fixed most of the failures.

(2) I realized when we put in limits for the EXPAPPROX fit, we put in +/-100 for the 3rd parameter, because I thought this was the noise.  But it turns out this is the noise^2.  Since the noise can be as high as ~40, I changed this to +/-2000.

(3) One of the CCDs was still failing in the wres routine at this step: https://github.com/lsst/cp_pipe/blob/f4cdebacd2b778bffd6c5ba5ba4fb438b4052ce2/python/lsst/cp/pipe/astierCovPtcFit.py#L528.  The failure was caused by self.maskMu being all NaNs.  I put in a try/except, as follows.  This clearly isn't a long-term fix, but fixed the problem.  We need to understand how it gets to be all NaNs.

 try:  maskedWeightedRes = weightedRes[self.maskMu]         except:              maskedWeightedRes = weightedRes * 0.0

(4) Many of the single amp failures were caused by the code just rejecting too many points in the iterative outlier rejection routine.  I think this routine doesn't make sense.  Once it has rejected a point, that point can never be recovered.  What happens is illustrated in the following sketch .  The first iteration, in red, rejects Pt 6 (correctly), but also rejects Pt 4 and Pt 5.  The second iteration is in green, but Pt 4 and Pt 5, which are good fits, have already been rejected and can't be recovered.  So I propose changing it so that points can be recovered if they are good fits in subsequent iterations.  We need to discuss this.

.

Show
Craig Lage added a comment - I was able to get good results on basically the whole focal plane with FULLCOVARIANCE.  All CCDs passed and only 3 amps fell out, the same three that failed with EXPAPPROX, including the two known dead amps.  These results and plots are in /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_FullCov_12673A  . Below is a list of proposed changes.  Some of these I think should go in, but some others we need to discuss: (1) Most of the failures were caused by the following error.  We were already testing for np.sum(w) == 0, which is basically the number of good pixels in the flat pair, at this step: https://github.com/lsst/cp_pipe/blob/f4cdebacd2b778bffd6c5ba5ba4fb438b4052ce2/python/lsst/cp/pipe/ptc.py#L683 But it turns out that if the number of good pixels is small (less than a few hundred or maybe a few thousand), then the FFT routine that calculates the covariances fails because some of the nPix values come out zero.  So I changed the np.sum(w) test to be np.sum(w) < 10000.  This number could perhaps be smaller, but if a flat pair has less than 10,000 good pixels, we probably don't want to include it anyway.  This fixed most of the failures. (2) I realized when we put in limits for the EXPAPPROX fit, we put in +/-100 for the 3rd parameter, because I thought this was the noise.  But it turns out this is the noise^2.  Since the noise can be as high as ~40, I changed this to +/-2000. (3) One of the CCDs was still failing in the wres routine at this step: https://github.com/lsst/cp_pipe/blob/f4cdebacd2b778bffd6c5ba5ba4fb438b4052ce2/python/lsst/cp/pipe/astierCovPtcFit.py#L528.   The failure was caused by self.maskMu being all NaNs.  I put in a try/except, as follows.  This clearly isn't a long-term fix, but fixed the problem.  We need to understand how it gets to be all NaNs. try : maskedWeightedRes = weightedRes[self.maskMu]        except:            maskedWeightedRes = weightedRes * 0.0 (4) Many of the single amp failures were caused by the code just rejecting too many points in the iterative outlier rejection routine.  I think this routine doesn't make sense.  Once it has rejected a point, that point can never be recovered.  What happens is illustrated in the following sketch .  The first iteration, in red, rejects Pt 6 (correctly), but also rejects Pt 4 and Pt 5.  The second iteration is in green, but Pt 4 and Pt 5, which are good fits, have already been rejected and can't be recovered.  So I propose changing it so that points can be recovered if they are good fits in subsequent iterations.  We need to discuss this. .
Hide
Craig Lage added a comment -

Thinking more about this, we need to understand why we have flat pairs with zero or a small number of good pixels.  I don't think any of the amplifiers are so defective to justify this, so it seems that something must have gone wrong in the defect finding routine.  I'm going to try to dig into this today.

Show
Craig Lage added a comment - Thinking more about this, we need to understand why we have flat pairs with zero or a small number of good pixels.  I don't think any of the amplifiers are so defective to justify this, so it seems that something must have gone wrong in the defect finding routine.  I'm going to try to dig into this today.

#### People

Assignee:
Andrés Alejandro Plazas Malagón
Reporter:
Andrés Alejandro Plazas Malagón
Reviewers:
Christopher Waters
Watchers:
Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage