Show
added a comment - BOT data:
measurePhotonTransferCurve.py /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11 --output /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11 --id detector=94 --visit-pairs 3019101200480,3019101200481 3019101300081,3019101300082 3019101300102,3019101300103 3019101200468,3019101200469 3019101200385,3019101200386 3019101200236,3019101200237 3019101300012,3019101300013 3019101200429,3019101200430 3019101300240,3019101300241 3019101200302,3019101200303 -c maxMeanSignal=100000 maskNameList=[BAD,NO_DATA,SUSPECT] --clobber-config
(the config options in the command above are passed in a config file via -C . The command is pasted here like this for reference to the repository and the data).
Plots with the bad amplifiers: PTC_DM25934_det94.pdf
Plots before DM-25934 : PTC_W28_det94.pdf
A lot of points when fitting the PTC are being rejected:
Before DM-25934 (W_28):
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C10: 2 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C11: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C12: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C13: 2 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C14: 2 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C15: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C16: 2 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C17: 2 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C07: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C06: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C05: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C04: 2 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C03: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C02: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C01: 3 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C00: 2 out of 10
After:
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C11: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C11 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C12: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C12 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C13: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C13 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C14: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C14 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C15: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C15 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C16: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C16 to BAD.
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C17: 2 out of 10
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C07: 3 out of 10
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C06: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C06 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C05: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C05 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C04: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C04 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C03: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C03 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C02: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C02 to BAD.
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C01: 10 out of 10
measurePhotonTransferCurve WARN:
SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C01 to BAD.
measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]
measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C00: 2 out of 10
Craig reports that The failing amps have a nan value in the meanVecOrig array that isn't present in the W28 code, and this is causing the mask array to be all bad. One of the amps which passes:
Initial, meanVecOrig C00 [ 191.1760555 1136.83632573 289.33204124 4852.14738495
575.43153566 852.88256794 19155.95146335 42638.27287383
87798.04994242 158683.6015625 ]
Initial, goodPoints C00 [False True True True True True True True True False]
Initial, mask C00 [False True True True True True True True True False]
In loop, counter = 1, 2 dropped points for amp C00
In loop, counter = 2, 2 dropped points for amp C00
One of the amps which fails:
Initial, meanVecOrig C01 [ 191.3205181 1136.98362194 289.46842014 4850.88752042
575.55600463 852.64695499 19154.27634537 42655.94247219
87883.20756922 nan]
Initial, goodPoints C01 [False False False False False False False False False False]
Initial, mask C01 [False False False False False False False False False False]
In loop, counter = 1, 10 dropped points for amp C01
In loop, counter = 2, 10 dropped points for amp C01
Crag: In the W28 code, the rawMean value for that highest point is 158683.6015625, but in the DM-25934 code it is nan. The nans are the cause of the problem:
Mu1, Mu2, Mu 158933.296875 158433.90625 158683.6015625
/lsstdata/user/staff/jhome/cslage/alternate_branches/cp_pipe/python/lsst/cp/pipe/astierCovPtcUtils.py:89: RuntimeWarning: invalid value encountered in double_scalars
cov1 = self.pCov[dy, dx]/nPix1-self.pMean[dy, dx]*self.pMean[-dy, -dx]/(nPix1*nPix1)
/lsstdata/user/staff/jhome/cslage/alternate_branches/cp_pipe/python/lsst/cp/pipe/astierCovPtcUtils.py:93: RuntimeWarning: invalid value encountered in double_scalars
cov2 = self.pCov[-dy, dx]/nPix2-self.pMean[-dy, dx]*self.pMean[dy, -dx]/(nPix2*nPix2)
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu 158933.296875 158433.90625 158683.6015625
Mu1, Mu2, Mu 158933.296875 158433.90625 158683.6015625
Mu1, Mu2, Mu 158933.296875 nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu nan nan nan
Mu1, Mu2, Mu 158933.296875 158433.90625 158683.6015625
The number of pixels that are being masked is only ~200.

BOT data:

`measurePhotonTransferCurve.py /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11 --output /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11 --id detector=94 --visit-pairs 3019101200480,3019101200481 3019101300081,3019101300082 3019101300102,3019101300103 3019101200468,3019101200469 3019101200385,3019101200386 3019101200236,3019101200237 3019101300012,3019101300013 3019101200429,3019101200430 3019101300240,3019101300241 3019101200302,3019101200303 -c maxMeanSignal=100000 maskNameList=[BAD,NO_DATA,SUSPECT] --clobber-config`

(the config options in the command above are passed in a config file via

-C. The command is pasted here like this for reference to the repository and the data).Plots with the bad amplifiers: PTC_DM25934_det94.pdf

^{}Plots before

~~DM-25934~~: PTC_W28_det94.pdf^{}A lot of points when fitting the PTC are being rejected:

Before

~~DM-25934~~(W_28):`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C10: 2 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C11: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C12: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C13: 2 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C14: 2 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C15: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C16: 2 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C17: 2 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C07: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C06: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C05: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C04: 2 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C03: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C02: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C01: 3 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C00: 2 out of 10`

After:

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C11: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C11 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C12: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C12 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C13: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C13 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C14: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C14 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C15: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C15 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C16: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C16 to BAD.`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C17: 2 out of 10`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C07: 3 out of 10`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C06: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C06 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C05: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C05 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C04: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C04 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C03: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C03 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C02: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C02 to BAD.`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C01: 10 out of 10`

`measurePhotonTransferCurve WARN:`

`SERIOUS: Not enough data points (0) compared to the number ofparameters of the PTC model(4). Setting C01 to BAD.`

`measurePhotonTransferCurve WARN: Found 1 zeros in array at elements [9]`

`measurePhotonTransferCurve INFO: Number of points discarded in PTC of amplifier C00: 2 out of 10`

Craig reports that The failing amps have a nan value in the meanVecOrig array that isn't present in the W28 code, and this is causing the mask array to be all bad. One of the amps which passes:

`Initial, meanVecOrig C00 [ 191.1760555 1136.83632573 289.33204124 4852.14738495`

`575.43153566 852.88256794 19155.95146335 42638.27287383`

`87798.04994242 158683.6015625 ]`

`Initial, goodPoints C00 [False True True True True True True True True False]`

`Initial, mask C00 [False True True True True True True True True False]`

`In loop, counter = 1, 2 dropped points for amp C00`

`In loop, counter = 2, 2 dropped points for amp C00`

One of the amps which fails:

`Initial, meanVecOrig C01 [ 191.3205181 1136.98362194 289.46842014 4850.88752042`

`575.55600463 852.64695499 19154.27634537 42655.94247219`

`87883.20756922 nan]`

`Initial, goodPoints C01 [False False False False False False False False False False]`

`Initial, mask C01 [False False False False False False False False False False]`

`In loop, counter = 1, 10 dropped points for amp C01`

`In loop, counter = 2, 10 dropped points for amp C01`

Crag: In the W28 code, the rawMean value for that highest point is 158683.6015625, but in the

~~DM-25934~~code it is nan. The nans are the cause of the problem:`Mu1, Mu2, Mu 158933.296875 158433.90625 158683.6015625`

`/lsstdata/user/staff/jhome/cslage/alternate_branches/cp_pipe/python/lsst/cp/pipe/astierCovPtcUtils.py:89: RuntimeWarning: invalid value encountered in double_scalars`

`cov1 = self.pCov[dy, dx]/nPix1-self.pMean[dy, dx]*self.pMean[-dy, -dx]/(nPix1*nPix1)`

`/lsstdata/user/staff/jhome/cslage/alternate_branches/cp_pipe/python/lsst/cp/pipe/astierCovPtcUtils.py:93: RuntimeWarning: invalid value encountered in double_scalars`

`cov2 = self.pCov[-dy, dx]/nPix2-self.pMean[-dy, dx]*self.pMean[dy, -dx]/(nPix2*nPix2)`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu 158933.296875 158433.90625 158683.6015625`

`Mu1, Mu2, Mu 158933.296875 158433.90625 158683.6015625`

`Mu1, Mu2, Mu 158933.296875 nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu nan nan nan`

`Mu1, Mu2, Mu 158933.296875 158433.90625 158683.6015625`

The number of pixels that are being masked is only ~200.