# Recent tests of ptc.py show strange results

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Data Release Production
• Urgent?:
No

#### Description

I've been running ptc.py on the BOT data for some time with good results.  Recently, I've been working with the defect calibration files to mask out the edge pixels.  On 17-Jul, I ran ptc.py on the BOT data for detector 94.  This was with

 w_2020_28

You can see the python script that ran that at

 /project/cslage/BOT_lspdev/standalone/Run_PTC_Edge_15Jul20.py

and the output repo is at

 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11

Everything looked completely normal, and the gains agreed better with the Eotest Fe55 results.  So I set up to run all of the CCDs that way.  This took a while to get everything running, but it is now finished with all 81 CCDs.  The python script is at

 /project/cslage/BOT_lspdev/standalone/Run_PTC_All_29Jul20.py

and the output repo is at

 /project/shared/BOT/rerun/cslage/PTC_6790D_All

This ran with

 w_2020_30

The

 PTC_det94.pdf

plot looks OK at first glance, but the values are all messed up.  The gain comes out ~1.8 instead of ~1.0.  The PTC curve, instead of turning over at ~100,000 ADU like it should, turns over at ~60,000 ADU. I'm not sure what went wrong.  I don't know if it is something I did or if the code has changed.  Note that both the good run above and the bad run are after the refactoring of ptc.py

#### Attachments

1. pklcomp-DM-26166.py
1 kB
2. PostISR_Images_30Jul0.pdf
29 kB
3. PTC_det94-DM-26166_2020AUG05.pdf
86 kB

#### Activity

Hide
Craig Lage added a comment -

I compared the postISRCCD data of these two runs, and you can see them in the attached plots.  The data looks the same in the plots, but somehow the mean of the more recent(bad) run is about 60% of the earlier (good) run.  So this explains the problem.  Probably there is nothing wrong in ptc.py.  Whatever went wrong was in the ISR.  I'll try to identify what happened, but may run out of time before I leave on vacation.  In any case, I think the ball is in my court at this point, unless there are suggestions of what might have gone wrong in the ISR.

Show
Craig Lage added a comment - I compared the postISRCCD data of these two runs, and you can see them in the attached plots.  The data looks the same in the plots, but somehow the mean of the more recent(bad) run is about 60% of the earlier (good) run.  So this explains the problem.  Probably there is nothing wrong in ptc.py.  Whatever went wrong was in the ISR.  I'll try to identify what happened, but may run out of time before I leave on vacation.  In any case, I think the ball is in my court at this point, unless there are suggestions of what might have gone wrong in the ISR.
Hide
Christopher Waters added a comment - - edited

The N checks I looked at before having to leave for the day:

• config differences:
• sdiff -w 200 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge4_10_R22S11/config/runIsr.py /project/shared/BOT/rerun/cslage/PTC_6790D_All/config/runIsr.py | less
• Trivial rearrangement of imports
• sdiff -w 200 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge4_10_R22S11/runIsr_metadata/3019101300211-SDSSi~ND_OD1.0/R22/runIsrMetadata_3019101300211-SDSSi~ND_OD1.0-R22-S11-det094.yaml /project/shared/BOT/rerun/cslage/PTC_6790D_All/runIsr_metadata/3019101300211-SDSSi~ND_OD1.0/R22/runIsrMetadata_3019101300211-SDSSi~ND_OD1.0-R22-S11-det094.yaml
• Differences in times/process times/context switches.  No difference in overscan/amplifier statistics.
• diff code:
• git diff e98722466eb4e9c04785687313a8e42416e3940b fcb36e1abca3cbdebaa85688f1e2e56209d32726
• The only difference is an update in some daf_butler imports, which should be completely independent of gen2 processing.
• measurePTC config differences:
• sdiff -w 140 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11/config/measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_6790D_All/config/measurePhotonTransferCurve.py
• The E2V_6790D_Gain_Edge7_R22S11 includes the "DETECTED" mask plane to exclude.
• diff code:
• git diff c53c7b65681f94c8d573b0b28f92ac5b0b7fc46b 20dd8cc83f8785efb327b1db8653542def10ce9a
• Significant change, DM-26067 merged.
• Check ptc outputs:
• pklcomp-DM-26166.py
• Differences in many of the parameters, including visits used, rawMeans, and gains.

So this must be a consequence of DM-26067, so @plazas will need to look deeper.

Show
Christopher Waters added a comment - - edited The N checks I looked at before having to leave for the day: config differences: sdiff -w 200 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge4_10_R22S11/config/runIsr.py /project/shared/BOT/rerun/cslage/PTC_6790D_All/config/runIsr.py | less Trivial rearrangement of imports runIsr metadata differences: sdiff -w 200 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge4_10_R22S11/runIsr_metadata/3019101300211-SDSSi~ND_OD1.0/R22/runIsrMetadata_3019101300211-SDSSi~ND_OD1.0-R22-S11-det094.yaml /project/shared/BOT/rerun/cslage/PTC_6790D_All/runIsr_metadata/3019101300211-SDSSi~ND_OD1.0/R22/runIsrMetadata_3019101300211-SDSSi~ND_OD1.0-R22-S11-det094.yaml Differences in times/process times/context switches.  No difference in overscan/amplifier statistics. diff code: git diff e98722466eb4e9c04785687313a8e42416e3940b fcb36e1abca3cbdebaa85688f1e2e56209d32726 The only difference is an update in some daf_butler imports, which should be completely independent of gen2 processing. measurePTC config differences: sdiff -w 140 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11/config/measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_6790D_All/config/measurePhotonTransferCurve.py The E2V_6790D_Gain_Edge7_R22S11 includes the "DETECTED" mask plane to exclude. diff code: git diff c53c7b65681f94c8d573b0b28f92ac5b0b7fc46b 20dd8cc83f8785efb327b1db8653542def10ce9a Significant change, DM-26067 merged. Check ptc outputs: pklcomp- DM-26166 .py Differences in many of the parameters, including visits used, rawMeans, and gains. So this must be a consequence of DM-26067 , so @plazas will need to look deeper.
Hide
Merlin Fisher-Levine added a comment -

Intrigued by the above, I decided to poke to. Findings so far: running minimal ISR (no bias or anything, just overscan and assembly) gives identical means between w_28 and w_30.

Show
Merlin Fisher-Levine added a comment - Intrigued by the above, I decided to poke to. Findings so far: running minimal ISR (no bias or anything, just overscan and assembly) gives identical means between w_28 and w_30.
Hide
Merlin Fisher-Levine added a comment -

I suspected it might lie in some weird afw stats thing + DM-25934 but that seems innocent. I ran

 runIsr.py /project/shared/BOT/ --output /home/mfl/scratch --id detector=94 expId=3019101300315 -c isr.doBias=False isr.doDark=False isr.doFlat=False isr.doDefect=False isr.doCrosstalk=False 

to make a minimally reduced postISRCCD and then

 measurePhotonTransferCurve.py /project/shared/BOT/ --output /home/mfl/scratch --id detector=94 --visit-pairs 3019101300315,3019101300315 

and confirmed that the means (and covDiffAstier) were identical before/after that merge (specifically on master vs the checkout just before that merge).

Note that doesn't involve a bias subtraction, but certainly the stats part is off the hook. Bed now though I'm afraid.

Show
Merlin Fisher-Levine added a comment - I suspected it might lie in some weird afw stats thing + DM-25934 but that seems innocent. I ran runIsr.py /project/shared/BOT/ --output /home/mfl/scratch --id detector= 94 expId= 3019101300315 -c isr.doBias=False isr.doDark=False isr.doFlat=False isr.doDefect=False isr.doCrosstalk=False to make a minimally reduced postISRCCD and then measurePhotonTransferCurve.py /project/shared/BOT/ --output /home/mfl/scratch --id detector= 94 --visit-pairs 3019101300315 , 3019101300315 and confirmed that the means (and covDiffAstier ) were identical before/after that merge (specifically on master vs the checkout just before that merge). Note that doesn't involve a bias subtraction, but certainly the stats part is off the hook. Bed now though I'm afraid.
Hide
Andrés Alejandro Plazas Malagón added a comment - - edited

I used

 /project/cslage/BOT_lspdev/standalone/Run_PTC_Edge_15Jul20.py 

with a few modifications to ingest the defects: I added

  DEFECT_DIR = RERUN_DIR + '/calibrations/LSSTCam/defects'  cmd = 'ingestCuratedCalibs.py %s %s --calib %s --config clobber=True --ignore-ingested'%(DATA_DIR, DEFECT_DIR, CALIB_DIR)  runCommand(cmd)

to ingest the defects that were generated, and then I had to modify the calibration registry by hand to change the start validity date. Once the defects were ingested with an appropiate start date, I ran Craig's script, and got some more reasonable gains (closer to 1 instead of to 1.8) and turning off at about ~100k ADU: PTC_det94-DM-26166_2020AUG05.pdf

The script I ran is in

  /home/plazas/code_andres/Run_PTC_Edge_15Jul20_ANDRES.py

(basically the same as Craig's, but with the modification above), and the command was

  python Run_PTC_Edge_15Jul20_ANDRES.py --dir /project/plazas/DM-26166-cuatro --run 6790D --raft R22 --sensor S11 --num 70

. This was in

  w_2020_31

with the DM-25711 version of cp_pipe. I'll re-run when DM-25711 has been merged, to confirm the results.

Show
Andrés Alejandro Plazas Malagón added a comment - - edited I used /project/cslage/BOT_lspdev/standalone/Run_PTC_Edge_15Jul20.py with a few modifications to ingest the defects: I added DEFECT_DIR = RERUN_DIR + '/calibrations/LSSTCam/defects' cmd = 'ingestCuratedCalibs.py %s %s --calib %s --config clobber=True --ignore-ingested' % (DATA_DIR, DEFECT_DIR, CALIB_DIR) runCommand(cmd) to ingest the defects that were generated, and then I had to modify the calibration registry by hand to change the start validity date. Once the defects were ingested with an appropiate start date, I ran Craig's script, and got some more reasonable gains (closer to 1 instead of to 1.8) and turning off at about ~100k ADU: PTC_det94-DM-26166_2020AUG05.pdf The script I ran is in /home/plazas/code_andres/Run_PTC_Edge_15Jul20_ANDRES.py (basically the same as Craig's, but with the modification above), and the command was python Run_PTC_Edge_15Jul20_ANDRES.py --dir /project/plazas/DM-26166-cuatro --run 6790D --raft R22 --sensor S11 --num 70 . This was in w_2020_31 with the DM-25711 version of cp_pipe. I'll re-run when DM-25711 has been merged, to confirm the results.
Hide
Craig Lage added a comment -

I now understand how to fix this, but I don't understand why it started happening.  The postISRCCD mean shift is caused by the ISR performing flat fielding.  If I set isr.doFlat=False in the runIsr.py command, then the ISR means and the PTC gains go back where they should be.  Before ptc.py was refactored, doFlat was automatically set to False.  But even after the refactor, I got good results without specifying doFlat=False.  At some point something changed and runIsr.py started doing flat fielding.  I don't know what changed, but my problem is fixed if I just set isr.doFLat=False.  I've run one CCD this way, and am now running the rest. So I guess this ticket can be closed.  With the refactoring of ptc.py, how do we prevent in the future that someone runs ptc.py with postISRCCD data that has been flat fielded? Is this necessary, or do we just rely on people to know what they are doing?

Show
Craig Lage added a comment - I now understand how to fix this, but I don't understand why it started happening.  The postISRCCD mean shift is caused by the ISR performing flat fielding.  If I set isr.doFlat=False in the runIsr.py command, then the ISR means and the PTC gains go back where they should be.  Before ptc.py was refactored, doFlat was automatically set to False.  But even after the refactor, I got good results without specifying doFlat=False.  At some point something changed and runIsr.py started doing flat fielding.  I don't know what changed, but my problem is fixed if I just set isr.doFLat=False.  I've run one CCD this way, and am now running the rest. So I guess this ticket can be closed.  With the refactoring of ptc.py, how do we prevent in the future that someone runs ptc.py with postISRCCD data that has been flat fielded? Is this necessary, or do we just rely on people to know what they are doing?
Hide
Andrés Alejandro Plazas Malagón added a comment -

> With the refactoring of ptc.py, how do we prevent in the future that someone runs ptc.py with postISRCCD data that has been flat fielded? Is this necessary, or do we just rely on people to know what they are doing?

Thanks, Craig. This is a good question. After discussing a bit with Chris, the PTC task in gen3 should probably take as input the configuration output of the ISR task and validate that it matches its own options (and raise otherwise).

Show
Andrés Alejandro Plazas Malagón added a comment - > With the refactoring of ptc.py, how do we prevent in the future that someone runs ptc.py with postISRCCD data that has been flat fielded? Is this necessary, or do we just rely on people to know what they are doing? Thanks, Craig. This is a good question. After discussing a bit with Chris, the PTC task in gen3 should probably take as input the configuration output of the ISR task and validate that it matches its own options (and raise otherwise).

#### People

Assignee:
Andrés Alejandro Plazas Malagón
Reporter:
Craig Lage
Reviewers:
Christopher Waters, Merlin Fisher-Levine
Watchers:
Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine