Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26166

Recent tests of ptc.py show strange results

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
      None

      Description

      I've been running ptc.py on the BOT data for some time with good results.  Recently, I've been working with the defect calibration files to mask out the edge pixels.  On 17-Jul, I ran ptc.py on the BOT data for detector 94.  This was with 

      w_2020_28

        You can see the python script that ran that at 

      /project/cslage/BOT_lspdev/standalone/Run_PTC_Edge_15Jul20.py

      and the output repo is at

      /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11

      Everything looked completely normal, and the gains agreed better with the Eotest Fe55 results.  So I set up to run all of the CCDs that way.  This took a while to get everything running, but it is now finished with all 81 CCDs.  The python script is at 

      /project/cslage/BOT_lspdev/standalone/Run_PTC_All_29Jul20.py

      and the output repo is at 

      /project/shared/BOT/rerun/cslage/PTC_6790D_All

      This ran with

      w_2020_30

      The

      PTC_det94.pdf

      plot looks OK at first glance, but the values are all messed up.  The gain comes out ~1.8 instead of ~1.0.  The PTC curve, instead of turning over at ~100,000 ADU like it should, turns over at ~60,000 ADU. I'm not sure what went wrong.  I don't know if it is something I did or if the code has changed.  Note that both the good run above and the bad run are after the refactoring of ptc.py

        Attachments

          Issue Links

            Activity

            Hide
            cslage Craig Lage added a comment -

            I compared the postISRCCD data of these two runs, and you can see them in the attached plots.  The data looks the same in the plots, but somehow the mean of the more recent(bad) run is about 60% of the earlier (good) run.  So this explains the problem.  Probably there is nothing wrong in ptc.py.  Whatever went wrong was in the ISR.  I'll try to identify what happened, but may run out of time before I leave on vacation.  In any case, I think the ball is in my court at this point, unless there are suggestions of what might have gone wrong in the ISR.

            Show
            cslage Craig Lage added a comment - I compared the postISRCCD data of these two runs, and you can see them in the attached plots.  The data looks the same in the plots, but somehow the mean of the more recent(bad) run is about 60% of the earlier (good) run.  So this explains the problem.  Probably there is nothing wrong in ptc.py.  Whatever went wrong was in the ISR.  I'll try to identify what happened, but may run out of time before I leave on vacation.  In any case, I think the ball is in my court at this point, unless there are suggestions of what might have gone wrong in the ISR.
            Hide
            czw Christopher Waters added a comment - - edited

            The N checks I looked at before having to leave for the day:

            • config differences:
              • `sdiff -w 200 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge4_10_R22S11/config/runIsr.py /project/shared/BOT/rerun/cslage/PTC_6790D_All/config/runIsr.py | less`
              • Trivial rearrangement of imports
            • runIsr metadata differences:
              • `sdiff -w 200 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge4_10_R22S11/runIsr_metadata/3019101300211-SDSSi~ND_OD1.0/R22/runIsrMetadata_3019101300211-SDSSi~ND_OD1.0-R22-S11-det094.yaml /project/shared/BOT/rerun/cslage/PTC_6790D_All/runIsr_metadata/3019101300211-SDSSi~ND_OD1.0/R22/runIsrMetadata_3019101300211-SDSSi~ND_OD1.0-R22-S11-det094.yaml`
              • Differences in times/process times/context switches.  No difference in overscan/amplifier statistics.
            • diff code:
              • `git diff e98722466eb4e9c04785687313a8e42416e3940b fcb36e1abca3cbdebaa85688f1e2e56209d32726`
              • The only difference is an update in some `daf_butler` imports, which should be completely independent of gen2 processing.
            • measurePTC config differences:
              • `sdiff -w 140 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11/config/measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_6790D_All/config/measurePhotonTransferCurve.py`
              • The E2V_6790D_Gain_Edge7_R22S11 includes the "DETECTED" mask plane to exclude.
            • diff code:
              • `git diff c53c7b65681f94c8d573b0b28f92ac5b0b7fc46b 20dd8cc83f8785efb327b1db8653542def10ce9a`
              • Significant change, DM-26067 merged.
            • Check ptc outputs:
              • `pklcomp-DM-26166.py`
              • Differences in many of the parameters, including visits used, rawMeans, and gains.

            So this must be a consequence of DM-26067, so @plazas will need to look deeper.

            Show
            czw Christopher Waters added a comment - - edited The N checks I looked at before having to leave for the day: config differences: `sdiff -w 200 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge4_10_R22S11/config/runIsr.py /project/shared/BOT/rerun/cslage/PTC_6790D_All/config/runIsr.py | less` Trivial rearrangement of imports runIsr metadata differences: `sdiff -w 200 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge4_10_R22S11/runIsr_metadata/3019101300211-SDSSi~ND_OD1.0/R22/runIsrMetadata_3019101300211-SDSSi~ND_OD1.0-R22-S11-det094.yaml /project/shared/BOT/rerun/cslage/PTC_6790D_All/runIsr_metadata/3019101300211-SDSSi~ND_OD1.0/R22/runIsrMetadata_3019101300211-SDSSi~ND_OD1.0-R22-S11-det094.yaml` Differences in times/process times/context switches.  No difference in overscan/amplifier statistics. diff code: `git diff e98722466eb4e9c04785687313a8e42416e3940b fcb36e1abca3cbdebaa85688f1e2e56209d32726` The only difference is an update in some `daf_butler` imports, which should be completely independent of gen2 processing. measurePTC config differences: `sdiff -w 140 /project/cslage/BOT_lspdev/E2V_6790D_Gain_Edge7_R22S11/config/measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_6790D_All/config/measurePhotonTransferCurve.py` The E2V_6790D_Gain_Edge7_R22S11 includes the "DETECTED" mask plane to exclude. diff code: `git diff c53c7b65681f94c8d573b0b28f92ac5b0b7fc46b 20dd8cc83f8785efb327b1db8653542def10ce9a` Significant change, DM-26067 merged. Check ptc outputs: `pklcomp- DM-26166 .py` Differences in many of the parameters, including visits used, rawMeans, and gains. So this must be a consequence of DM-26067 , so @plazas will need to look deeper.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Intrigued by the above, I decided to poke to. Findings so far: running minimal ISR (no bias or anything, just overscan and assembly) gives identical means between w_28 and w_30.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Intrigued by the above, I decided to poke to. Findings so far: running minimal ISR (no bias or anything, just overscan and assembly) gives identical means between w_28 and w_30.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            I suspected it might lie in some weird afw stats thing + DM-25934 but that seems innocent. I ran

            runIsr.py /project/shared/BOT/ --output /home/mfl/scratch --id detector=94 expId=3019101300315 -c isr.doBias=False isr.doDark=False isr.doFlat=False isr.doDefect=False isr.doCrosstalk=False
            

            to make a minimally reduced postISRCCD and then

            measurePhotonTransferCurve.py /project/shared/BOT/ --output /home/mfl/scratch --id detector=94 --visit-pairs 3019101300315,3019101300315
            

            and confirmed that the means (and covDiffAstier) were identical before/after that merge (specifically on master vs the checkout just before that merge).

            Note that doesn't involve a bias subtraction, but certainly the stats part is off the hook. Bed now though I'm afraid.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - I suspected it might lie in some weird afw stats thing + DM-25934 but that seems innocent. I ran runIsr.py /project/shared/BOT/ --output /home/mfl/scratch --id detector= 94 expId= 3019101300315 -c isr.doBias=False isr.doDark=False isr.doFlat=False isr.doDefect=False isr.doCrosstalk=False to make a minimally reduced postISRCCD and then measurePhotonTransferCurve.py /project/shared/BOT/ --output /home/mfl/scratch --id detector= 94 --visit-pairs 3019101300315 , 3019101300315 and confirmed that the means (and covDiffAstier ) were identical before/after that merge (specifically on master vs the checkout just before that merge). Note that doesn't involve a bias subtraction, but certainly the stats part is off the hook. Bed now though I'm afraid.
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            I used

            /project/cslage/BOT_lspdev/standalone/Run_PTC_Edge_15Jul20.py
             

            with a few modifications to ingest the defects: I added

             DEFECT_DIR = RERUN_DIR + '/calibrations/LSSTCam/defects'
             cmd = 'ingestCuratedCalibs.py %s %s --calib %s --config clobber=True --ignore-ingested'%(DATA_DIR, DEFECT_DIR, CALIB_DIR)
             runCommand(cmd) 

            to ingest the defects that were generated, and then I had to modify the calibration registry by hand to change the start validity date. Once the defects were ingested with an appropiate start date, I ran Craig's script, and got some more reasonable gains (closer to 1 instead of to 1.8) and turning off at about ~100k ADU: PTC_det94-DM-26166_2020AUG05.pdf

            The script I ran is in

             /home/plazas/code_andres/Run_PTC_Edge_15Jul20_ANDRES.py 

            (basically the same as Craig's, but with the modification above), and the command was

             python Run_PTC_Edge_15Jul20_ANDRES.py --dir /project/plazas/DM-26166-cuatro --run 6790D --raft R22 --sensor S11 --num 70

            . This was in

             w_2020_31 

            with the DM-25711 version of cp_pipe. I'll re-run when DM-25711 has been merged, to confirm the results.

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited I used /project/cslage/BOT_lspdev/standalone/Run_PTC_Edge_15Jul20.py with a few modifications to ingest the defects: I added DEFECT_DIR = RERUN_DIR + '/calibrations/LSSTCam/defects' cmd = 'ingestCuratedCalibs.py %s %s --calib %s --config clobber=True --ignore-ingested' % (DATA_DIR, DEFECT_DIR, CALIB_DIR) runCommand(cmd) to ingest the defects that were generated, and then I had to modify the calibration registry by hand to change the start validity date. Once the defects were ingested with an appropiate start date, I ran Craig's script, and got some more reasonable gains (closer to 1 instead of to 1.8) and turning off at about ~100k ADU: PTC_det94-DM-26166_2020AUG05.pdf The script I ran is in /home/plazas/code_andres/Run_PTC_Edge_15Jul20_ANDRES.py (basically the same as Craig's, but with the modification above), and the command was python Run_PTC_Edge_15Jul20_ANDRES.py --dir /project/plazas/DM-26166-cuatro --run 6790D --raft R22 --sensor S11 --num 70 . This was in w_2020_31 with the DM-25711 version of cp_pipe. I'll re-run when DM-25711 has been merged, to confirm the results.
            Hide
            cslage Craig Lage added a comment -

            I now understand how to fix this, but I don't understand why it started happening.  The postISRCCD mean shift is caused by the ISR performing flat fielding.  If I set isr.doFlat=False in the runIsr.py command, then the ISR means and the PTC gains go back where they should be.  Before ptc.py was refactored, doFlat was automatically set to False.  But even after the refactor, I got good results without specifying doFlat=False.  At some point something changed and runIsr.py started doing flat fielding.  I don't know what changed, but my problem is fixed if I just set isr.doFLat=False.  I've run one CCD this way, and am now running the rest. So I guess this ticket can be closed.  With the refactoring of ptc.py, how do we prevent in the future that someone runs ptc.py with postISRCCD data that has been flat fielded? Is this necessary, or do we just rely on people to know what they are doing?

            Show
            cslage Craig Lage added a comment - I now understand how to fix this, but I don't understand why it started happening.  The postISRCCD mean shift is caused by the ISR performing flat fielding.  If I set isr.doFlat=False in the runIsr.py command, then the ISR means and the PTC gains go back where they should be.  Before ptc.py was refactored, doFlat was automatically set to False.  But even after the refactor, I got good results without specifying doFlat=False.  At some point something changed and runIsr.py started doing flat fielding.  I don't know what changed, but my problem is fixed if I just set isr.doFLat=False.  I've run one CCD this way, and am now running the rest. So I guess this ticket can be closed.  With the refactoring of ptc.py, how do we prevent in the future that someone runs ptc.py with postISRCCD data that has been flat fielded? Is this necessary, or do we just rely on people to know what they are doing?
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            > With the refactoring of ptc.py, how do we prevent in the future that someone runs ptc.py with postISRCCD data that has been flat fielded? Is this necessary, or do we just rely on people to know what they are doing?

            Thanks, Craig. This is a good question. After discussing a bit with Chris, the PTC task in gen3 should probably take as input the configuration output of the ISR task and validate that it matches its own options (and raise otherwise).

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - > With the refactoring of ptc.py, how do we prevent in the future that someone runs ptc.py with postISRCCD data that has been flat fielded? Is this necessary, or do we just rely on people to know what they are doing? Thanks, Craig. This is a good question. After discussing a bit with Chris, the PTC task in gen3 should probably take as input the configuration output of the ISR task and validate that it matches its own options (and raise otherwise).

              People

              Assignee:
              plazas Andrés Alejandro Plazas Malagón
              Reporter:
              cslage Craig Lage
              Reviewers:
              Christopher Waters, Merlin Fisher-Levine
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: