Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31896

Clarify PTC code

    XMLWordPrintable

Details

    • Improvement
    • Status: Done
    • Resolution: Done
    • None
    • cp_pipe
    • None

    Description

      Identify PTC methods that are never called, and remove them (getCCov is an example).

      Determine if single-use methods are clearer than having that code in-line.

      Update documentation to clearly diagram how the code is called, to allow easier future debugging and modification.

      Attachments

        Activity

          pipetask run -j 16 -d "detector IN (0) AND instrument='LSSTCam' AND exposure IN (3021111500030..3021111500159) " -b /repo/main -i u/plazas/ptc_20211209-test1,LSSTCam/raw/all,LSSTCam/calib,LSSTCam/calib/u/cslage/20211117A -o u/plazas/ptc_20211216-test1 -p /project/plazas/WORK/DM-31896-clarify-PTC-code/measurePhotonTransferCurve.yaml --register-dataset-types
          

          plazas Andrés Alejandro Plazas Malagón added a comment - pipetask run -j 16 -d "detector IN (0) AND instrument='LSSTCam' AND exposure IN (3021111500030..3021111500159) " -b /repo/main -i u/plazas/ptc_20211209-test1,LSSTCam/raw/all,LSSTCam/calib,LSSTCam/calib/u/cslage/20211117A -o u/plazas/ptc_20211216-test1 -p /project/plazas/WORK/DM-31896-clarify-PTC-code/measurePhotonTransferCurve.yaml --register-dataset-types

          Results agree with the code in main:

          main:

           run -j 32 -d "detector IN (55) AND instrument='LSSTCam' AND exposure IN (3021120600575..3021120700825) AND exposure.observation_type='flat'" -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,u/cslage/calib/13144/calib.20220107 -o u/plazas/ptc_test-BOT-13144-2020JAN11.1 -p /project/cslage/BOT_LSSTCam/pipelines/measurePhotonTransferCurve.yaml --register-dataset-types
          

          file: /repo/main/u/plazas/ptc_test-BOT-13144-2020JAN11.1/20220111T213315Z/ptc/ptc_LSSTCam_R13_S01_u_plazas_ptc_test-BOT-13144-2020JAN11_1_20220111T213315Z.fits

          PTC_det55-DM31896-main-BOT-13144.pdf

          DM-31896:

          pipetask run -j 32 -d "detector IN (55) AND instrument='LSSTCam' AND exposure IN (3021120600575..3021120700825) AND exposure.observation_type='flat'" -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,u/cslage/calib/13144/calib.20220107 -o u/plazas/ptc_test-BOT-13144-branch-2020JAN11.1 -p /project/cslage/BOT_LSSTCam/pipelines/measurePhotonTransferCurve.yaml --register-dataset-types
          

          file: /repo/main/u/plazas/ptc_test-BOT-13144-branch-2020JAN11.1/20220111T233407Z/ptc/ptc_LSSTCam_R13_S01_u_plazas_ptc_test-BOT-13144-branch-2020JAN11_1_20220111T233407Z.fits

          PTC_det55-DM31896-branch-BOT-13144.pdf

          plazas Andrés Alejandro Plazas Malagón added a comment - Results agree with the code in main : main : run -j 32 -d "detector IN (55) AND instrument='LSSTCam' AND exposure IN (3021120600575..3021120700825) AND exposure.observation_type='flat'" -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,u/cslage/calib/13144/calib.20220107 -o u/plazas/ptc_test-BOT-13144-2020JAN11.1 -p /project/cslage/BOT_LSSTCam/pipelines/measurePhotonTransferCurve.yaml --register-dataset-types file : /repo/main/u/plazas/ptc_test-BOT-13144-2020JAN11.1/20220111T213315Z/ptc/ptc_LSSTCam_R13_S01_u_plazas_ptc_test-BOT-13144-2020JAN11_1_20220111T213315Z.fits PTC_det55-DM31896-main-BOT-13144.pdf DM-31896 : pipetask run -j 32 -d "detector IN (55) AND instrument='LSSTCam' AND exposure IN (3021120600575..3021120700825) AND exposure.observation_type='flat'" -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,u/cslage/calib/13144/calib.20220107 -o u/plazas/ptc_test-BOT-13144-branch-2020JAN11.1 -p /project/cslage/BOT_LSSTCam/pipelines/measurePhotonTransferCurve.yaml --register-dataset-types file: /repo/main/u/plazas/ptc_test-BOT-13144-branch-2020JAN11.1/20220111T233407Z/ptc/ptc_LSSTCam_R13_S01_u_plazas_ptc_test-BOT-13144-branch-2020JAN11_1_20220111T233407Z.fits PTC_det55-DM31896-branch-BOT-13144.pdf

          Using Craig's notebook and comparing the results from this branch with a previous run by him (with his help), the results also show agreement.

          bps script:

          pipelineYaml: /project/cslage/BOT_LSSTCam/pipelines/measurePhotonTransferCurve.yaml # Changed to EXPAPPROX, doLinearize: false
          project: BOT            # I just picked some string that maybe means something to you.
          campaign: PTC           # I just picked some string that maybe means something to you.
          computeSite: ncsapool
          requestMemory: 2048     # can leave this out, but easy to keep updated.
          requestCpus: 1          # can leave this out, but easy to keep updated.
          extraInitOptions: "--no-versions"  # note: non-dev runs should never do this
          extraRunQuantumOptions: "--no-versions"  # note: non-dev runs should never do this
           
          payload:
            payloadName: bps_13144Q_2021JAN12.2      # again this should mean something to you about the input data set + pipeline
            butlerConfig: /repo/main
            inCollection: LSSTCam/raw/all,LSSTCam/calib,u/cslage/calib/13144/calib.20220110
            dataQuery: detector IN (0..188) AND instrument='LSSTCam' AND exposure IN (3021120600575..3021120700825) AND exposure.observation_type='flat'
          pipetask:
            assembleCoadd:
              requestMemory: 4096
            ptcExtract:
              requestMemory: 4096
          

          Files produced by Craig (/project/cslage/BOT_LSSTCam/notebooks/Gain_Comparison_General_10Jan22.ipynb):

          A00_Comparison_Plazas_13Jan22.pdf Gains_Comparison_Plazas_13Jan22.pdf Noise_Comparison_Plazas_13Jan22.pdf Turnoff_Comparison_Plazas_13Jan22.pdf

          plazas Andrés Alejandro Plazas Malagón added a comment - Using Craig's notebook and comparing the results from this branch with a previous run by him (with his help), the results also show agreement. bps script: pipelineYaml: /project/cslage/BOT_LSSTCam/pipelines/measurePhotonTransferCurve.yaml # Changed to EXPAPPROX, doLinearize: false project: BOT # I just picked some string that maybe means something to you. campaign: PTC # I just picked some string that maybe means something to you. computeSite: ncsapool requestMemory: 2048 # can leave this out, but easy to keep updated. requestCpus: 1 # can leave this out, but easy to keep updated. extraInitOptions: "--no-versions" # note: non-dev runs should never do this extraRunQuantumOptions: "--no-versions" # note: non-dev runs should never do this   payload: payloadName: bps_13144Q_2021JAN12.2 # again this should mean something to you about the input data set + pipeline butlerConfig: /repo/main inCollection: LSSTCam/raw/all,LSSTCam/calib,u/cslage/calib/13144/calib.20220110 dataQuery: detector IN (0..188) AND instrument='LSSTCam' AND exposure IN (3021120600575..3021120700825) AND exposure.observation_type='flat' pipetask: assembleCoadd: requestMemory: 4096 ptcExtract: requestMemory: 4096 Files produced by Craig ( /project/cslage/BOT_LSSTCam/notebooks/Gain_Comparison_General_10Jan22.ipynb ): A00_Comparison_Plazas_13Jan22.pdf Gains_Comparison_Plazas_13Jan22.pdf Noise_Comparison_Plazas_13Jan22.pdf Turnoff_Comparison_Plazas_13Jan22.pdf
          plazas Andrés Alejandro Plazas Malagón added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35713/pipeline
          plazas Andrés Alejandro Plazas Malagón added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35766/pipeline

          Looks good to me.  A few typos are all that's left.

          czw Christopher Waters added a comment - Looks good to me.  A few typos are all that's left.

          People

            plazas Andrés Alejandro Plazas Malagón
            czw Christopher Waters
            Christopher Waters
            Andrés Alejandro Plazas Malagón, Christopher Waters
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.