Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31938

Have PTC dataset unpad results

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None

      Description

      The PTC code currently pads finalVars, finalMeans, and covariances (and other?) with NaN values to ensure that all amplifiers have the same length for FITS compatibility.  It might be useful to have the read methods unpad these values, so the returned data is whatever length was populated for that amplifier.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          No builds found.
          czw Christopher Waters created issue -
          yusra Yusra AlSayyad made changes -
          Field Original Value New Value
          Epic Link DM-31830 [ 720558 ] DM-32828 [ 916516 ]
          czw Christopher Waters made changes -
          Epic Link DM-32828 [ 916516 ] DM-34493 [ 1449388 ]
          plazas Andrés Alejandro Plazas Malagón made changes -
          Story Points 2
          Hide
          Permalink
          plazas Andrés Alejandro Plazas Malagón added a comment - - edited 

          pipetask run -j 32 -d "detector IN (55) AND instrument='LSSTCam' AND exposure IN (3021111500030..3021111500150) " -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,LSSTCam/calib/u/cslage/20211117A -p ${CP_PIPE_DIR}/pipelines/cpPtc.yaml -c isr:doFlat=False -c isr:doCrosstalk=False -c ptcSolve:ptcFitType=FULLCOVARIANCE -o u/plazas/DM-31938.test.10 --register-dataset-types


              import lsst.daf.butler as dB
          		 
              import numpy as np
          		 
             genCollection = "u/plazas/DM-31938.test.10"
          		 
             butler = dB.Butler("/repo/main", collections=genCollection)
          		 
             ptcDataset = butler.get("ptc", instrument="LSSTCam", detector=55, exposure=3021111500150)


          In [5]: ptcDataset.finalMeans['C15']
          		 
          Out[5]: 
          [2131.914311622597,
          		 
           2599.4489882208527,
          		 
           3176.2206420720277,
          		 
           3874.6031636729003,
          		 
           4730.795394857724,
          		 
           5778.363401045608,
          		 
           54817.668863291736,
          		 
           56470.07757644713,
          		 
           63628.38805759857,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan,
          		 
           nan]

          After:

          In [2]: ptcDataset.finalMeans['C15']
          		 
          Out[2]: 
          [2131.914311622597,
          		 
           2599.4489882208527,
          		 
           3176.2206420720277,
          		 
           3874.6031636729003,
          		 
           4730.795394857724,
          		 
           5778.363401045608,
          		 
           54817.668863291736,
          		 
           56470.07757644713,
          		 
           63628.38805759857]

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - - edited pipetask run -j 32 -d "detector IN (55) AND instrument='LSSTCam' AND exposure IN (3021111500030..3021111500150) " -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,LSSTCam/calib/u/cslage/20211117A -p ${CP_PIPE_DIR}/pipelines/cpPtc.yaml -c isr:doFlat=False -c isr:doCrosstalk=False -c ptcSolve:ptcFitType=FULLCOVARIANCE -o u/plazas/DM-31938.test.10 --register-dataset-types import lsst.daf.butler as dB import numpy as np genCollection = "u/plazas/DM-31938.test.10" butler = dB.Butler( "/repo/main" , collections = genCollection) ptcDataset = butler.get( "ptc" , instrument = "LSSTCam" , detector = 55 , exposure = 3021111500150 ) In [ 5 ]: ptcDataset.finalMeans[ 'C15' ] Out[ 5 ]: [ 2131.914311622597 , 2599.4489882208527 , 3176.2206420720277 , 3874.6031636729003 , 4730.795394857724 , 5778.363401045608 , 54817.668863291736 , 56470.07757644713 , 63628.38805759857 , nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan] After: In [ 2 ]: ptcDataset.finalMeans[ 'C15' ] Out[ 2 ]: [ 2131.914311622597 , 2599.4489882208527 , 3176.2206420720277 , 3874.6031636729003 , 4730.795394857724 , 5778.363401045608 , 54817.668863291736 , 56470.07757644713 , 63628.38805759857 ]
          plazas Andrés Alejandro Plazas Malagón made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          plazas Andrés Alejandro Plazas Malagón made changes -
          Reviewers Christopher Waters [ cwaters ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          Permalink
          czw Christopher Waters added a comment -

          Looks good to me.

          Show
          czw Christopher Waters added a comment - Looks good to me.
          czw Christopher Waters made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          plazas Andrés Alejandro Plazas Malagón made changes -
          Status Reviewed [ 10101 ] In Review [ 10004 ]
          Hide
          Permalink
          plazas Andrés Alejandro Plazas Malagón added a comment -

          Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36811/pipeline/

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36811/pipeline/
          plazas Andrés Alejandro Plazas Malagón made changes -
          Story Points 2 4
          czw Christopher Waters made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          Hide
          Permalink
          plazas Andrés Alejandro Plazas Malagón added a comment -

          Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36842/pipeline/

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36842/pipeline/
          plazas Andrés Alejandro Plazas Malagón made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]
          yusra Yusra AlSayyad made changes -
          Epic Link DM-34493 [ 1449388 ] DM-35762 [ 1880232 ]

            People

            Assignee:
            plazas Andrés Alejandro Plazas Malagón
            Reporter:
            czw Christopher Waters
            Reviewers:
            Christopher Waters
            Watchers:
            Andrés Alejandro Plazas Malagón, Christopher Waters
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.