Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27185

ptc.py fails with ptcFitType=FULLCOVARIANCE

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
      None

      Description

      Aaron Roodman has requested feedback on the long-range correlations problem, which was seen on E2V sensors in the 9 raft data.  We believe it is fixed, but want confirmation.  This needs data on the pixel-pixel correlations.  I tried to run this with ptcFitType=FULLCOVARIANCE, but it failed.  I tried 3 different CCDs, and got two different errors.  Command string and errors are below.

      Command string:

      measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_FullCov_12606 --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216     -c maxMeanSignal=100000 ptcFitType=FULLCOVARIANCE doPhotodiode=False sigmaCutPtcOutliers=5.0 initialNonLinearityExclusionThresholdPositive=0.25  --clobber-config --clobber-version -j 1

      Error 1 (detector=94):

      Traceback (most recent call last):
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/bin/measurePhotonTransferCurve.py", line 27, in <module>
      		 
          MeasurePhotonTransferCurveTask.parseAndRun()
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/cmdLineTask.py", line 610, in parseAndRun
      		 
          resultList = taskRunner.run(parsedCmd)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/cmdLineTask.py", line 221, in run
      		 
          resultList = list(mapFunc(self, targetList))
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/20.0.0-30-g1d38f5b5+80082c6d28/python/lsst/pipe/tasks/getRepositoryData.py", line 57, in __call__
      		 
          result = task.runDataRef(dataRefList)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper
      		 
          res = func(self, *args, **keyArgs)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/ptc.py", line 364, in runDataRef
      		 
          'detectorName': detName, 'calibDate': calibDate})
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/20.0.0-1-gb88604f+acecce4127/python/lsst/daf/persistence/butler.py", line 1453, in put
      		 
          location.getRepository().write(location, obj)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/20.0.0-1-gb88604f+acecce4127/python/lsst/daf/persistence/repository.py", line 182, in write
      		 
          return butlerLocationStorage.write(butlerLocation, obj)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/20.0.0-1-gb88604f+acecce4127/python/lsst/daf/persistence/posixStorage.py", line 257, in write
      		 
          writeFormatter(butlerLocation, obj)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/20.0.0-1-gb88604f+acecce4127/python/lsst/daf/persistence/posixStorage.py", line 821, in writeFitsCatalogStorage
      		 
          obj.writeFits(logLoc.locString(), **kwds)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/calibType.py", line 336, in writeFits
      		 
          tableList = self.toTable()
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/ptcDataset.py", line 492, in toTable
      		 
          } for ampName in self.ampNames])
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/ptcDataset.py", line 492, in <listcomp>
      		 
          } for ampName in self.ampNames])
      		 
      ValueError: cannot reshape array of size 1152 into shape (1344,)

      Error 2 (detectors 92 or 93):

      Traceback (most recent call last):
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/bin/measurePhotonTransferCurve.py", line 27, in <module>
      		 
          MeasurePhotonTransferCurveTask.parseAndRun()
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/cmdLineTask.py", line 610, in parseAndRun
      		 
          resultList = taskRunner.run(parsedCmd)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/cmdLineTask.py", line 221, in run
      		 
          resultList = list(mapFunc(self, targetList))
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/20.0.0-30-g1d38f5b5+80082c6d28/python/lsst/pipe/tasks/getRepositoryData.py", line 57, in __call__
      		 
          result = task.runDataRef(dataRefList)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper
      		 
          res = func(self, *args, **keyArgs)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/ptc.py", line 334, in runDataRef
      		 
          datasetPtc = self.fitCovariancesAstier(datasetPtc, covariancesWithTags)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/ptc.py", line 447, in fitCovariancesAstier
      		 
          maxIterFullFit=self.config.maxIterFullFitCovariancesAstier)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/astierCovPtcUtils.py", line 376, in fitData
      		 
          covFitList = loadData(tupleName, lparams)
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/astierCovPtcUtils.py", line 311, in loadData
      		 
          cc.initFit()  # allows to get a crude gain.
      		 
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/astierCovPtcFit.py", line 330, in initFit
      		 
          2, w=self.sqrtW[:, i, j])
      		 
        File "<__array_function__ internals>", line 6, in polyfit
      		 
        File "/software/lsstsw/stack_20200922/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/polynomial.py", line 629, in polyfit
      		 
          c, resids, rank, s = lstsq(lhs, rhs, rcond)
      		 
        File "<__array_function__ internals>", line 6, in lstsq
      		 
        File "/software/lsstsw/stack_20200922/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/linalg/linalg.py", line 2306, in lstsq
      		 
          x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
      		 
      ValueError: On entry to DLASCL parameter number 4 had an illegal value

        Attachments

        1. Det183_C10.png
          Det183_C10.png
          23 kB
        2. Det94_C15_Check.png
          Det94_C15_Check.png
          19 kB
        3. Det94_C15_Code.png
          Det94_C15_Code.png
          27 kB
        4. mean_var_scrambling.png
          mean_var_scrambling.png
          84 kB
        5. PTC_det94_EXPAPPROXIMATION_2020OCT17.pdf
          81 kB
        6. PTC_det94_FULLCOV_2020OCT17.pdf
          222 kB
        7. PTC_det94_kink.pdf
          224 kB
        8. PTC_det94_nokink.pdf
          223 kB

          Issue Links

          is contained by

          Improvement - An improvement or enhancement to an existing feature or task. DM-27161 PTC BOT issues.

          • Done
          is parent task of

          Improvement - An improvement or enhancement to an existing feature or task. DM-27458 FULLCOVARIANCE in PTC task is rejecting more points than it should for some BOT data detectors

          • Done
          mitigates

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27267 Fix docstring and comments in utils/funcPolynomial

          • Invalid
          mentioned in

          Page Loading...

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            11 older comments
            Hide
            Permalink
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Thank you, Craig. I added the padding in toTable, but I padded the covariance weights with zeroes instead of NaNs, so that it propagates correctly to the mask and then it plots without having to add your second piece of code.

            I also now let the cov NaN go on and then catch it later, as you suggested

            if (amp in covFits) and (covFits[amp].covParams is not None) and (covFitsNoB[amp].covParams is not None):

            The fir does get slower, maybe because it is rejecting all those points, as you say (I don't know if we might need to add some other weight to make the fit more robust, like the Cauchy weights added here by Chris: https://github.com/lsst/cp_pipe/blob/tickets/DM-26545/python/lsst/cp/pipe/linearity.py#L369).
            This is still something to investigate, along with the kink that you mention.

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Thank you, Craig. I added the padding in toTable , but I padded the covariance weights with zeroes instead of NaNs , so that it propagates correctly to the mask and then it plots without having to add your second piece of code. I also now let the cov NaN go on and then catch it later, as you suggested if (amp in covFits) and (covFits[amp].covParams is not None ) and (covFitsNoB[amp].covParams is not None ): The fir does get slower, maybe because it is rejecting all those points, as you say (I don't know if we might need to add some other weight to make the fit more robust, like the Cauchy weights added here by Chris: https://github.com/lsst/cp_pipe/blob/tickets/DM-26545/python/lsst/cp/pipe/linearity.py#L369 ). This is still something to investigate, along with the kink that you mention.
            Hide
            Permalink
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            Craig found out that the covariances get sorted in order of increasing mean in astierCovPtcFit.makeCovArray, but rawMeans and rawVars don't get sorted. Sorting these arrays by the index in rawMeans (in addition to rawExpTimes) in the extract subtask, solves the "kink" (it also helps with other plots): PTC_det94_nokink.pdf vs PTC_det94_kink.pdf

            Command (w_2020_43)

            measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c maxMeanSignal=120000 ptcFitType=FULLCOV doPhotodiode=False maxIterFullFitCovariancesAstier=10 sigmaClipFullFitCovariancesAstier=15 --clobber-config --clobber-version -j 1


            plotPhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=94 -c datasetFileName=/project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/DM-27185/calibrations/ptc/ptcDataset-det094.fits --clobber-versions --clobber-config -j 1

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited Craig found out that the covariances get sorted in order of increasing mean in astierCovPtcFit.makeCovArray , but rawMeans and rawVars don't get sorted. Sorting these arrays by the index in rawMeans (in addition to rawExpTimes ) in the extract subtask, solves the "kink" (it also helps with other plots): PTC_det94_nokink.pdf vs PTC_det94_kink.pdf Command ( w_2020_43 ) measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c maxMeanSignal=120000 ptcFitType=FULLCOV doPhotodiode=False maxIterFullFitCovariancesAstier=10 sigmaClipFullFitCovariancesAstier=15 --clobber-config --clobber-version -j 1 plotPhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=94 -c datasetFileName=/project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/DM-27185/calibrations/ptc/ptcDataset-det094.fits --clobber-versions --clobber-config -j 1
            Hide
            Permalink
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Craig also suggested adding bounds to the parameters fit in EXPAPPROXIMATION (lowers = [-1E-4, 0.5, -100], uppers = [1E-4, 2.5, 100]); this helps to get more reasonable noise values at the first iteration, and then the fit doesn't discard as many points unnecessarily. I have added that too in this ticket.

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Craig also suggested adding bounds to the parameters fit in EXPAPPROXIMATION (lowers = [-1E-4, 0.5, -100] , uppers = [1E-4, 2.5, 100] ); this helps to get more reasonable noise values at the first iteration, and then the fit doesn't discard as many points unnecessarily. I have added that too in this ticket.
            Hide
            Permalink
            plazas Andrés Alejandro Plazas Malagón added a comment -

            I'll send this ticket to review the changes that have already been made, and I have moved the last issue to ticket DM-27458

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - I'll send this ticket to review the changes that have already been made, and I have moved the last issue to ticket DM-27458
            Hide
            Permalink
            plazas Andrés Alejandro Plazas Malagón added a comment -

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33063/pipeline

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33063/pipeline

              People

              Assignee:
              plazas Andrés Alejandro Plazas Malagón
              Reporter:
              cslage Craig Lage
              Reviewers:
              Christopher Waters
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.