Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27185

ptc.py fails with ptcFitType=FULLCOVARIANCE

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
      None

      Description

      Aaron Roodman has requested feedback on the long-range correlations problem, which was seen on E2V sensors in the 9 raft data.  We believe it is fixed, but want confirmation.  This needs data on the pixel-pixel correlations.  I tried to run this with ptcFitType=FULLCOVARIANCE, but it failed.  I tried 3 different CCDs, and got two different errors.  Command string and errors are below.

      Command string:

      measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_FullCov_12606 --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216     -c maxMeanSignal=100000 ptcFitType=FULLCOVARIANCE doPhotodiode=False sigmaCutPtcOutliers=5.0 initialNonLinearityExclusionThresholdPositive=0.25  --clobber-config --clobber-version -j 1

      Error 1 (detector=94):

      Traceback (most recent call last):
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/bin/measurePhotonTransferCurve.py", line 27, in <module>
          MeasurePhotonTransferCurveTask.parseAndRun()
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/cmdLineTask.py", line 610, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/cmdLineTask.py", line 221, in run
          resultList = list(mapFunc(self, targetList))
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/20.0.0-30-g1d38f5b5+80082c6d28/python/lsst/pipe/tasks/getRepositoryData.py", line 57, in __call__
          result = task.runDataRef(dataRefList)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/ptc.py", line 364, in runDataRef
          'detectorName': detName, 'calibDate': calibDate})
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/20.0.0-1-gb88604f+acecce4127/python/lsst/daf/persistence/butler.py", line 1453, in put
          location.getRepository().write(location, obj)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/20.0.0-1-gb88604f+acecce4127/python/lsst/daf/persistence/repository.py", line 182, in write
          return butlerLocationStorage.write(butlerLocation, obj)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/20.0.0-1-gb88604f+acecce4127/python/lsst/daf/persistence/posixStorage.py", line 257, in write
          writeFormatter(butlerLocation, obj)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/20.0.0-1-gb88604f+acecce4127/python/lsst/daf/persistence/posixStorage.py", line 821, in writeFitsCatalogStorage
          obj.writeFits(logLoc.locString(), **kwds)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/calibType.py", line 336, in writeFits
          tableList = self.toTable()
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/ptcDataset.py", line 492, in toTable
          } for ampName in self.ampNames])
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/ptcDataset.py", line 492, in <listcomp>
          } for ampName in self.ampNames])
      ValueError: cannot reshape array of size 1152 into shape (1344,)

      Error 2 (detectors 92 or 93):

      Traceback (most recent call last):
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/bin/measurePhotonTransferCurve.py", line 27, in <module>
          MeasurePhotonTransferCurveTask.parseAndRun()
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/cmdLineTask.py", line 610, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/cmdLineTask.py", line 221, in run
          resultList = list(mapFunc(self, targetList))
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/20.0.0-30-g1d38f5b5+80082c6d28/python/lsst/pipe/tasks/getRepositoryData.py", line 57, in __call__
          result = task.runDataRef(dataRefList)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/ptc.py", line 334, in runDataRef
          datasetPtc = self.fitCovariancesAstier(datasetPtc, covariancesWithTags)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/ptc.py", line 447, in fitCovariancesAstier
          maxIterFullFit=self.config.maxIterFullFitCovariancesAstier)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/astierCovPtcUtils.py", line 376, in fitData
          covFitList = loadData(tupleName, lparams)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/astierCovPtcUtils.py", line 311, in loadData
          cc.initFit()  # allows to get a crude gain.
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/cp_pipe/20.0.0-18-g541646f+c4459f5943/python/lsst/cp/pipe/astierCovPtcFit.py", line 330, in initFit
          2, w=self.sqrtW[:, i, j])
        File "<__array_function__ internals>", line 6, in polyfit
        File "/software/lsstsw/stack_20200922/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/polynomial.py", line 629, in polyfit
          c, resids, rank, s = lstsq(lhs, rhs, rcond)
        File "<__array_function__ internals>", line 6, in lstsq
        File "/software/lsstsw/stack_20200922/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/linalg/linalg.py", line 2306, in lstsq
          x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
      ValueError: On entry to DLASCL parameter number 4 had an illegal value
      

        Attachments

        1. Det183_C10.png
          Det183_C10.png
          23 kB
        2. Det94_C15_Check.png
          Det94_C15_Check.png
          19 kB
        3. Det94_C15_Code.png
          Det94_C15_Code.png
          27 kB
        4. mean_var_scrambling.png
          mean_var_scrambling.png
          84 kB
        5. PTC_det94_EXPAPPROXIMATION_2020OCT17.pdf
          81 kB
        6. PTC_det94_FULLCOV_2020OCT17.pdf
          222 kB
        7. PTC_det94_kink.pdf
          224 kB
        8. PTC_det94_nokink.pdf
          223 kB

          Issue Links

            Activity

            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Thank you, Craig. I added the padding in toTable, but I padded the covariance weights with zeroes instead of NaNs, so that it propagates correctly to the mask and then it plots without having to add your second piece of code.

            I also now let the cov NaN go on and then catch it later, as you suggested

            if (amp in covFits) and (covFits[amp].covParams is not None) and (covFitsNoB[amp].covParams is not None):
            

            The fir does get slower, maybe because it is rejecting all those points, as you say (I don't know if we might need to add some other weight to make the fit more robust, like the Cauchy weights added here by Chris: https://github.com/lsst/cp_pipe/blob/tickets/DM-26545/python/lsst/cp/pipe/linearity.py#L369).
            This is still something to investigate, along with the kink that you mention.

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Thank you, Craig. I added the padding in toTable , but I padded the covariance weights with zeroes instead of NaNs , so that it propagates correctly to the mask and then it plots without having to add your second piece of code. I also now let the cov NaN go on and then catch it later, as you suggested if (amp in covFits) and (covFits[amp].covParams is not None ) and (covFitsNoB[amp].covParams is not None ): The fir does get slower, maybe because it is rejecting all those points, as you say (I don't know if we might need to add some other weight to make the fit more robust, like the Cauchy weights added here by Chris: https://github.com/lsst/cp_pipe/blob/tickets/DM-26545/python/lsst/cp/pipe/linearity.py#L369 ). This is still something to investigate, along with the kink that you mention.
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            Craig found out that the covariances get sorted in order of increasing mean in astierCovPtcFit.makeCovArray, but rawMeans and rawVars don't get sorted. Sorting these arrays by the index in rawMeans (in addition to rawExpTimes) in the extract subtask, solves the "kink" (it also helps with other plots): PTC_det94_nokink.pdf vs PTC_det94_kink.pdf

            Command (w_2020_43)

            measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c maxMeanSignal=120000 ptcFitType=FULLCOV doPhotodiode=False maxIterFullFitCovariancesAstier=10 sigmaClipFullFitCovariancesAstier=15 --clobber-config --clobber-version -j 1
            

            plotPhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=94 -c datasetFileName=/project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/DM-27185/calibrations/ptc/ptcDataset-det094.fits --clobber-versions --clobber-config -j 1
            

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited Craig found out that the covariances get sorted in order of increasing mean in astierCovPtcFit.makeCovArray , but rawMeans and rawVars don't get sorted. Sorting these arrays by the index in rawMeans (in addition to rawExpTimes ) in the extract subtask, solves the "kink" (it also helps with other plots): PTC_det94_nokink.pdf vs PTC_det94_kink.pdf Command ( w_2020_43 ) measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c maxMeanSignal=120000 ptcFitType=FULLCOV doPhotodiode=False maxIterFullFitCovariancesAstier=10 sigmaClipFullFitCovariancesAstier=15 --clobber-config --clobber-version -j 1 plotPhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-27185 --id detector=94 -c datasetFileName=/project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/DM-27185/calibrations/ptc/ptcDataset-det094.fits --clobber-versions --clobber-config -j 1
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Craig also suggested adding bounds to the parameters fit in EXPAPPROXIMATION (lowers = [-1E-4, 0.5, -100], uppers = [1E-4, 2.5, 100]); this helps to get more reasonable noise values at the first iteration, and then the fit doesn't discard as many points unnecessarily. I have added that too in this ticket.

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Craig also suggested adding bounds to the parameters fit in EXPAPPROXIMATION (lowers = [-1E-4, 0.5, -100] , uppers = [1E-4, 2.5, 100] ); this helps to get more reasonable noise values at the first iteration, and then the fit doesn't discard as many points unnecessarily. I have added that too in this ticket.
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            I'll send this ticket to review the changes that have already been made, and I have moved the last issue to ticket DM-27458

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - I'll send this ticket to review the changes that have already been made, and I have moved the last issue to ticket DM-27458
            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33063/pipeline

              People

              Assignee:
              plazas Andrés Alejandro Plazas Malagón
              Reporter:
              cslage Craig Lage
              Reviewers:
              Christopher Waters
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: