Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28346

Gen2 defects no longer ingestable as a curated calibration

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
      None

      Description

      It did in fact used to work correctly in that ingestCuratedCalibs.py could read the output of findDefects.py. I pulled up the w_2020_40 code, and there was a function in cp/pipe/defects.py called _writeData that got called at the end of the FindDefectTask. This wrote things out with the correct file structure so that ingestCuratedCalibs.py could read it. I looked at an output file from that time frame and there is a statement at the end of findDefects.py that says:
      findDefects INFO: Writing defects to /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12638/calibrations/LSSTCam/defects/r01_s00/2020-10-13T01:07:04.651000 in format: BOTH
      This is the directory structure that ingestCuratedCalibs.py is looking for. When I pull up the current code, that _writeData function is gone, so the output gets written into the defects directory and ingestCuratedCalibs.py fails when it tries to read it. Perhaps I'm going about things the wrong way. I'm happy to do things a different way if you can point me in the right direction.

        Attachments

          Issue Links

            Activity

            Hide
            cslage Craig Lage added a comment -

            I pulled down tickets/DM-28346 on cp_pipe and ip_isr to test out the fix.  I ran it on both ComCam and AuxTel data., and both failed in the same way, as detailed below for the auxTel.  I ran the following steps, and with these the defects files were appropriately created and ingested:

            findDefects.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --id detector=0 expId=2021012100660 -c measure.badOnAndOffPixelColumnThreshold=500 --clobber-config --clobber-versions

            ingestAssistant.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/curated /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/defects/*/*.fits 

            ingestCuratedCalibs.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/curated/latiss/defects --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --config clobber=True --ignore-ingested
            

            The appropriate defect files were then in the CALIB directory, with the file names like this:

            /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB/defects/2021-01-26T09\:15\:21.014929/defects-2021-01-26T09\:15\:21.014929-RXX-RXX_S00-000.fits

            However, when I tried to run ISR, it was unable to find the defect files.  This command, with doDefect=False successfully ran:

            runIsr.py /project/shared/auxTel/ --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --id detector=0 expId=2021012100538 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1

            But this command, with doDefect=True, gave a 'No registry for lookup' error.  I think there is a problem with the filename and it is not finding it:

            runIsr.py /project/shared/auxTel/ --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --id detector=0 expId=2021012100538 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1

            Here is the full error log:

            runIsr FATAL: Failed on dataId={'detector': 0, 'expId': 2021012100538, 'dayObs': '2021-01-21'}: RuntimeError: No registry for lookup
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/cmdLineTask.py", line 431, in __call__
                result = self.runTask(task, dataRef, kwargs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/cmdLineTask.py", line 495, in runTask
                return task.runDataRef(dataRef, **kwargs)
              File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 2621, in runDataRef
                return self.isr.runDataRef(dataRef)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/timer.py", line 157, in wrapper
                res = func(self, *args, **keyArgs)
              File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 1684, in runDataRef
                isrData = self.readIsrData(sensorRef, ccdExposure)
              File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 1121, in readIsrData
                defectList = (dataRef.get("defects")
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
                return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butler.py", line 1397, in get
                location = self._locate(datasetType, dataId, write=False)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butler.py", line 1316, in _locate
                location = repoData.repo.map(datasetType, dataId, write=write)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/repository.py", line 239, in map
                loc = self._mapper.map(*args, **kwargs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/mapper.py", line 163, in map
                return func(self.validate(dataId), write)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/cameraMapper.py", line 391, in mapClosure
                return mapping.map(mapper, dataId, write)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 460, in map
                location = Mapping.map(self, mapper, dataId, write=write)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 155, in map
                actualId = self.need(iter(self.keyDict.keys()), dataId)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 322, in need
                lookups = self.lookup(newProps, newId)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 519, in lookup
                return Mapping.lookup(self, properties, newId)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 209, in lookup
                raise RuntimeError("No registry for lookup")
            RuntimeError: No registry for lookup

            Here's the ISR command for the ComCam, if that is easier to deal with:

            runIsr.py /project/shared/comCam-CCS --rerun /project/shared/comCam-CCS/rerun/cslage/PTCTemp_2020-12-29 --calib /project/shared/comCam-CCS/rerun/cslage/PTCTemp_2020-12-29/CALIB --id detector=0 expId=3020122900043 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doSaturation=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1

            Show
            cslage Craig Lage added a comment - I pulled down tickets/ DM-28346 on cp_pipe and ip_isr to test out the fix.  I ran it on both ComCam and AuxTel data., and both failed in the same way, as detailed below for the auxTel.  I ran the following steps, and with these the defects files were appropriately created and ingested: findDefects.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --id detector=0 expId=2021012100660 -c measure.badOnAndOffPixelColumnThreshold=500 --clobber-config --clobber-versions ingestAssistant.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/curated /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/defects/*/*.fits ingestCuratedCalibs.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/curated/latiss/defects --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --config clobber=True --ignore-ingested The appropriate defect files were then in the CALIB directory, with the file names like this: /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB/defects/2021-01-26T09\:15\:21.014929/defects-2021-01-26T09\:15\:21.014929-RXX-RXX_S00-000.fits However, when I tried to run ISR, it was unable to find the defect files.  This command, with doDefect=False successfully ran: runIsr.py /project/shared/auxTel/ --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --id detector=0 expId=2021012100538 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1 But this command, with doDefect=True, gave a 'No registry for lookup' error.  I think there is a problem with the filename and it is not finding it: runIsr.py /project/shared/auxTel/ --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --id detector=0 expId=2021012100538 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1 Here is the full error log: runIsr FATAL: Failed on dataId={'detector': 0, 'expId': 2021012100538, 'dayObs': '2021-01-21'}: RuntimeError: No registry for lookup Traceback (most recent call last): File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/cmdLineTask.py", line 431, in __call__ result = self.runTask(task, dataRef, kwargs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/cmdLineTask.py", line 495, in runTask return task.runDataRef(dataRef, **kwargs) File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 2621, in runDataRef return self.isr.runDataRef(dataRef) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/timer.py", line 157, in wrapper res = func(self, *args, **keyArgs) File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 1684, in runDataRef isrData = self.readIsrData(sensorRef, ccdExposure) File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 1121, in readIsrData defectList = (dataRef.get("defects") File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butlerSubset.py", line 203, in get return self.butlerSubset.butler.get(datasetType, self.dataId, **rest) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butler.py", line 1397, in get location = self._locate(datasetType, dataId, write=False) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butler.py", line 1316, in _locate location = repoData.repo.map(datasetType, dataId, write=write) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/repository.py", line 239, in map loc = self._mapper.map(*args, **kwargs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/mapper.py", line 163, in map return func(self.validate(dataId), write) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/cameraMapper.py", line 391, in mapClosure return mapping.map(mapper, dataId, write) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 460, in map location = Mapping.map(self, mapper, dataId, write=write) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 155, in map actualId = self.need(iter(self.keyDict.keys()), dataId) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 322, in need lookups = self.lookup(newProps, newId) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 519, in lookup return Mapping.lookup(self, properties, newId) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 209, in lookup raise RuntimeError("No registry for lookup") RuntimeError: No registry for lookup Here's the ISR command for the ComCam, if that is easier to deal with: runIsr.py /project/shared/comCam-CCS --rerun /project/shared/comCam-CCS/rerun/cslage/PTCTemp_2020-12-29 --calib /project/shared/comCam-CCS/rerun/cslage/PTCTemp_2020-12-29/CALIB --id detector=0 expId=3020122900043 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doSaturation=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1
            Hide
            czw Christopher Waters added a comment -

            Looking at your repositories, I see that:

            sqlite3 /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB/calibRegistry.sqlite3 \
                "select * from defects;"
            id      filter  raftName detectorName  detector        calibDate               validStart      validEnd
            1       None    RXX     RXX_S00        0       2021-01-26T09:15:21.014929      2021-01-26      2037-12-31

            Comparing to the above command, it looks like you're testing on expId=2021012100538, which looks to be prior to the validStart applied to the defects.  Manually editing the sqlite UPDATE defect SET validStart='2021-01-01' WHERE id=1 is one solution.  For the future, passing a --date 2021-01-01 argument to the ingestAssistant.py command should do this automatically set this.  Testing this in a temp directory yields a curated file with the correct date for ingestCuratedCalibs.py.

            This is one of the failing points in the gen2 code: calibration date ranges are automatically but inconsistently set at various stages of the generation.  This is resolved in gen3, as the certification step explicitly requires the date ranges to be set.

            Show
            czw Christopher Waters added a comment - Looking at your repositories, I see that: sqlite3 /project/shared/auxTel/rerun/cslage/PTC_Defect_2021- 01 - 21 /CALIB/calibRegistry.sqlite3 \ "select * from defects;" id filter raftName detectorName detector calibDate validStart validEnd 1 None RXX RXX_S00 0 2021 - 01 -26T09: 15 : 21.014929 2021 - 01 - 26 2037 - 12 - 31 Comparing to the above command, it looks like you're testing on expId=2021012100538, which looks to be prior to the validStart applied to the defects.  Manually editing the sqlite UPDATE defect SET validStart='2021-01-01' WHERE id=1 is one solution.  For the future, passing a  --date 2021-01-01  argument to the ingestAssistant.py command should do this automatically set this.  Testing this in a temp directory yields a curated file with the correct date for ingestCuratedCalibs.py . This is one of the failing points in the gen2 code: calibration date ranges are automatically but inconsistently set at various stages of the generation.  This is resolved in gen3, as the certification step explicitly requires the date ranges to be set.
            Hide
            cslage Craig Lage added a comment -

            The manual sqlite edit fixed the problem.  I've had that problem before - I should have thought of that.  I haven't tried the --date option yet.

            Show
            cslage Craig Lage added a comment - The manual sqlite edit fixed the problem.  I've had that problem before - I should have thought of that.  I haven't tried the --date option yet.
            Hide
            cslage Craig Lage added a comment -

            I re-ran it with a new rerun directory with the --date 2021-01-01 option for ingestAssistant.py.  This edited the defect filename with the 2021-01-01 date, but the sqlite registry still showed 2021-01-26 for the valid date, and the ISR failed as before.

            Show
            cslage Craig Lage added a comment - I re-ran it with a new rerun directory with the --date 2021-01-01 option for ingestAssistant.py.  This edited the defect filename with the 2021-01-01 date, but the sqlite registry still showed 2021-01-26 for the valid date, and the ISR failed as before.
            Hide
            cslage Craig Lage added a comment -

            I pulled down the latest code and now the --date option is working and the ingestion and ISR ran correctly.  However now the makePhotonTransferCurve failed.  I think when I got the new code I also got the latest code there, and Andrés had warned me that there were changes because of Gen3, so I need to work on that.  I don't think I need any more at this point, but I might ask if I get stuck.

            Show
            cslage Craig Lage added a comment - I pulled down the latest code and now the --date option is working and the ingestion and ISR ran correctly.  However now the makePhotonTransferCurve failed.  I think when I got the new code I also got the latest code there, and Andrés had warned me that there were changes because of Gen3, so I need to work on that.  I don't think I need any more at this point, but I might ask if I get stuck.

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              czw Christopher Waters
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated: