[DM-28346] Gen2 defects no longer ingestable as a curated calibration - Jira

XML

Word

Printable

Details

Type: Bug
Status: Won't Fix
Resolution: Done
Fix Version/s: None
Component/s: cp_pipe
Labels:
None

Epic Link:
f21a-drp-isr
Team:
Data Release Production
Urgent?:
No

Description

It did in fact used to work correctly in that ingestCuratedCalibs.py could read the output of findDefects.py. I pulled up the w_2020_40 code, and there was a function in cp/pipe/defects.py called _writeData that got called at the end of the FindDefectTask. This wrote things out with the correct file structure so that ingestCuratedCalibs.py could read it. I looked at an output file from that time frame and there is a statement at the end of findDefects.py that says:
findDefects INFO: Writing defects to /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12638/calibrations/LSSTCam/defects/r01_s00/2020-10-13T01:07:04.651000 in format: BOTH
This is the directory structure that ingestCuratedCalibs.py is looking for. When I pull up the current code, that _writeData function is gone, so the output gets written into the defects directory and ingestCuratedCalibs.py fails when it tries to read it. Perhaps I'm going about things the wrong way. I'm happy to do things a different way if you can point me in the right direction.

Attachments

Issue Links

relates to

DM-29581 runISR.py can't find defect files

Invalid

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Craig Lage added a comment - 26/Jan/21 5:37 PM

I pulled down tickets/~~DM-28346~~ on cp_pipe and ip_isr to test out the fix. I ran it on both ComCam and AuxTel data., and both failed in the same way, as detailed below for the auxTel. I ran the following steps, and with these the defects files were appropriately created and ingested:

findDefects.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --id detector=0 expId=2021012100660 -c measure.badOnAndOffPixelColumnThreshold=500 --clobber-config --clobber-versions

ingestAssistant.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/curated /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/defects/*/*.fits

ingestCuratedCalibs.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/curated/latiss/defects --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --config clobber=True --ignore-ingested

The appropriate defect files were then in the CALIB directory, with the file names like this:

/project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB/defects/2021-01-26T09\:15\:21.014929/defects-2021-01-26T09\:15\:21.014929-RXX-RXX_S00-000.fits

However, when I tried to run ISR, it was unable to find the defect files. This command, with doDefect=False successfully ran:

runIsr.py /project/shared/auxTel/ --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --id detector=0 expId=2021012100538 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1

But this command, with doDefect=True, gave a 'No registry for lookup' error. I think there is a problem with the filename and it is not finding it:

runIsr.py /project/shared/auxTel/ --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --id detector=0 expId=2021012100538 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1

Here is the full error log:

runIsr FATAL: Failed on dataId={'detector': 0, 'expId': 2021012100538, 'dayObs': '2021-01-21'}: RuntimeError: No registry for lookup

Traceback (most recent call last):

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/cmdLineTask.py", line 431, in __call__

    result = self.runTask(task, dataRef, kwargs)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/cmdLineTask.py", line 495, in runTask

    return task.runDataRef(dataRef, **kwargs)

  File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 2621, in runDataRef

    return self.isr.runDataRef(dataRef)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/timer.py", line 157, in wrapper

    res = func(self, *args, **keyArgs)

  File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 1684, in runDataRef

    isrData = self.readIsrData(sensorRef, ccdExposure)

  File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 1121, in readIsrData

    defectList = (dataRef.get("defects")

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butlerSubset.py", line 203, in get

    return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butler.py", line 1397, in get

    location = self._locate(datasetType, dataId, write=False)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butler.py", line 1316, in _locate

    location = repoData.repo.map(datasetType, dataId, write=write)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/repository.py", line 239, in map

    loc = self._mapper.map(*args, **kwargs)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/mapper.py", line 163, in map

    return func(self.validate(dataId), write)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/cameraMapper.py", line 391, in mapClosure

    return mapping.map(mapper, dataId, write)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 460, in map

    location = Mapping.map(self, mapper, dataId, write=write)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 155, in map

    actualId = self.need(iter(self.keyDict.keys()), dataId)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 322, in need

    lookups = self.lookup(newProps, newId)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 519, in lookup

    return Mapping.lookup(self, properties, newId)

  File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 209, in lookup

    raise RuntimeError("No registry for lookup")

RuntimeError: No registry for lookup

Here's the ISR command for the ComCam, if that is easier to deal with:

runIsr.py /project/shared/comCam-CCS --rerun /project/shared/comCam-CCS/rerun/cslage/PTCTemp_2020-12-29 --calib /project/shared/comCam-CCS/rerun/cslage/PTCTemp_2020-12-29/CALIB --id detector=0 expId=3020122900043 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doSaturation=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1

Show

Craig Lage added a comment - 26/Jan/21 5:37 PM I pulled down tickets/ DM-28346 on cp_pipe and ip_isr to test out the fix. I ran it on both ComCam and AuxTel data., and both failed in the same way, as detailed below for the auxTel. I ran the following steps, and with these the defects files were appropriately created and ingested: findDefects.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --id detector=0 expId=2021012100660 -c measure.badOnAndOffPixelColumnThreshold=500 --clobber-config --clobber-versions ingestAssistant.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/curated /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/defects/*/*.fits ingestCuratedCalibs.py /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/curated/latiss/defects --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --config clobber=True --ignore-ingested The appropriate defect files were then in the CALIB directory, with the file names like this: /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB/defects/2021-01-26T09\:15\:21.014929/defects-2021-01-26T09\:15\:21.014929-RXX-RXX_S00-000.fits However, when I tried to run ISR, it was unable to find the defect files. This command, with doDefect=False successfully ran: runIsr.py /project/shared/auxTel/ --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --id detector=0 expId=2021012100538 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1 But this command, with doDefect=True, gave a 'No registry for lookup' error. I think there is a problem with the filename and it is not finding it: runIsr.py /project/shared/auxTel/ --rerun /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21 --calib /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB --id detector=0 expId=2021012100538 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1 Here is the full error log: runIsr FATAL: Failed on dataId={'detector': 0, 'expId': 2021012100538, 'dayObs': '2021-01-21'}: RuntimeError: No registry for lookup Traceback (most recent call last): File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/cmdLineTask.py", line 431, in __call__ result = self.runTask(task, dataRef, kwargs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/cmdLineTask.py", line 495, in runTask return task.runDataRef(dataRef, **kwargs) File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 2621, in runDataRef return self.isr.runDataRef(dataRef) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+55f5a6a5c9/python/lsst/pipe/base/timer.py", line 157, in wrapper res = func(self, *args, **keyArgs) File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 1684, in runDataRef isrData = self.readIsrData(sensorRef, ccdExposure) File "/lsstdata/user/staff/jhome/cslage/alternate_branches/ip_isr/python/lsst/ip/isr/isrTask.py", line 1121, in readIsrData defectList = (dataRef.get("defects") File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butlerSubset.py", line 203, in get return self.butlerSubset.butler.get(datasetType, self.dataId, **rest) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butler.py", line 1397, in get location = self._locate(datasetType, dataId, write=False) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/butler.py", line 1316, in _locate location = repoData.repo.map(datasetType, dataId, write=write) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/repository.py", line 239, in map loc = self._mapper.map(*args, **kwargs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/daf_persistence/21.0.0-4-g5873dc9+9a92674037/python/lsst/daf/persistence/mapper.py", line 163, in map return func(self.validate(dataId), write) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/cameraMapper.py", line 391, in mapClosure return mapping.map(mapper, dataId, write) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 460, in map location = Mapping.map(self, mapper, dataId, write=write) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 155, in map actualId = self.need(iter(self.keyDict.keys()), dataId) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 322, in need lookups = self.lookup(newProps, newId) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 519, in lookup return Mapping.lookup(self, properties, newId) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/obs_base/21.0.0-19-g7c7630f+a88ebbf2d9/python/lsst/obs/base/mapping.py", line 209, in lookup raise RuntimeError("No registry for lookup") RuntimeError: No registry for lookup Here's the ISR command for the ComCam, if that is easier to deal with: runIsr.py /project/shared/comCam-CCS --rerun /project/shared/comCam-CCS/rerun/cslage/PTCTemp_2020-12-29 --calib /project/shared/comCam-CCS/rerun/cslage/PTCTemp_2020-12-29/CALIB --id detector=0 expId=3020122900043 -c isr.doBias=True isr.doDark=False isr.doFringe=False isr.doSuspect=True isr.doDefect=True isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 isr.doFlat=False isr.doSaturation=False isr.doCrosstalk=False isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 --clobber-config --clobber-versions -j 1

Hide

Permalink

Christopher Waters added a comment - 26/Jan/21 7:36 PM

Looking at your repositories, I see that:

sqlite3 /project/shared/auxTel/rerun/cslage/PTC_Defect_2021-01-21/CALIB/calibRegistry.sqlite3 \

    "select * from defects;"

id      filter  raftName detectorName  detector        calibDate               validStart      validEnd

1       None    RXX     RXX_S00        0       2021-01-26T09:15:21.014929      2021-01-26      2037-12-31

Comparing to the above command, it looks like you're testing on expId=2021012100538, which looks to be prior to the validStart applied to the defects. Manually editing the sqlite UPDATE defect SET validStart='2021-01-01' WHERE id=1 is one solution. For the future, passing a --date 2021-01-01 argument to the ingestAssistant.py command should do this automatically set this. Testing this in a temp directory yields a curated file with the correct date for ingestCuratedCalibs.py.

This is one of the failing points in the gen2 code: calibration date ranges are automatically but inconsistently set at various stages of the generation. This is resolved in gen3, as the certification step explicitly requires the date ranges to be set.

Show

Christopher Waters added a comment - 26/Jan/21 7:36 PM Looking at your repositories, I see that: sqlite3 /project/shared/auxTel/rerun/cslage/PTC_Defect_2021- 01 - 21 /CALIB/calibRegistry.sqlite3 \ "select * from defects;" id filter raftName detectorName detector calibDate validStart validEnd 1 None RXX RXX_S00 0 2021 - 01 -26T09: 15 : 21.014929 2021 - 01 - 26 2037 - 12 - 31 Comparing to the above command, it looks like you're testing on expId=2021012100538, which looks to be prior to the validStart applied to the defects. Manually editing the sqlite UPDATE defect SET validStart='2021-01-01' WHERE id=1 is one solution. For the future, passing a --date 2021-01-01 argument to the ingestAssistant.py command should do this automatically set this. Testing this in a temp directory yields a curated file with the correct date for ingestCuratedCalibs.py . This is one of the failing points in the gen2 code: calibration date ranges are automatically but inconsistently set at various stages of the generation. This is resolved in gen3, as the certification step explicitly requires the date ranges to be set.

Hide

Permalink

Craig Lage added a comment - 26/Jan/21 9:54 PM

The manual sqlite edit fixed the problem. I've had that problem before - I should have thought of that. I haven't tried the --date option yet.

Show

Craig Lage added a comment - 26/Jan/21 9:54 PM The manual sqlite edit fixed the problem. I've had that problem before - I should have thought of that. I haven't tried the --date option yet.

Hide

Permalink

Craig Lage added a comment - 26/Jan/21 10:21 PM

I re-ran it with a new rerun directory with the --date 2021-01-01 option for ingestAssistant.py. This edited the defect filename with the 2021-01-01 date, but the sqlite registry still showed 2021-01-26 for the valid date, and the ISR failed as before.

Show

Craig Lage added a comment - 26/Jan/21 10:21 PM I re-ran it with a new rerun directory with the --date 2021-01-01 option for ingestAssistant.py. This edited the defect filename with the 2021-01-01 date, but the sqlite registry still showed 2021-01-26 for the valid date, and the ISR failed as before.

Hide

Permalink

Craig Lage added a comment - 27/Jan/21 1:24 PM

I pulled down the latest code and now the --date option is working and the ingestion and ISR ran correctly. However now the makePhotonTransferCurve failed. I think when I got the new code I also got the latest code there, and Andrés had warned me that there were changes because of Gen3, so I need to work on that. I don't think I need any more at this point, but I might ask if I get stuck.

Show

Craig Lage added a comment - 27/Jan/21 1:24 PM I pulled down the latest code and now the --date option is working and the ingestion and ISR ran correctly. However now the makePhotonTransferCurve failed. I think when I got the new code I also got the latest code there, and Andrés had warned me that there were changes because of Gen3, so I need to work on that. I don't think I need any more at this point, but I might ask if I get stuck.

Hide

Permalink

Christopher Waters added a comment - 06/Jul/21 9:07 PM

I'm setting this to Won't Fix as we're well on the way to using Gen3 everywhere, and it seems like this isn't an issue any more.

Show

Christopher Waters added a comment - 06/Jul/21 9:07 PM I'm setting this to Won't Fix as we're well on the way to using Gen3 everywhere, and it seems like this isn't an issue any more.

People

Assignee:: Christopher Waters

Reporter:: Christopher Waters

Watchers:: Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 14/Jan/21 5:51 PM

Updated:: 06/Jul/21 9:07 PM

Resolved:: 06/Jul/21 9:07 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins