Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19874

Add support for decam illumcor calibration products

    Details

      Description

      Partway through DM-18417, it became clear that illumination correction appears to be needed to eliminate false positive sources from difference imaging on nights with high sky brightness. While illumination correction is a standard step in the DECam community pipeline (CP), and "illumcor" CP data products exist for the HiTS2015 dataset, we have never used them with the LSST Science Pipelines.

      This ticket is to add the ability to ingest, and subsequently use during ISR, the illumination correction calibration products provided by the DECam community pipeline. They are multi-extension FITS files much like the "cpBIAS" and "cpFLAT" masterCals we have been using, and should be able to be ingested and used in a similar way.

      There will be Mappers.

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            The present issue is shown below, when run on lest-dev from /project/mrawls/illumcor_party. It appears that the new calib_hdu column is not making it all the way into the dataRef.

            (lsst-scipipe) [mrawls@lsst-dev01 illumcor_party]$ processCcd.py . --calib calib6 --rerun test_illumcor8 --id visit=411420 ccdnum=20 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py -c calibrate.doAstrometry=False calibrate.doPhotoCal=False isr.doDefect=False --clobber-versions
            root INFO: Loading config overrride file '/project/mrawls/obs_decam/config/processCcd.py'
            CameraMapper INFO: Loading exposure registry from /project/mrawls/illumcor_party/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /project/mrawls/illumcor_party/calib6/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /project/mrawls/illumcor_party/calib6/calibRegistry.sqlite3
            root INFO: Running: /project/mrawls/pipe_tasks/bin/processCcd.py . --calib calib6 --rerun test_illumcor8 --id visit=411420 ccdnum=20 -C /project/mrawls/obs_decam/config/processCcdCpIsr.py -c calibrate.doAstrometry=False calibrate.doPhotoCal=False isr.doDefect=False --clobber-versions
            WARNING: You are using OpenBLAS with multiple threads (24), but have not
            specified the number of threads using one of the OpenBLAS environment variables:
            OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
            This may indicate that you are unintentionally using multiple threads, which may
            cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
            what you are doing and want threads enabled implicitly, set the environment
            variable LSST_ALLOW_IMPLICIT_THREADS.
            processCcd INFO: Processing {'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}
            processCcd.isr INFO: Performing ISR on sensor {'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}
            CameraMapper WARN: I can only set the VisitInfo if you provide a mapper
            CameraMapper WARN: argDict[exposureId] is None; stripping
            CameraMapper INFO: Loading exposure registry from ./registry.sqlite3
            obs.decam.DecamCrosstalkConfig INFO: Reading crosstalk coefficient data
            CameraMapper WARN: I can only set the VisitInfo if you provide a mapper
            CameraMapper WARN: I can only set the VisitInfo if you provide a mapper
            processCcd.isr.crosstalk INFO: Correcting victim 20A from crosstalk source 19A
            processCcd.isr.crosstalk INFO: Correcting source 19A overscan before using to correct crosstalk
            CameraMapper WARN: I can only set the VisitInfo if you provide a mapper
            processCcd.isr.crosstalk INFO: Correcting victim 20A from crosstalk source 19B
            processCcd.isr.crosstalk INFO: Correcting source 19B overscan before using to correct crosstalk
            CameraMapper WARN: I can only set the VisitInfo if you provide a mapper
            processCcd.isr.crosstalk INFO: Correcting victim 20A from crosstalk source 20B
            processCcd.isr.crosstalk INFO: Correcting source 20B overscan before using to correct crosstalk
            CameraMapper WARN: I can only set the VisitInfo if you provide a mapper
            processCcd.isr.crosstalk INFO: Correcting victim 20B from crosstalk source 19B
            processCcd.isr.crosstalk INFO: Correcting source 19B overscan before using to correct crosstalk
            CameraMapper WARN: I can only set the VisitInfo if you provide a mapper
            processCcd.isr.crosstalk INFO: Correcting victim 20B from crosstalk source 20A
            processCcd.isr.crosstalk INFO: Correcting source 20A overscan before using to correct crosstalk
            CameraMapper WARN: argDict[exposureId] is None; stripping
            processCcd FATAL: Failed on dataId={'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}: RuntimeError: Unable to retrieve cpIllumcor for {'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}: No locations for get: datasetType:cpIllumcor dataId:DataId(initialdata={'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}, tag=set())
            Traceback (most recent call last):
              File "/project/mrawls/ip_isr/python/lsst/ip/isr/isrTask.py", line 1471, in getIsrExposure
                exp = dataRef.get(datasetType, immediate=immediate)
              File "/project/mrawls/daf_persistence/python/lsst/daf/persistence/butlerSubset.py", line 206, in get
                return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
              File "/project/mrawls/daf_persistence/python/lsst/daf/persistence/butler.py", line 1380, in get
                raise NoResults("No locations for get:", datasetType, dataId)
            lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:cpIllumcor dataId:DataId(initialdata={'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}, tag=set())
             
            During handling of the above exception, another exception occurred:
             
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-3-g924bd39+2/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
                result = self.runTask(task, dataRef, kwargs)
              File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-3-g924bd39+2/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
                return task.runDataRef(dataRef, **kwargs)
              File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-3-g924bd39+2/python/lsst/pipe/base/timer.py", line 150, in wrapper
                res = func(self, *args, **keyArgs)
              File "/project/mrawls/pipe_tasks/python/lsst/pipe/tasks/processCcd.py", line 184, in runDataRef
                exposure = self.isr.runDataRef(sensorRef).exposure
              File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-3-g924bd39+2/python/lsst/pipe/base/timer.py", line 150, in wrapper
                res = func(self, *args, **keyArgs)
              File "/project/mrawls/ip_isr/python/lsst/ip/isr/isrTask.py", line 1432, in runDataRef
                isrData = self.readIsrData(sensorRef, ccdExposure)
              File "/project/mrawls/ip_isr/python/lsst/ip/isr/isrTask.py", line 970, in readIsrData
                filterName in self.config.illumFilters)
              File "/project/mrawls/ip_isr/python/lsst/ip/isr/isrTask.py", line 1474, in getIsrExposure
                raise RuntimeError("Unable to retrieve %s for %s: %s" % (datasetType, dataRef.dataId, exc1))
            RuntimeError: Unable to retrieve cpIllumcor for {'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}: No locations for get: datasetType:cpIllumcor dataId:DataId(initialdata={'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}, tag=set())
            

             As a side note, I realized I was editing some things in decamCpIsr.py, but it is no longer used as far as I can tell, since it purports to import obs_decam's own isr.py which no longer exists. And some of the changes I made in there on this ticket branch are completely wrong, so we should probably just delete the file outright.

            Show
            mrawls Meredith Rawls added a comment - The present issue is shown below, when run on lest-dev from /project/mrawls/illumcor_party . It appears that the new  calib_hdu  column is not making it all the way into the dataRef . (lsst-scipipe) [mrawls@lsst-dev01 illumcor_party]$ processCcd.py . --calib calib6 --rerun test_illumcor8 --id visit=411420 ccdnum=20 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py -c calibrate.doAstrometry=False calibrate.doPhotoCal=False isr.doDefect=False --clobber-versions root INFO: Loading config overrride file '/project/mrawls/obs_decam/config/processCcd.py' CameraMapper INFO: Loading exposure registry from /project/mrawls/illumcor_party/registry.sqlite3 CameraMapper INFO: Loading calib registry from /project/mrawls/illumcor_party/calib6/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /project/mrawls/illumcor_party/calib6/calibRegistry.sqlite3 root INFO: Running: /project/mrawls/pipe_tasks/bin/processCcd.py . --calib calib6 --rerun test_illumcor8 --id visit=411420 ccdnum=20 -C /project/mrawls/obs_decam/config/processCcdCpIsr.py -c calibrate.doAstrometry=False calibrate.doPhotoCal=False isr.doDefect=False --clobber-versions WARNING: You are using OpenBLAS with multiple threads (24), but have not specified the number of threads using one of the OpenBLAS environment variables: OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS. This may indicate that you are unintentionally using multiple threads, which may cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know what you are doing and want threads enabled implicitly, set the environment variable LSST_ALLOW_IMPLICIT_THREADS. processCcd INFO: Processing {'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'} processCcd.isr INFO: Performing ISR on sensor {'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'} CameraMapper WARN: I can only set the VisitInfo if you provide a mapper CameraMapper WARN: argDict[exposureId] is None; stripping CameraMapper INFO: Loading exposure registry from ./registry.sqlite3 obs.decam.DecamCrosstalkConfig INFO: Reading crosstalk coefficient data CameraMapper WARN: I can only set the VisitInfo if you provide a mapper CameraMapper WARN: I can only set the VisitInfo if you provide a mapper processCcd.isr.crosstalk INFO: Correcting victim 20A from crosstalk source 19A processCcd.isr.crosstalk INFO: Correcting source 19A overscan before using to correct crosstalk CameraMapper WARN: I can only set the VisitInfo if you provide a mapper processCcd.isr.crosstalk INFO: Correcting victim 20A from crosstalk source 19B processCcd.isr.crosstalk INFO: Correcting source 19B overscan before using to correct crosstalk CameraMapper WARN: I can only set the VisitInfo if you provide a mapper processCcd.isr.crosstalk INFO: Correcting victim 20A from crosstalk source 20B processCcd.isr.crosstalk INFO: Correcting source 20B overscan before using to correct crosstalk CameraMapper WARN: I can only set the VisitInfo if you provide a mapper processCcd.isr.crosstalk INFO: Correcting victim 20B from crosstalk source 19B processCcd.isr.crosstalk INFO: Correcting source 19B overscan before using to correct crosstalk CameraMapper WARN: I can only set the VisitInfo if you provide a mapper processCcd.isr.crosstalk INFO: Correcting victim 20B from crosstalk source 20A processCcd.isr.crosstalk INFO: Correcting source 20A overscan before using to correct crosstalk CameraMapper WARN: argDict[exposureId] is None; stripping processCcd FATAL: Failed on dataId={'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}: RuntimeError: Unable to retrieve cpIllumcor for {'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}: No locations for get: datasetType:cpIllumcor dataId:DataId(initialdata={'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}, tag=set()) Traceback (most recent call last): File "/project/mrawls/ip_isr/python/lsst/ip/isr/isrTask.py", line 1471, in getIsrExposure exp = dataRef.get(datasetType, immediate=immediate) File "/project/mrawls/daf_persistence/python/lsst/daf/persistence/butlerSubset.py", line 206, in get return self.butlerSubset.butler.get(datasetType, self.dataId, **rest) File "/project/mrawls/daf_persistence/python/lsst/daf/persistence/butler.py", line 1380, in get raise NoResults("No locations for get:", datasetType, dataId) lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:cpIllumcor dataId:DataId(initialdata={'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}, tag=set())   During handling of the above exception, another exception occurred:   Traceback (most recent call last): File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-3-g924bd39+2/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__ result = self.runTask(task, dataRef, kwargs) File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-3-g924bd39+2/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask return task.runDataRef(dataRef, **kwargs) File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-3-g924bd39+2/python/lsst/pipe/base/timer.py", line 150, in wrapper res = func(self, *args, **keyArgs) File "/project/mrawls/pipe_tasks/python/lsst/pipe/tasks/processCcd.py", line 184, in runDataRef exposure = self.isr.runDataRef(sensorRef).exposure File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-3-g924bd39+2/python/lsst/pipe/base/timer.py", line 150, in wrapper res = func(self, *args, **keyArgs) File "/project/mrawls/ip_isr/python/lsst/ip/isr/isrTask.py", line 1432, in runDataRef isrData = self.readIsrData(sensorRef, ccdExposure) File "/project/mrawls/ip_isr/python/lsst/ip/isr/isrTask.py", line 970, in readIsrData filterName in self.config.illumFilters) File "/project/mrawls/ip_isr/python/lsst/ip/isr/isrTask.py", line 1474, in getIsrExposure raise RuntimeError("Unable to retrieve %s for %s: %s" % (datasetType, dataRef.dataId, exc1)) RuntimeError: Unable to retrieve cpIllumcor for {'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}: No locations for get: datasetType:cpIllumcor dataId:DataId(initialdata={'visit': 411420, 'ccdnum': 20, 'date': '2015-02-18', 'filter': 'g', 'hdu': 8, 'object': 'Blind15A_40'}, tag=set())  As a side note, I realized I was editing some things in decamCpIsr.py , but it is no longer used as far as I can tell, since it purports to import obs_decam 's own isr.py  which no longer exists. And some of the changes I made in there on this ticket branch are completely wrong, so we should probably just delete the file outright.
            Hide
            mrawls Meredith Rawls added a comment -

            I have implemented ingestion of decam-community-pipeline "illumcor" data products and tested that manually turning on illumination correction during ISR will work. Many thanks to help from Colin Slater.

            A full ap_pipe run (processing, image differencing, and association) with illumination correction turned on is available on lsst-dev in /project/mrawls/illumcor_party/rerun/cw_illumcor3.

            This passed Jenkins a little while ago and I have since rebased pipe_tasks and obs_decam, so I will do a final Jenkins run after the review is complete.

            Show
            mrawls Meredith Rawls added a comment - I have implemented ingestion of decam-community-pipeline "illumcor" data products and tested that manually turning on illumination correction during ISR will work. Many thanks to help from Colin Slater . A full ap_pipe run (processing, image differencing, and association) with illumination correction turned on is available on lsst-dev in /project/mrawls/illumcor_party/rerun/cw_illumcor3 . This passed Jenkins a little while ago and I have since rebased pipe_tasks and obs_decam, so I will do a final Jenkins run after the review is complete.
            Hide
            czw Christopher Waters added a comment -

            This looks fine to me, other than some ISR ordering issues for where the illumination correction is applied.  Shifting the IC shouldn't change the DECam results, but will keep things consistent wrt interpolation.

            Show
            czw Christopher Waters added a comment - This looks fine to me, other than some ISR ordering issues for where the illumination correction is applied.  Shifting the IC shouldn't change the DECam results, but will keep things consistent wrt interpolation.
            Hide
            mrawls Meredith Rawls added a comment -

            Thank you for the fast review! I will move illumination correction to before interpolation, verify the output is unchanged, and run Jenkins one final time before merging.

            Show
            mrawls Meredith Rawls added a comment - Thank you for the fast review! I will move illumination correction to before interpolation, verify the output is unchanged, and run Jenkins one final time before merging.

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                mrawls Meredith Rawls
                Reviewers:
                Christopher Waters
                Watchers:
                Christopher Waters, Colin Slater, John Swinbank, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel