Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31811

Add detection significance to footprint peaks

    XMLWordPrintable

    Details

      Description

      The DPDD includes a signal-to-noise field in the DIASource catalogs, and makes clear that this is the detection SNR rather than the output of aperture or PSF measurements:

      SNR : The signal-to-noise ratio at which this source was detected in the difference image. This is not necessarily the same as apFlux/apFluxErr, as the flux measurement algorithm maybe more accurate than the detection algorithm.

      At present we do not provide this information in our DIASource catalogs, which complicates debugging difference imaging near the detection threshold. This ticket is to propagate up the peak SNR for each detected footprint in the DIASource catalog This ticket is to calculate the detection significance in afw/meas_algorithms, so that we can propagate it up to the DIASource catalogs in a later ticket.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-32147 Investigate "noise" DIASources

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34539 Add detection significance to DIASources and apdb

          • In Review
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. DM-35302 Source detection config metadata propagated to output catalogs

          • To Do
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            9 older comments
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Post-review jenkins, with ci_hsc/ci_imsim: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36503/pipeline

            Show
            Parejkoj John Parejko added a comment - Post-review jenkins, with ci_hsc/ci_imsim: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36503/pipeline
            Hide
            Permalink
            Parejkoj John Parejko added a comment - - edited

            It looks like this change breaks ci_hsc somehow. Here's [my failed Jenkins run](https://ci.lsst.codes/job/stack-os-matrix/36503/artifact/centos-7-conda/lsstsw/build/ci_hsc_gen3/_build.log), with the relevant log below. I don't understand why this is failing in daf_butler. I'm doing a local ci_hsc build to have something to test on.

            [2022-05-02T22:38:19.095564Z] ERROR 2022-05-02T22:38:18.882+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 10, visit: 903342, ...})(singleQuantumExecutor.py:214) - Execution of task 'imageDifference' on quantum {instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 10, visit: 903342, ...} failed. Exception RuntimeError: Failed to serialize dataset goodSeeingDiff_diaSrc@{instrument: 'HSC', detector: 10, visit: 903342, ...}, sc=SourceCatalog] (id=40cc1763-883c-4ee5-91ff-b3d33e8fcc8a) of type <class 'lsst.afw.table.SourceCatalog'> to location file:///j/ws/stack-os-matrix/centos-7-conda/lsstsw/build/ci_hsc_gen3/DATA/HSC/runs/ci_hsc/20220502T221625Z/goodSeeingDiff_diaSrc/20130617/r/HSC-R/903342/goodSeeingDiff_diaSrc_HSC_r_HSC-R_903342_1_36_HSC_runs_ci_hsc_20220502T221625Z.fits
            		 
            [2022-05-02T22:38:19.105281Z] INFO 2022-05-02T22:38:19.089+00:00 lsst.imageDifference.scaleVariance (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 12, visit: 903346, ...})(scaleVariance.py:131) - Renormalizing variance by 0.004830
            		 
            [2022-05-02T22:38:19.113978Z] INFO 2022-05-02T22:38:19.103+00:00 lsst.imageDifference (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 12, visit: 903346, ...})(imageDifference.py:1072) - Diffim variance scaling factor: 0.00
            		 
            [2022-05-02T22:38:19.286223Z] WARNING 2022-05-02T22:38:19.112+00:00 py.warnings (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 12, visit: 903346, ...})(warnings.py:109) - /j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/meas_algorithms/g0f5292b13c+593c6060c9/python/lsst/meas/algorithms/detection.py:413: FutureWarning: Default position argument overload is deprecated and will be removed in version 24.0.  Please explicitly specify a position.
            		 
            [2022-05-02T22:38:19.286589Z]   sigma = psf.computeShape().getDeterminantRadius()
            		 
            [2022-05-02T22:38:19.286602Z] 
            [2022-05-02T22:38:19.289600Z] WARNING 2022-05-02T22:38:19.113+00:00 py.warnings (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 12, visit: 903346, ...})(warnings.py:109) - /j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/meas_algorithms/g0f5292b13c+593c6060c9/python/lsst/meas/algorithms/detection.py:450: FutureWarning: Default position argument overload is deprecated and will be removed in version 24.0.  Please explicitly specify a position.
            		 
            [2022-05-02T22:38:19.289936Z]   sigma = psf.computeShape().getDeterminantRadius()
            		 
            [2022-05-02T22:38:19.289948Z] 
            [2022-05-02T22:38:19.415714Z] INFO 2022-05-02T22:38:19.282+00:00 lsst.ip_diffim_decorrelateALKernel (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 22, visit: 903334, ...})(imageDecorrelation.py:214) - Using matching kernel computed at (1024, 2088)
            		 
            [2022-05-02T22:38:19.416363Z] Process task-{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 10, visit: 903342, ...}:
            		 
            [2022-05-02T22:38:19.416603Z] Traceback (most recent call last):
            		 
            [2022-05-02T22:38:19.417971Z]   File "/j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/daf_butler/g479bf8aceb+6728436f39/python/lsst/daf/butler/datastores/fileDatastore.py", line 1126, in _write_in_memory_to_artifact
            		 
            [2022-05-02T22:38:19.418207Z]     formatter.write(inMemoryDataset)
            		 
            [2022-05-02T22:38:19.419360Z]   File "/j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/daf_butler/g479bf8aceb+6728436f39/python/lsst/daf/butler/formatters/file.py", line 259, in write
            		 
            [2022-05-02T22:38:19.419606Z]     self._writeFile(inMemoryDataset)
            		 
            [2022-05-02T22:38:19.420814Z]   File "/j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/obs_base/gc18b5ea8f9+5a1f5b94fd/python/lsst/obs/base/formatters/fitsGeneric.py", line 76, in _writeFile
            		 
            [2022-05-02T22:38:19.421226Z]     inMemoryDataset.writeFits(self.fileDescriptor.location.path)
            		 
            [2022-05-02T22:38:19.421494Z] lsst.pex.exceptions.wrappers.LogicError: 
            [2022-05-02T22:38:19.422298Z]   File "src/table/BaseRecord.cc", line 124, in void lsst::afw::table::BaseRecord::assign(const lsst::afw::table::BaseRecord&)
            		 
            [2022-05-02T22:38:19.422599Z]     Unequal schemas in record assignment. {0}
            		 
            [2022-05-02T22:38:19.423075Z] lsst::pex::exceptions::LogicError: 'Unequal schemas in record assignment.'

            Show
            Parejkoj John Parejko added a comment - - edited It looks like this change breaks ci_hsc somehow. Here's [my failed Jenkins run] ( https://ci.lsst.codes/job/stack-os-matrix/36503/artifact/centos-7-conda/lsstsw/build/ci_hsc_gen3/_build.log ), with the relevant log below. I don't understand why this is failing in daf_butler. I'm doing a local ci_hsc build to have something to test on. [2022-05-02T22:38:19.095564Z] ERROR 2022-05-02T22:38:18.882+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 10, visit: 903342, ...})(singleQuantumExecutor.py:214) - Execution of task 'imageDifference' on quantum {instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 10, visit: 903342, ...} failed. Exception RuntimeError: Failed to serialize dataset goodSeeingDiff_diaSrc@{instrument: 'HSC', detector: 10, visit: 903342, ...}, sc=SourceCatalog] (id=40cc1763-883c-4ee5-91ff-b3d33e8fcc8a) of type <class 'lsst.afw.table.SourceCatalog'> to location file:///j/ws/stack-os-matrix/centos-7-conda/lsstsw/build/ci_hsc_gen3/DATA/HSC/runs/ci_hsc/20220502T221625Z/goodSeeingDiff_diaSrc/20130617/r/HSC-R/903342/goodSeeingDiff_diaSrc_HSC_r_HSC-R_903342_1_36_HSC_runs_ci_hsc_20220502T221625Z.fits [2022-05-02T22:38:19.105281Z] INFO 2022-05-02T22:38:19.089+00:00 lsst.imageDifference.scaleVariance (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 12, visit: 903346, ...})(scaleVariance.py:131) - Renormalizing variance by 0.004830 [2022-05-02T22:38:19.113978Z] INFO 2022-05-02T22:38:19.103+00:00 lsst.imageDifference (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 12, visit: 903346, ...})(imageDifference.py:1072) - Diffim variance scaling factor: 0.00 [2022-05-02T22:38:19.286223Z] WARNING 2022-05-02T22:38:19.112+00:00 py.warnings (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 12, visit: 903346, ...})(warnings.py:109) - /j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/meas_algorithms/g0f5292b13c+593c6060c9/python/lsst/meas/algorithms/detection.py:413: FutureWarning: Default position argument overload is deprecated and will be removed in version 24.0. Please explicitly specify a position. [2022-05-02T22:38:19.286589Z] sigma = psf.computeShape().getDeterminantRadius() [2022-05-02T22:38:19.286602Z] [2022-05-02T22:38:19.289600Z] WARNING 2022-05-02T22:38:19.113+00:00 py.warnings (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 12, visit: 903346, ...})(warnings.py:109) - /j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/meas_algorithms/g0f5292b13c+593c6060c9/python/lsst/meas/algorithms/detection.py:450: FutureWarning: Default position argument overload is deprecated and will be removed in version 24.0. Please explicitly specify a position. [2022-05-02T22:38:19.289936Z] sigma = psf.computeShape().getDeterminantRadius() [2022-05-02T22:38:19.289948Z] [2022-05-02T22:38:19.415714Z] INFO 2022-05-02T22:38:19.282+00:00 lsst.ip_diffim_decorrelateALKernel (imageDifference:{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 22, visit: 903334, ...})(imageDecorrelation.py:214) - Using matching kernel computed at (1024, 2088) [2022-05-02T22:38:19.416363Z] Process task-{instrument: 'HSC', skymap: 'discrete/ci_hsc', detector: 10, visit: 903342, ...}: [2022-05-02T22:38:19.416603Z] Traceback (most recent call last): [2022-05-02T22:38:19.417971Z] File "/j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/daf_butler/g479bf8aceb+6728436f39/python/lsst/daf/butler/datastores/fileDatastore.py", line 1126, in _write_in_memory_to_artifact [2022-05-02T22:38:19.418207Z] formatter.write(inMemoryDataset) [2022-05-02T22:38:19.419360Z] File "/j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/daf_butler/g479bf8aceb+6728436f39/python/lsst/daf/butler/formatters/file.py", line 259, in write [2022-05-02T22:38:19.419606Z] self._writeFile(inMemoryDataset) [2022-05-02T22:38:19.420814Z] File "/j/ws/stack-os-matrix/centos-7-conda/lsstsw/stack/lsst-scipipe-3.0.0/Linux64/obs_base/gc18b5ea8f9+5a1f5b94fd/python/lsst/obs/base/formatters/fitsGeneric.py", line 76, in _writeFile [2022-05-02T22:38:19.421226Z] inMemoryDataset.writeFits(self.fileDescriptor.location.path) [2022-05-02T22:38:19.421494Z] lsst.pex.exceptions.wrappers.LogicError: [2022-05-02T22:38:19.422298Z] File "src/table/BaseRecord.cc", line 124, in void lsst::afw::table::BaseRecord::assign(const lsst::afw::table::BaseRecord&) [2022-05-02T22:38:19.422599Z] Unequal schemas in record assignment. {0} [2022-05-02T22:38:19.423075Z] lsst::pex::exceptions::LogicError: 'Unequal schemas in record assignment.'
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Following up on this: I believe the problem is in FootprintSet.merge: it goes and re-detects all the footprints and peaks, and then merges all the Peaks into the new footprint, without looking at the schema of the old peaks.

            Show
            Parejkoj John Parejko added a comment - Following up on this: I believe the problem is in FootprintSet.merge : it goes and re-detects all the footprints and peaks, and then merges all the Peaks into the new footprint, without looking at the schema of the old peaks.
            Hide
            Permalink
            Parejkoj John Parejko added a comment - - edited

            New commit in afw that I believe fixes the problem. New Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36515/pipeline

            Show
            Parejkoj John Parejko added a comment - - edited New commit in afw that I believe fixes the problem. New Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36515/pipeline
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Fix looks fine, but I would prefer a check on schema equality, and I have a suggestion that might shrink the change a bit; see PR.

            Show
            jbosch Jim Bosch added a comment - Fix looks fine, but I would prefer a check on schema equality, and I have a suggestion that might shrink the change a bit; see PR.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              ebellm Eric Bellm
              Reviewers:
              Jim Bosch
              Watchers:
              Eric Bellm, Fred Moolekamp, Ian Sullivan, Jim Bosch, John Parejko, Nate Lust, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.