Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17923

Inconsistent source detection workflow with difference imaging

    Details

    • Templates:
    • Epic Link:
    • Team:
      Alert Production

      Description

      If a user wants to do difference imaging with some calexps as science images and some coadds as templates and decides to run ImageDifferenceTask with doSelectSources=True (the default!), the result is an error about how template sources are not available. Differencing works fine with doSelectSources=False.

      The standard AP workflow doesn't include running detectCoaddSources, because we do source detection on the difference images. I may be wrong, but I don't think there is any benefit to the result of image differencing a catalog of sources previously detected in the template coadd exists. This problem is easy to encounter and difficult to troubleshoot if you haven't seen it before. A sample traceback thanks to Hayden Smotherman is below.

      A resolution to this ticket can be one of (1) convince me I should be doing source detection on my templates before running difference imaging, (2) change the default to doSelectSources=False, OR (3) Your Idea Welcome Here.

       

      imageDifference INFO: Processing DataId(initialdata={'visit': 433932, 'ccdnum': 12, 'filter': 'g'}, tag=set())
      imageDifference.getTemplate INFO: Using skyMap tract 0
      imageDifference.getTemplate INFO: Assembling 2 coadd patches
      imageDifference.getTemplate INFO: exposure dimensions=(2046, 4094); coadd dimensions=(3247, 1644)
      imageDifference.getTemplate INFO: Reading patch {'datasetType': 'deepCoadd_sub', 'bbox': Box2I(minimum=Point2I(3666, 17405), dimensions=Extent2I(434, 1644)), 'tract': 0, 'patch': '0,4', 'numSubfilters': 3}
      imageDifference.getTemplate INFO: Reading patch {'datasetType': 'deepCoadd_sub', 'bbox': Box2I(minimum=Point2I(3900, 17405), dimensions=Extent2I(3013, 1644)), 'tract': 0, 'patch': '1,4', 'numSubfilters': 3}
      imageDifference INFO: Source selection via src product
      imageDifference FATAL: Failed on dataId=DataId(initialdata={'visit': 433932, 'ccdnum': 12, 'filter': 'g'}, tag=set()): RuntimeError: doSelectSources=True and kernelSourcesFromRef=False,but template sources not available. Cannot match science sources with template sources. Run process* on data from which templates are built.
      Traceback (most recent call last):
        File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-30-g6787e8a+1/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-30-g6787e8a+1/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-30-g6787e8a+1/python/lsst/pipe/base/timer.py", line 149, in wrapper
          res = func(self, *args, **keyArgs)
        File "/astro/store/epyc/users/smotherh/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_tasks/16.0-61-gb2b2650a/python/lsst/pipe/tasks/imageDifference.py", line 432, in runDataRef
          raise RuntimeError("doSelectSources=True and kernelSourcesFromRef=False,"
      RuntimeError: doSelectSources=True and kernelSourcesFromRef=False,but template sources not available. Cannot match science sources with template sources. Run process* on data from which templates are built.
      

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  mrawls Meredith Rawls
                  Watchers:
                  Colin Slater, Eric Bellm, Gabor Kovacs, John Swinbank, Meredith Rawls
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Summary Panel