Data ManagementIcon indicating the project type

Data Management

Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21237

BBox error in some DCR image differencing templates

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP F19-5 (October), AP F19-6 (November), AP S20-1 (December)
    • Team:
      Alert Production

      Description

      Some images have been failing during image differencing using the DCR-matched template, that succeeded with a standard coadd template. These images appear to fail for patches covering the extreme edges of the area with data, but more investigation is needed. Example error message below:

       

       
      		 
      apPipe INFO: Running ImageDifference...
      		 
      apPipe.differencer INFO: Processing DataId(initialdata=\{'ccdnum': 44, 'filter': 'g', 'visit': 410929, 'date': '2015-02-17', 'object': 'Blind15A_40'}, tag=set())
      		 
      apPipe.differencer.getTemplate INFO: Using skyMap tract 0
      		 
      apPipe.differencer.getTemplate INFO: Assembling 6 coadd patches
      		 
      apPipe.differencer.getTemplate INFO: exposure dimensions=(2048, 4096); coadd dimensions=(5424, 2740)
      		 
      apPipe.differencer.getTemplate INFO: Constructing DCR-matched template for patch \{'datasetType': 'dcrCoadd_sub', 'bbox': Box2I(minimum=Point2I(60087, 34140), dimensions=Extent2I(13, 1960)), 'tract': 0, 'patch': '14,8', 'numSubfilters': 3}
      		 
      apPipe FATAL: Failed on dataId=\{'ccdnum': 44, 'filter': 'g', 'visit': 410929, 'date': '2015-02-17', 'hdu': 48, 'object': 'Blind15A_40'}: LengthError: 
       File "src/image/Image.cc", line 89, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_makeSubView(const Extent2I&, const Extent2I&, const _view_t&) [with PixelT = float; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<float, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]
      		 
       Box2I(Point2I(-60087,-34140),lsst::geom::Extent2I(0,0)) doesn't fit in image 13x1960 \{0}
      		 
      lsst::pex::exceptions::LengthError: 'Box2I(Point2I(-60087,-34140),lsst::geom::Extent2I(0,0)) doesn't fit in image 13x1960'
      		 
       
      		 
      /software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pex_config/18.1.0-4-gc96e915/python/lsst/pex/config/config.py:1276: FutureWarning: Config field ccdProcessor.isr.doAddDistortionModel is deprecated: Camera geometry is incorporated when reading the raw files. This option no longer is used, and will be removed after v19.
      		 
       FutureWarning)
      		 
      /software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ap_association/18.1.0-8-g5734da6/python/lsst/ap/association/mapApData.py:185: YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      		 
       table_list = list(yaml.load_all(yaml_stream))
      		 
      /software/lsstsw/stack_20190330/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/astropy/units/function/logarithmic.py:43: RuntimeWarning: invalid value encountered in log10
      		 
       return dex.to(self._function_unit, np.log10(x))
      		 
      Traceback (most recent call last):
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
      		 
       result = self.runTask(task, dataRef, kwargs)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
      		 
       return task.runDataRef(dataRef, **kwargs)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/timer.py", line 150, in wrapper
      		 
       res = func(self, *args, **keyArgs)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ap_pipe/18.1.0-4-gcd8f52c+10/python/lsst/ap/pipe/ap_pipe.py", line 202, in runDataRef
      		 
       diffImResults = self.runDiffIm(calexpRef, templateIds)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/timer.py", line 150, in wrapper
      		 
       res = func(self, *args, **keyArgs)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ap_pipe/18.1.0-4-gcd8f52c+10/python/lsst/ap/pipe/ap_pipe.py", line 276, in runDiffIm
      		 
       return self.differencer.runDataRef(sensorRef, templateIdList=templateIds)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/timer.py", line 150, in wrapper
      		 
       res = func(self, *args, **keyArgs)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0-15-gc153667b/python/lsst/pipe/tasks/imageDifference.py", line 353, in runDataRef
      		 
       template = self.getTemplate.run(exposure, sensorRef, templateIdList=templateIdList)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/getTemplate.py", line 153, in run
      		 
       visitInfo=exposure.getInfo().getVisitInfo())
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/dcrModel.py", line 386, in buildMatchedExposure
      		 
       bbox=bbox, wcs=wcs, mask=mask)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/dcrModel.py", line 349, in buildMatchedTemplate
      		 
       refModel = self.getReferenceImage(bbox)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/dcrModel.py", line 274, in getReferenceImage
      		 
       return np.mean([model[bbox].array for model in self], axis=0)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/dcrModel.py", line 274, in <listcomp>
      		 
       return np.mean([model[bbox].array for model in self], axis=0)
      		 
       File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/afw/18.1.0-15-gbf79fb8fa/python/lsst/afw/image/slicing.py", line 262, in __getitem__
      		 
       return self.subset(box, origin=origin)
      		 
      lsst.pex.exceptions.wrappers.LengthError: 
       File "src/image/Image.cc", line 89, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_makeSubView(const Extent2I&, const Extent2I&, const _view_t&) [with PixelT = float; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<float, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]
      		 
       Box2I(Point2I(-60087,-34140),lsst::geom::Extent2I(0,0)) doesn't fit in image 13x1960 \{0}
      		 
      lsst::pex::exceptions::LengthError: 'Box2I(Point2I(-60087,-34140),lsst::geom::Extent2I(0,0)) doesn't fit in image 13x1960'

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21230 Compare ap_pipe output using higher resolution templates

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            sullivan Ian Sullivan added a comment -

            This is a simple fix for some rare warnings we've seen when running image differencing using DCR templates. On further investigation, it looks like the problem is that a failing patch only had a thin strip of pixels with data, which was smaller than the buffer we clip from the edges. The fix is to skip the patch and print a log message.

            Show
            sullivan Ian Sullivan added a comment - This is a simple fix for some rare warnings we've seen when running image differencing using DCR templates. On further investigation, it looks like the problem is that a failing patch only had a thin strip of pixels with data, which was smaller than the buffer we clip from the edges. The fix is to skip the patch and print a log message.
            Hide
            Permalink
            gkovacs Gabor Kovacs added a comment -

            Looks all right.

            Show
            gkovacs Gabor Kovacs added a comment - Looks all right.

              People

              • Assignee:
                sullivan Ian Sullivan
                Reporter:
                sullivan Ian Sullivan
                Reviewers:
                Gabor Kovacs
                Watchers:
                Gabor Kovacs, Ian Sullivan
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: