[DM-21237] BBox error in some DCR image differencing templates - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: ip_diffim
Labels:
- SciencePipelines

Story Points:
2
Epic Link:
f19b-dcr-cleanup
Sprint:
AP F19-5 (October), AP F19-6 (November), AP S20-1 (December)
Team:
Alert Production

Description

Some images have been failing during image differencing using the DCR-matched template, that succeeded with a standard coadd template. These images appear to fail for patches covering the extreme edges of the area with data, but more investigation is needed. Example error message below:

apPipe INFO: Running ImageDifference...

apPipe.differencer INFO: Processing DataId(initialdata=\{'ccdnum': 44, 'filter': 'g', 'visit': 410929, 'date': '2015-02-17', 'object': 'Blind15A_40'}, tag=set())

apPipe.differencer.getTemplate INFO: Using skyMap tract 0

apPipe.differencer.getTemplate INFO: Assembling 6 coadd patches

apPipe.differencer.getTemplate INFO: exposure dimensions=(2048, 4096); coadd dimensions=(5424, 2740)

apPipe.differencer.getTemplate INFO: Constructing DCR-matched template for patch \{'datasetType': 'dcrCoadd_sub', 'bbox': Box2I(minimum=Point2I(60087, 34140), dimensions=Extent2I(13, 1960)), 'tract': 0, 'patch': '14,8', 'numSubfilters': 3}

apPipe FATAL: Failed on dataId=\{'ccdnum': 44, 'filter': 'g', 'visit': 410929, 'date': '2015-02-17', 'hdu': 48, 'object': 'Blind15A_40'}: LengthError:

 File "src/image/Image.cc", line 89, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_makeSubView(const Extent2I&, const Extent2I&, const _view_t&) [with PixelT = float; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<float, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]

 Box2I(Point2I(-60087,-34140),lsst::geom::Extent2I(0,0)) doesn't fit in image 13x1960 \{0}

lsst::pex::exceptions::LengthError: 'Box2I(Point2I(-60087,-34140),lsst::geom::Extent2I(0,0)) doesn't fit in image 13x1960'

/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pex_config/18.1.0-4-gc96e915/python/lsst/pex/config/config.py:1276: FutureWarning: Config field ccdProcessor.isr.doAddDistortionModel is deprecated: Camera geometry is incorporated when reading the raw files. This option no longer is used, and will be removed after v19.

 FutureWarning)

/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ap_association/18.1.0-8-g5734da6/python/lsst/ap/association/mapApData.py:185: YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

 table_list = list(yaml.load_all(yaml_stream))

/software/lsstsw/stack_20190330/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/astropy/units/function/logarithmic.py:43: RuntimeWarning: invalid value encountered in log10

 return dex.to(self._function_unit, np.log10(x))

Traceback (most recent call last):

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__

 result = self.runTask(task, dataRef, kwargs)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask

 return task.runDataRef(dataRef, **kwargs)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/timer.py", line 150, in wrapper

 res = func(self, *args, **keyArgs)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ap_pipe/18.1.0-4-gcd8f52c+10/python/lsst/ap/pipe/ap_pipe.py", line 202, in runDataRef

 diffImResults = self.runDiffIm(calexpRef, templateIds)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/timer.py", line 150, in wrapper

 res = func(self, *args, **keyArgs)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ap_pipe/18.1.0-4-gcd8f52c+10/python/lsst/ap/pipe/ap_pipe.py", line 276, in runDiffIm

 return self.differencer.runDataRef(sensorRef, templateIdList=templateIds)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0-4-g6c9d669+5/python/lsst/pipe/base/timer.py", line 150, in wrapper

 res = func(self, *args, **keyArgs)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0-15-gc153667b/python/lsst/pipe/tasks/imageDifference.py", line 353, in runDataRef

 template = self.getTemplate.run(exposure, sensorRef, templateIdList=templateIdList)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/getTemplate.py", line 153, in run

 visitInfo=exposure.getInfo().getVisitInfo())

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/dcrModel.py", line 386, in buildMatchedExposure

 bbox=bbox, wcs=wcs, mask=mask)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/dcrModel.py", line 349, in buildMatchedTemplate

 refModel = self.getReferenceImage(bbox)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/dcrModel.py", line 274, in getReferenceImage

 return np.mean([model[bbox].array for model in self], axis=0)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/ip_diffim/18.1.0-9-gae7190a+1/python/lsst/ip/diffim/dcrModel.py", line 274, in <listcomp>

 return np.mean([model[bbox].array for model in self], axis=0)

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/afw/18.1.0-15-gbf79fb8fa/python/lsst/afw/image/slicing.py", line 262, in __getitem__

 return self.subset(box, origin=origin)

lsst.pex.exceptions.wrappers.LengthError:

 File "src/image/Image.cc", line 89, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_makeSubView(const Extent2I&, const Extent2I&, const _view_t&) [with PixelT = float; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<float, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]

 Box2I(Point2I(-60087,-34140),lsst::geom::Extent2I(0,0)) doesn't fit in image 13x1960 \{0}

lsst::pex::exceptions::LengthError: 'Box2I(Point2I(-60087,-34140),lsst::geom::Extent2I(0,0)) doesn't fit in image 13x1960'

Attachments

Issue Links

relates to

DM-21230 Compare ap_pipe output using higher resolution templates

Done

Activity

Ascending order - Click to sort in descending order

Ian Sullivan created issue - 10/Sep/19 5:59 PM

Ian Sullivan made changes - 10/Sep/19 5:59 PM

Field	Original Value	New Value
Epic Link		~~DM-19980~~ [ 307532 ]

Ian Sullivan made changes - 10/Sep/19 5:59 PM

Link

This issue relates to ~~DM-21230~~ [ ~~DM-21230~~ ]

John Swinbank made changes - 13/Sep/19 10:12 PM

Sprint

AP F19-3 [ 941 ]

John Swinbank made changes - 13/Sep/19 10:12 PM

Rank

Ranked higher

John Swinbank made changes - 25/Sep/19 10:48 PM

Epic Link

~~DM-19980~~ [ 307532 ]

~~DM-21438~~ [ 423045 ]

John Swinbank made changes - 27/Sep/19 4:28 PM

Sprint

AP F19-5 (October) [ 957 ]

Eric Bellm made changes - 06/Nov/19 9:21 PM

Sprint

AP F19-5 (October) [ 957 ]

AP F19-5 (October), AP F19-6 (November) [ 957, 958 ]

John Swinbank made changes - 09/Dec/19 7:49 PM

Sprint

AP F19-5 (October), AP F19-6 (November) [ 957, 958 ]

AP F19-5 (October), AP F19-6 (November), AP S20-1 (December) [ 957, 958, 981 ]

Ian Sullivan made changes - 27/Dec/19 9:30 PM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Hide

Permalink

Ian Sullivan added a comment - 27/Dec/19 9:34 PM

This is a simple fix for some rare warnings we've seen when running image differencing using DCR templates. On further investigation, it looks like the problem is that a failing patch only had a thin strip of pixels with data, which was smaller than the buffer we clip from the edges. The fix is to skip the patch and print a log message.

Show

Ian Sullivan added a comment - 27/Dec/19 9:34 PM This is a simple fix for some rare warnings we've seen when running image differencing using DCR templates. On further investigation, it looks like the problem is that a failing patch only had a thin strip of pixels with data, which was smaller than the buffer we clip from the edges. The fix is to skip the patch and print a log message.

Ian Sullivan made changes - 27/Dec/19 9:34 PM

Reviewers		Gabor Kovacs [ gkovacs ]
Status	In Progress [ 3 ]	In Review [ 10004 ]

Hide

Permalink

Gabor Kovacs added a comment - 01/Jan/20 1:11 AM

Looks all right.

Show

Gabor Kovacs added a comment - 01/Jan/20 1:11 AM Looks all right.

Gabor Kovacs made changes - 01/Jan/20 1:11 AM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Ian Sullivan made changes - 03/Jan/20 10:32 PM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

People

Assignee:: Ian Sullivan

Reporter:: Ian Sullivan

Reviewers:: Gabor Kovacs

Watchers:: Gabor Kovacs, Ian Sullivan

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 10/Sep/19 5:59 PM

Updated:: 03/Jan/20 10:32 PM

Resolved:: 03/Jan/20 10:32 PM

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates