Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18703

constructFlat.py --config isr.doCrosstalkBeforeAssemble=False raises a LengthError exception

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      DRP S19-5
    • Team:
      Data Release Production

      Description

      Here's a full command line that produces the error:

      constructFlat.py output --rerun jchiang/xtalk_after_config \
      --id raftName=R22 detectorName=S11 --config isr.doBias=False \
      isr.doDark=False isr.doCrosstalkBeforeAssemble=False --longlog \
      --batch-type=None

      and the last several lines of the traceback (full log attached):

        File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/ip_isr/17.0.1-7-g69836a1+1/python/lsst/ip/isr/isrTask.py", line 1195, in run
          self.crosstalk.run(ccdExposure, crosstalkSources=crosstalkSources)
        File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/ip_isr/17.0.1-7-g69836a1+1/python/lsst/ip/isr/crosstalk.py", line 90, in run
          crosstalkStr=self.config.crosstalkMaskPlane)
        File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/ip_isr/17.0.1-7-g69836a1+1/python/lsst/ip/isr/crosstalk.py", line 212, in subtractCrosstalk
          jImage = extractAmp(mi, jAmp, iAmp.getReadoutCorner())
        File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/ip_isr/17.0.1-7-g69836a1+1/python/lsst/ip/isr/crosstalk.py", line 130, in extractAmp
          output = image[amp.getBBox() if isTrimmed else amp.getRawDataBBox()]
        File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/afw/17.0.1-4-g41c8d5dc0+1/python/lsst/afw/image/slicing.py", line 262, in __getitem__
          return self.subset(box, origin=origin)
      lsst.pex.exceptions.wrappers.LengthError: 
        File "src/image/Image.cc", line 83, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_makeSubView(const Extent2I&, const Extent2I&, const _view_t&) [with PixelT = float; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<float, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>]
          Box2I(Point2I(3840,0),lsst::geom::Extent2I(509,2000)) doesn't fit in image 4072x4000 {0}
      lsst::pex::exceptions::LengthError: 'Box2I(Point2I(3840,0),lsst::geom::Extent2I(509,2000)) doesn't fit in image 4072x4000'

        Attachments

          Issue Links

            Activity

            Hide
            jchiang James Chiang added a comment -

            Data to reproduce this are in /scratch/jchiang/DC2/constructFlat_crosstalk_bug at lsst-dev.

            Show
            jchiang James Chiang added a comment - Data to reproduce this are in /scratch/jchiang/DC2/constructFlat_crosstalk_bug at lsst-dev.
            Hide
            czw Christopher Waters added a comment -

            I had thought that doCrosstalkBeforeAssemble=False was the default.  I don't remember if I had any logic for the choice, or simply set the value counter to how I was thinking.

            Regardless of that point, you shouldn't have gotten an exception, which seems to be caused by an `isTrimmed` parameter not being fully implemented in the crosstalk code.  I've resolved that on a ticket branch, and will see about getting it merged into master once I've convinced myself it hasn't introduced any issues for other cameras.

            Show
            czw Christopher Waters added a comment - I had thought that doCrosstalkBeforeAssemble=False was the default.  I don't remember if I had any logic for the choice, or simply set the value counter to how I was thinking. Regardless of that point, you shouldn't have gotten an exception, which seems to be caused by an `isTrimmed` parameter not being fully implemented in the crosstalk code.  I've resolved that on a ticket branch, and will see about getting it merged into master once I've convinced myself it hasn't introduced any issues for other cameras.
            Hide
            rhl Robert Lupton added a comment -

            As Eli Rykoff points out the problem is that a background level is removed from each amplifier image as the crosstalk is corrected.  It turns out that this was introduced when the code was ported and generalised from the HSC implementation which subtracted a background level from each CCD image; that's OK if there is no cross-CCD crosstalk.

            I wrote the original code that did this, and as far as I know this was done as a result to a natural repulsion to subtracting two large numbers;  but I don't think that this is in fact a problem.  So simply removing the call to calculateBackground in crosstalk.py should fix this problem.

            Show
            rhl Robert Lupton added a comment - As Eli Rykoff points out the problem is that a background level is removed from each amplifier image as the crosstalk is corrected.  It turns out that this was introduced when the code was ported and generalised from the HSC implementation which subtracted a background level from each CCD image ; that's OK if there is no cross-CCD crosstalk. I wrote the original code that did this, and as far as I know this was done as a result to a natural repulsion to subtracting two large numbers;  but I don't think that this is in fact a problem.  So simply removing the call to calculateBackground in crosstalk.py should fix this problem.
            Hide
            jchiang James Chiang added a comment -

            I confirm that if I use the code in the tickets/DM-18703 branch of ip_isr that constructFlat.py with isr.doCrosstalkBeforeAssemble=False runs to completion and that the resulting master flat still has the amp-level offsets.

             

            If I then remove the background subtraction from crosstalk.py and rerun, the resulting master flat looks correct.

            Show
            jchiang James Chiang added a comment - I confirm that if I use the code in the tickets/ DM-18703 branch of ip_isr that constructFlat.py with isr.doCrosstalkBeforeAssemble=False runs to completion and that the resulting master flat still has the amp-level offsets.   If I then remove the background subtraction from crosstalk.py and rerun, the resulting master flat looks correct.
            Hide
            rhl Robert Lupton added a comment -

            Eli points out that I was foolish and deluded to claim that it was ever correct to remove the sky;  in HSC it wasn't too bad as the sum of crosstalk coefficients is similar for all amplifiers.  Because we make the same mistake in the dome flats the error is cosmetically innocuous, but it causes incorrect photometry.

            Show
            rhl Robert Lupton added a comment - Eli points out that I was foolish and deluded to claim that it was ever correct to remove the sky;  in HSC it wasn't too bad as the sum of crosstalk coefficients is similar for all amplifiers.  Because we make the same mistake in the dome flats the error is cosmetically innocuous, but it causes incorrect photometry.

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              jchiang James Chiang
              Reviewers:
              Eli Rykoff
              Watchers:
              Christopher Waters, Eli Rykoff, James Chiang, Robert Lupton
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.