Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13462

Investigate sporadic FITS compression test failure

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: afw
    • Labels:
      None
    • Team:
      External

      Description

      Every now and then (rare, but apparently real), afw fails Jenkins with an error in the FITS compression quantisation test, e.g., here. This needs to be investigated.

      Error
      AssertionError: Images differ: maxDiff=0.942383 at position (0, 2); value=-0.464844 vs. 0.477539
      Stacktrace
      self = <test_fitsCompression.ImageCompressionTestCase testMethod=testQuantization>
          def testQuantization(self):
              """Test that our quantization produces the same values as cfitsio
          
                  Our quantization is more configurable (e.g., choice of scaling algorithm,
                  specifying mask planes) and extensible (logarithmic, asinh scalings)
                  than cfitsio's. However, cfitsio uses its own fuzz ("subtractive dithering")
                  when reading the data, so if we don't want to add random values twice,
                  we need to be sure that we're using the same random values. To check that,
                  we write one image with our scaling+compression, and one with cfitsio's
                  compression using exactly the BSCALE and dither seed we used for our own.
                  That way, the two codes will quantize independently, and we can compare
                  the results.
                  """
              bscaleSet = 1.0
              bzeroSet = self.background - 10*self.noise
              algorithm = ImageCompressionOptions.GZIP
              classList = (lsst.afw.image.ImageF, lsst.afw.image.ImageD)
              tilesList = ((4, 5), (0, 0), (0, 5), (4, 0), (0, 1))
              for cls, tiles in itertools.product(classList, tilesList):
                  tiles = np.array(tiles, dtype=np.int64)
                  compression = ImageCompressionOptions(algorithm, tiles, -bscaleSet)
                  original = self.makeImage(cls)
                  with lsst.utils.tests.getTempFilePath(self.extension) as filename:
                      with lsst.afw.fits.Fits(filename, "w") as fits:
                          options = lsst.afw.fits.ImageWriteOptions(compression)
                          original.writeFits(fits, options)
                      cfitsio = cls(filename)
                      header = lsst.afw.image.readMetadata(filename, 1)
                      seed = header.get("ZDITHER0")
                      self.assertEqual(header.get("BSCALE"), bscaleSet)
          
                  compression = ImageCompressionOptions(algorithm, tiles, 0.0)
                  scaling = ImageScalingOptions(ImageScalingOptions.MANUAL, 32, [u"BAD"], bscale=bscaleSet,
                                                bzero=bzeroSet, fuzz=True, seed=seed)
                  unpersisted = self.checkCompressedImage(cls, original, compression, scaling, atol=bscaleSet)
                  oursDiff = unpersisted.getArray() - original.getArray()
                  cfitsioDiff = cfitsio.getArray() - original.getArray()
      >           self.assertImagesAlmostEqual(oursDiff, cfitsioDiff, atol=0.0)
      tests/test_fitsCompression.py:720: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      python/lsst/afw/image/testUtils.py:118: in assertImagesAlmostEqual
          testCase.fail("%s: %s" % (msg, errStr))
      E   AssertionError: Images differ: maxDiff=0.942383 at position (0, 2); value=-0.464844 vs. 0.477539
      Standard Output
      <class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5
      Astropy currently doesn't read our compressed images perfectly.
      <class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5
      Astropy currently doesn't read our compressed images perfectly.
      <class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5
      Astropy currently doesn't read our compressed images perfectly.
      <class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5
      Astropy currently doesn't read our compressed images perfectly.
      <class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5
      Astropy currently doesn't read our compressed images perfectly.
      

        Attachments

          Activity

          price Paul Price created issue -

            People

            Assignee:
            price Paul Price
            Reporter:
            price Paul Price
            Watchers:
            John Parejko, Kian-Tat Lim, Paul Price
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:

                Jenkins

                No builds found.