[DM-13462] Investigate sporadic FITS compression test failure - Jira

Details

Type: Story
Status: To Do
Resolution: Unresolved
Fix Version/s: None
Component/s: afw
Labels:
None

Team:
External

Description

Every now and then (rare, but apparently real), afw fails Jenkins with an error in the FITS compression quantisation test, e.g., here. This needs to be investigated.

Error

AssertionError: Images differ: maxDiff=0.942383 at position (0, 2); value=-0.464844 vs. 0.477539

Stacktrace

self = <test_fitsCompression.ImageCompressionTestCase testMethod=testQuantization>

    def testQuantization(self):

        """Test that our quantization produces the same values as cfitsio

            Our quantization is more configurable (e.g., choice of scaling algorithm,

            specifying mask planes) and extensible (logarithmic, asinh scalings)

            than cfitsio's. However, cfitsio uses its own fuzz ("subtractive dithering")

            when reading the data, so if we don't want to add random values twice,

            we need to be sure that we're using the same random values. To check that,

            we write one image with our scaling+compression, and one with cfitsio's

            compression using exactly the BSCALE and dither seed we used for our own.

            That way, the two codes will quantize independently, and we can compare

            the results.

"""

        bscaleSet = 1.0

        bzeroSet = self.background - 10*self.noise

        algorithm = ImageCompressionOptions.GZIP

        classList = (lsst.afw.image.ImageF, lsst.afw.image.ImageD)

        tilesList = ((4, 5), (0, 0), (0, 5), (4, 0), (0, 1))

        for cls, tiles in itertools.product(classList, tilesList):

            tiles = np.array(tiles, dtype=np.int64)

            compression = ImageCompressionOptions(algorithm, tiles, -bscaleSet)

            original = self.makeImage(cls)

            with lsst.utils.tests.getTempFilePath(self.extension) as filename:

                with lsst.afw.fits.Fits(filename, "w") as fits:

                    options = lsst.afw.fits.ImageWriteOptions(compression)

                    original.writeFits(fits, options)

                cfitsio = cls(filename)

                header = lsst.afw.image.readMetadata(filename, 1)

                seed = header.get("ZDITHER0")

                self.assertEqual(header.get("BSCALE"), bscaleSet)

            compression = ImageCompressionOptions(algorithm, tiles, 0.0)

            scaling = ImageScalingOptions(ImageScalingOptions.MANUAL, 32, [u"BAD"], bscale=bscaleSet,

                                          bzero=bzeroSet, fuzz=True, seed=seed)

            unpersisted = self.checkCompressedImage(cls, original, compression, scaling, atol=bscaleSet)

            oursDiff = unpersisted.getArray() - original.getArray()

            cfitsioDiff = cfitsio.getArray() - original.getArray()

>           self.assertImagesAlmostEqual(oursDiff, cfitsioDiff, atol=0.0)

tests/test_fitsCompression.py:720:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

python/lsst/afw/image/testUtils.py:118: in assertImagesAlmostEqual

    testCase.fail("%s: %s" % (msg, errStr))

E   AssertionError: Images differ: maxDiff=0.942383 at position (0, 2); value=-0.464844 vs. 0.477539

Standard Output

<class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5

Astropy currently doesn't read our compressed images perfectly.

<class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5

Astropy currently doesn't read our compressed images perfectly.

<class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5

Astropy currently doesn't read our compressed images perfectly.

<class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5

Astropy currently doesn't read our compressed images perfectly.

<class 'lsst.afw.image.image.image.ImageF'> CompressionAlgorithm.GZIP 8640 5760.0 1.5

Astropy currently doesn't read our compressed images perfectly.

Attachments

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Kian-Tat Lim added a comment - 02/Sep/20 4:36 AM

This just happened again on CentOS 7. https://ci.lsst.codes/blue/organizations/jenkins/scipipe%2Flsst_distrib/detail/lsst_distrib/1112/tests

Show

Kian-Tat Lim added a comment - 02/Sep/20 4:36 AM This just happened again on CentOS 7. https://ci.lsst.codes/blue/organizations/jenkins/scipipe%2Flsst_distrib/detail/lsst_distrib/1112/tests

Hide

Permalink

Paul Price added a comment - 02/Sep/20 3:47 PM

I only have w.2020.12 available right now, but I ran that test under valgrind, and the result makes me suspect that the problem is in cfitsio:

==179377== Conditional jump or move depends on uninitialised value(s)

==179377==    at 0x2B6B10A1: fits_read_compressed_img (imcompress.c:4554)

==179377==    by 0x2B6458F1: ffgsve (getcole.c:257)

==179377==    by 0x2B63BCEE: ffgsv (getcol.c:521)

==179377==    by 0x29366CC6: void lsst::afw::fits::Fits::readImageImpl<float>(int, float*, long*, long*, long*) (fits.cc:1400)

==179377==    by 0x2955360F: readImage<float, 2> (fits.h:547)

==179377==    by 0x2955360F: ndarray::Array<float, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray<float>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageBaseFitsReader.cc:198)

==179377==    by 0x295552FA: lsst::afw::image::Image<float> lsst::afw::image::ImageFitsReader::read<float>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageFitsReader.cc:28)

==179377==    by 0x295440DE: lsst::afw::image::Image<float>::Image(std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (Image.cc:374)

==179377==    by 0x46C15C2C: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool>::execute<pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool)#1}, void, pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >&&, void (*)(pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) (init.h:61)

==179377==    by 0x46BE5D43: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (pybind11.h:656)

==179377==    by 0x243D26: _PyMethodDef_RawFastCallDict (call.c:515)

==179377==    by 0x243E70: _PyCFunction_FastCallDict (call.c:582)

==179377==    by 0x235472: _PyObject_Call_Prepend (call.c:904)

==179377== Conditional jump or move depends on uninitialised value(s)

==179377==    at 0x2B6B10A1: fits_read_compressed_img (imcompress.c:4554)

==179377==    by 0x2B642620: ffgsvd (getcold.c:257)

==179377==    by 0x2B63BD52: ffgsv (getcol.c:530)

==179377==    by 0x2936711C: void lsst::afw::fits::Fits::readImageImpl<double>(int, double*, long*, long*, long*) (fits.cc:1400)

==179377==    by 0x29553FDF: readImage<double, 2> (fits.h:547)

==179377==    by 0x29553FDF: ndarray::Array<double, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray<double>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageBaseFitsReader.cc:198)

==179377==    by 0x295553EA: lsst::afw::image::Image<double> lsst::afw::image::ImageFitsReader::read<double>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageFitsReader.cc:28)

==179377==    by 0x29545F6E: lsst::afw::image::Image<double>::Image(std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (Image.cc:374)

==179377==    by 0x46C161AC: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool>::execute<pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool)#1}, void, pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >&&, void (*)(pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) (init.h:61)

==179377==    by 0x46BE5D43: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (pybind11.h:656)

==179377==    by 0x243D26: _PyMethodDef_RawFastCallDict (call.c:515)

==179377==    by 0x243E70: _PyCFunction_FastCallDict (call.c:582)

==179377==    by 0x235472: _PyObject_Call_Prepend (call.c:904)

Show

Paul Price added a comment - 02/Sep/20 3:47 PM I only have w.2020.12 available right now, but I ran that test under valgrind, and the result makes me suspect that the problem is in cfitsio: ==179377== Conditional jump or move depends on uninitialised value(s) ==179377== at 0x2B6B10A1: fits_read_compressed_img (imcompress.c:4554) ==179377== by 0x2B6458F1: ffgsve (getcole.c:257) ==179377== by 0x2B63BCEE: ffgsv (getcol.c:521) ==179377== by 0x29366CC6: void lsst::afw::fits::Fits::readImageImpl<float>(int, float*, long*, long*, long*) (fits.cc:1400) ==179377== by 0x2955360F: readImage<float, 2> (fits.h:547) ==179377== by 0x2955360F: ndarray::Array<float, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray<float>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageBaseFitsReader.cc:198) ==179377== by 0x295552FA: lsst::afw::image::Image<float> lsst::afw::image::ImageFitsReader::read<float>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageFitsReader.cc:28) ==179377== by 0x295440DE: lsst::afw::image::Image<float>::Image(std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (Image.cc:374) ==179377== by 0x46C15C2C: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool>::execute<pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool)#1}, void, pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >&&, void (*)(pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) (init.h:61) ==179377== by 0x46BE5D43: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (pybind11.h:656) ==179377== by 0x243D26: _PyMethodDef_RawFastCallDict (call.c:515) ==179377== by 0x243E70: _PyCFunction_FastCallDict (call.c:582) ==179377== by 0x235472: _PyObject_Call_Prepend (call.c:904) ==179377== Conditional jump or move depends on uninitialised value(s) ==179377== at 0x2B6B10A1: fits_read_compressed_img (imcompress.c:4554) ==179377== by 0x2B642620: ffgsvd (getcold.c:257) ==179377== by 0x2B63BD52: ffgsv (getcol.c:530) ==179377== by 0x2936711C: void lsst::afw::fits::Fits::readImageImpl<double>(int, double*, long*, long*, long*) (fits.cc:1400) ==179377== by 0x29553FDF: readImage<double, 2> (fits.h:547) ==179377== by 0x29553FDF: ndarray::Array<double, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray<double>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageBaseFitsReader.cc:198) ==179377== by 0x295553EA: lsst::afw::image::Image<double> lsst::afw::image::ImageFitsReader::read<double>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageFitsReader.cc:28) ==179377== by 0x29545F6E: lsst::afw::image::Image<double>::Image(std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (Image.cc:374) ==179377== by 0x46C161AC: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool>::execute<pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool)#1}, void, pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >&&, void (*)(pybind11::detail::value_and_holder&, std::string const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) (init.h:61) ==179377== by 0x46BE5D43: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (pybind11.h:656) ==179377== by 0x243D26: _PyMethodDef_RawFastCallDict (call.c:515) ==179377== by 0x243E70: _PyCFunction_FastCallDict (call.c:582) ==179377== by 0x235472: _PyObject_Call_Prepend (call.c:904)

Hide

Permalink

Paul Price added a comment - 19/Oct/21 2:25 AM

I built afw with the newly-released cfitsio 4.0.0, ran tests_fitsCompression.py under valgrind, and found zero errors:

(lsst-scipipe) pprice@tiger2-sumire:~/Software/cfitsio-4.0.0 $ ./configure && make

(lsst-scipipe) pprice@tiger2-sumire:~/LSST/afw[tickets/DM-13462] $ LSST_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LSST_LIBRARY_PATH" LD_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LD_LIBRARY_PATH" scons CXXFLAGS="-I$HOME/Software/cfitsio-4.0.0/ $CXXFLAGS" LINKFLAGS="-L$HOME/Software/cfitsio-4.0.0/ $LINKFLAGS -lz -lcurl" LD_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LD_LIBRARY_PATH" LSST_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LSST_LIBRARY_PATH" --setenv lib python

(lsst-scipipe) pprice@tiger2-sumire:~/LSST/afw[tickets/DM-13462] $ PYTHONMALLOC=malloc LD_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LD_LIBRARY_PATH" valgrind --suppressions=$HOME/Software/valgrind-python.supp -s python tests/test_fitsCompression.py

==248133== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

When I repeated this technique with cfitsio 3.410, I got the modern equivalent of the above two errors:

==258524== 96 errors in context 1 of 2:

==258524== Conditional jump or move depends on uninitialised value(s)

==258524==    at 0x1A664904: fits_read_compressed_img.part.0 (in /home/pprice/LSST/afw/lib/libafw.so)

==258524==    by 0x1A5E9EAC: ffgsvd (in /home/pprice/LSST/afw/lib/libafw.so)

==258524==    by 0x1A5E30A6: ffgsv (in /home/pprice/LSST/afw/lib/libafw.so)

==258524==    by 0x19FA807C: void lsst::afw::fits::Fits::readImageImpl<double>(int, double*, long*, long*, long*) (fits.cc:1417)

==258524==    by 0x1A20C637: readImage<double, 2> (fits.h:547)

==258524==    by 0x1A20C637: ndarray::Array<double, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray<double>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageBaseFitsReader.cc:198)

==258524==    by 0x1A20DA90: lsst::afw::image::Image<double> lsst::afw::image::ImageFitsReader::read<double>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageFitsReader.cc:28)

==258524==    by 0x1A20755B: lsst::afw::image::Image<double>::Image(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (Image.cc:361)

==258524==    by 0x1DD3F0BF: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool>::execute<pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool)#1}, void, pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >&&, void (*)(pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) (init.h:61)

==258524==    by 0x1DCC992E: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (pybind11.h:821)

==258524==    by 0x233DF3: UnknownInlinedFun (call.c:743)

==258524==    by 0x233DF3: PyCFunction_Call (call.c:773)

==258524==    by 0x242F2D: _PyObject_MakeTpCall (call.c:159)

==258524==    by 0x2BA6BD: UnknownInlinedFun (abstract.h:125)

==258524==    by 0x2BA6BD: UnknownInlinedFun (abstract.h:115)

==258524==    by 0x2BA6BD: method_vectorcall (classobject.c:89)

==258524==

==258524==

==258524== 96 errors in context 2 of 2:

==258524== Conditional jump or move depends on uninitialised value(s)

==258524==    at 0x1A664904: fits_read_compressed_img.part.0 (in /home/pprice/LSST/afw/lib/libafw.so)

==258524==    by 0x1A5ED93C: ffgsve (in /home/pprice/LSST/afw/lib/libafw.so)

==258524==    by 0x1A5E305D: ffgsv (in /home/pprice/LSST/afw/lib/libafw.so)

==258524==    by 0x19FA7C38: void lsst::afw::fits::Fits::readImageImpl<float>(int, float*, long*, long*, long*) (fits.cc:1417)

==258524==    by 0x1A20BCE7: readImage<float, 2> (fits.h:547)

==258524==    by 0x1A20BCE7: ndarray::Array<float, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray<float>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageBaseFitsReader.cc:198)

==258524==    by 0x1A20D960: lsst::afw::image::Image<float> lsst::afw::image::ImageFitsReader::read<float>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageFitsReader.cc:28)

==258524==    by 0x1A20593B: lsst::afw::image::Image<float>::Image(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (Image.cc:361)

==258524==    by 0x1DD3ED5F: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool>::execute<pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool)#1}, void, pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >&&, void (*)(pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) (init.h:61)

==258524==    by 0x1DCC992E: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (pybind11.h:821)

==258524==    by 0x233DF3: UnknownInlinedFun (call.c:743)

==258524==    by 0x233DF3: PyCFunction_Call (call.c:773)

==258524==    by 0x242F2D: _PyObject_MakeTpCall (call.c:159)

==258524==    by 0x2BA6BD: UnknownInlinedFun (abstract.h:125)

==258524==    by 0x2BA6BD: UnknownInlinedFun (abstract.h:115)

==258524==    by 0x2BA6BD: method_vectorcall (classobject.c:89)

==258524==

==258524== ERROR SUMMARY: 192 errors from 2 contexts (suppressed: 0 from 0)

I conclude that upgrading to cfitsio 4.0.0 will solve this bug. However, cfitsio 4.0.0 only just today found its way into conda-forge, and it seems that we have multiple dependencies that pin an earlier version (including wcslib and healpy).

Show

Paul Price added a comment - 19/Oct/21 2:25 AM I built afw with the newly-released cfitsio 4.0.0, ran tests_fitsCompression.py under valgrind, and found zero errors: (lsst-scipipe) pprice@tiger2-sumire:~/Software/cfitsio-4.0.0 $ ./configure && make (lsst-scipipe) pprice@tiger2-sumire:~/LSST/afw[tickets/DM-13462] $ LSST_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LSST_LIBRARY_PATH" LD_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LD_LIBRARY_PATH" scons CXXFLAGS="-I$HOME/Software/cfitsio-4.0.0/ $CXXFLAGS" LINKFLAGS="-L$HOME/Software/cfitsio-4.0.0/ $LINKFLAGS -lz -lcurl" LD_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LD_LIBRARY_PATH" LSST_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LSST_LIBRARY_PATH" --setenv lib python (lsst-scipipe) pprice@tiger2-sumire:~/LSST/afw[tickets/DM-13462] $ PYTHONMALLOC=malloc LD_LIBRARY_PATH="$HOME/Software/cfitsio-4.0.0:$LD_LIBRARY_PATH" valgrind --suppressions=$HOME/Software/valgrind-python.supp -s python tests/test_fitsCompression.py ==248133== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) When I repeated this technique with cfitsio 3.410, I got the modern equivalent of the above two errors: ==258524== 96 errors in context 1 of 2: ==258524== Conditional jump or move depends on uninitialised value(s) ==258524== at 0x1A664904: fits_read_compressed_img.part.0 (in /home/pprice/LSST/afw/lib/libafw.so) ==258524== by 0x1A5E9EAC: ffgsvd (in /home/pprice/LSST/afw/lib/libafw.so) ==258524== by 0x1A5E30A6: ffgsv (in /home/pprice/LSST/afw/lib/libafw.so) ==258524== by 0x19FA807C: void lsst::afw::fits::Fits::readImageImpl<double>(int, double*, long*, long*, long*) (fits.cc:1417) ==258524== by 0x1A20C637: readImage<double, 2> (fits.h:547) ==258524== by 0x1A20C637: ndarray::Array<double, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray<double>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageBaseFitsReader.cc:198) ==258524== by 0x1A20DA90: lsst::afw::image::Image<double> lsst::afw::image::ImageFitsReader::read<double>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageFitsReader.cc:28) ==258524== by 0x1A20755B: lsst::afw::image::Image<double>::Image(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (Image.cc:361) ==258524== by 0x1DD3F0BF: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool>::execute<pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool)#1}, void, pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<lsst::afw::image::Image<double>, std::shared_ptr<lsst::afw::image::Image<double> >, lsst::afw::image::ImageBase<double> >&&, void (*)(pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) (init.h:61) ==258524== by 0x1DCC992E: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (pybind11.h:821) ==258524== by 0x233DF3: UnknownInlinedFun (call.c:743) ==258524== by 0x233DF3: PyCFunction_Call (call.c:773) ==258524== by 0x242F2D: _PyObject_MakeTpCall (call.c:159) ==258524== by 0x2BA6BD: UnknownInlinedFun (abstract.h:125) ==258524== by 0x2BA6BD: UnknownInlinedFun (abstract.h:115) ==258524== by 0x2BA6BD: method_vectorcall (classobject.c:89) ==258524== ==258524== ==258524== 96 errors in context 2 of 2: ==258524== Conditional jump or move depends on uninitialised value(s) ==258524== at 0x1A664904: fits_read_compressed_img.part.0 (in /home/pprice/LSST/afw/lib/libafw.so) ==258524== by 0x1A5ED93C: ffgsve (in /home/pprice/LSST/afw/lib/libafw.so) ==258524== by 0x1A5E305D: ffgsv (in /home/pprice/LSST/afw/lib/libafw.so) ==258524== by 0x19FA7C38: void lsst::afw::fits::Fits::readImageImpl<float>(int, float*, long*, long*, long*) (fits.cc:1417) ==258524== by 0x1A20BCE7: readImage<float, 2> (fits.h:547) ==258524== by 0x1A20BCE7: ndarray::Array<float, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray<float>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageBaseFitsReader.cc:198) ==258524== by 0x1A20D960: lsst::afw::image::Image<float> lsst::afw::image::ImageFitsReader::read<float>(lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (ImageFitsReader.cc:28) ==258524== by 0x1A20593B: lsst::afw::image::Image<float>::Image(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool) (Image.cc:361) ==258524== by 0x1DD3ED5F: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool>::execute<pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool)#1}, void, pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<lsst::afw::image::Image<float>, std::shared_ptr<lsst::afw::image::Image<float> >, lsst::afw::image::ImageBase<float> >&&, void (*)(pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<lsst::daf::base::PropertySet>, lsst::geom::Box2I const&, lsst::afw::image::ImageOrigin, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) (init.h:61) ==258524== by 0x1DCC992E: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (pybind11.h:821) ==258524== by 0x233DF3: UnknownInlinedFun (call.c:743) ==258524== by 0x233DF3: PyCFunction_Call (call.c:773) ==258524== by 0x242F2D: _PyObject_MakeTpCall (call.c:159) ==258524== by 0x2BA6BD: UnknownInlinedFun (abstract.h:125) ==258524== by 0x2BA6BD: UnknownInlinedFun (abstract.h:115) ==258524== by 0x2BA6BD: method_vectorcall (classobject.c:89) ==258524== ==258524== ERROR SUMMARY: 192 errors from 2 contexts (suppressed: 0 from 0) I conclude that upgrading to cfitsio 4.0.0 will solve this bug. However, cfitsio 4.0.0 only just today found its way into conda-forge, and it seems that we have multiple dependencies that pin an earlier version (including wcslib and healpy).

Hide

Permalink

Paul Price added a comment - 19/Oct/21 2:51 AM

Eli Rykoff asked me to check cfitsio 3.49. It's clean. cfitsio 3.49 might be a suitable replacement for cfitsio 4.0.0 if the latter proves troublesome to integrate.

Show

Paul Price added a comment - 19/Oct/21 2:51 AM Eli Rykoff asked me to check cfitsio 3.49. It's clean. cfitsio 3.49 might be a suitable replacement for cfitsio 4.0.0 if the latter proves troublesome to integrate.

Data Management

Investigate sporadic FITS compression test failure

Details

Description

Attachments

Activity

People

Dates

Jenkins