Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7976

coadd cannot be loaded directly as afw.image.ExposureF

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw

      Description

      Loading a deepCoadd as an afw Exposure gives the following error:

      >>> import lsst.afw.image as afwImage
      >>> coadd = afwImage.ExposureF("validation_data_hsc/DATA/rerun/20160805/deepCoadd/HSC-Y/0/7,7.fits")
      31424 [0x7fff74d45000] DEBUG afw.image.Mask null - Number of mask planes: 16
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/opt/sw/lsstsw/stack/DarwinX86/afw/12.1-4-gaba3f16/python/lsst/afw/image/imageLib.py", line 11623, in __init__
          this = _imageLib.new_ExposureF(*args)
      lsst.pex.exceptions.wrappers.LogicError: 
        File "include/lsst/afw/table/BaseRecord.h", line 95, in const typename Field<T>::Element *lsst::afw::table::BaseRecord::getElement(const Key<T> &) const [T = int]
          Key is not valid (if this is a SourceRecord, make sure slot aliases have been setup). {0}
        File "src/table/io/InputArchive.cc", line 105, in std::shared_ptr<Persistable> lsst::afw::table::io::InputArchive::Impl::get(int, const lsst::afw::table::io::InputArchive &)
          loading object with id=132, name='CoaddPsf' {1}
      lsst::pex::exceptions::LogicError: 'Key is not valid (if this is a SourceRecord, make sure slot aliases have been setup). {0}; loading object with id=132, name='CoaddPsf' {1}'
      

      This used to work, or at least it worked with a stack from Sep 8.

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The problem is seen when loading old coadds. New coadds, obtained by running ci_hsc with the current code, can be loaded without problems.

            The coadd from validation_data_hsc rerun/20160805 as in the description is copied to lsst-dev:/lsst8/hchiang2/coadd/7,7.fits

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The problem is seen when loading old coadds. New coadds, obtained by running ci_hsc with the current code, can be loaded without problems. The coadd from validation_data_hsc rerun/20160805 as in the description is copied to lsst-dev:/lsst8/hchiang2/coadd/7,7.fits
            Hide
            price Paul Price added a comment -

            Backtrace:

            #0  lsst::afw::table::BaseRecord::getElement<int> (this=0x1ba52a0, key=...)
                at include/lsst/afw/table/BaseRecord.h:92
            #1  0x00007fff9559c46e in get<int> (this=0x7fffffff2db0, input=..., output=
                ..., mapper=Unhandled dwarf expression opcode 0xf3
            ) at include/lsst/afw/table/BaseRecord.h:133
            #2  lsst::afw::table::(anonymous namespace)::PersistenceHelper::readRecord (
                this=0x7fffffff2db0, input=..., output=..., mapper=Unhandled dwarf expression opcode 0xf3
            )
                at src/table/Exposure.cc:132
            #3  0x00007fff955a1d3e in lsst::afw::table::ExposureCatalogT<lsst::afw::table::ExposureRecord>::readFromArchive (archive=..., catalog=...)
                at src/table/Exposure.cc:446
            #4  0x00007fffd752900e in lsst::meas::algorithms::CoaddPsf::Factory::read (
                this=Unhandled dwarf expression opcode 0xf3
            ) at src/CoaddPsf.cc:369
            #5  0x00007fff9566a284 in lsst::afw::table::io::InputArchive::Impl::get (this=
                0x1c82860, id=132, self=...) at src/table/io/InputArchive.cc:101
            #6  0x00007fff95667275 in lsst::afw::table::io::InputArchive::get (this=Unhandled dwarf expression opcode 0xf3
            )
                at src/table/io/InputArchive.cc:184
            #7  0x00007fff9577f65e in get<lsst::afw::detection::Psf> (this=0xb09260, 
                fitsfile=Unhandled dwarf expression opcode 0xf3
            ) at include/lsst/afw/table/io/InputArchive.h:57
            #8  lsst::afw::image::ExposureInfo::_readFits (this=0xb09260, fitsfile=Unhandled dwarf expression opcode 0xf3
            )
                at src/image/ExposureInfo.cc:275
            #9  0x00007fff956e8504 in lsst::afw::image::Exposure<float, unsigned short, float>::_readFits (this=0x1ac6b30, fitsfile=..., bbox=Unhandled dwarf expression opcode 0xf3
            )
                at src/image/Exposure.cc:219
            

            The problem appears to be that the introduction of VisitInfo didn't preserve backwards compatibility (I'm not making any statement about the desirability of such backwards compatibility).

            Show
            price Paul Price added a comment - Backtrace: #0 lsst::afw::table::BaseRecord::getElement<int> (this=0x1ba52a0, key=...) at include/lsst/afw/table/BaseRecord.h:92 #1 0x00007fff9559c46e in get<int> (this=0x7fffffff2db0, input=..., output= ..., mapper=Unhandled dwarf expression opcode 0xf3 ) at include/lsst/afw/table/BaseRecord.h:133 #2 lsst::afw::table::(anonymous namespace)::PersistenceHelper::readRecord ( this=0x7fffffff2db0, input=..., output=..., mapper=Unhandled dwarf expression opcode 0xf3 ) at src/table/Exposure.cc:132 #3 0x00007fff955a1d3e in lsst::afw::table::ExposureCatalogT<lsst::afw::table::ExposureRecord>::readFromArchive (archive=..., catalog=...) at src/table/Exposure.cc:446 #4 0x00007fffd752900e in lsst::meas::algorithms::CoaddPsf::Factory::read ( this=Unhandled dwarf expression opcode 0xf3 ) at src/CoaddPsf.cc:369 #5 0x00007fff9566a284 in lsst::afw::table::io::InputArchive::Impl::get (this= 0x1c82860, id=132, self=...) at src/table/io/InputArchive.cc:101 #6 0x00007fff95667275 in lsst::afw::table::io::InputArchive::get (this=Unhandled dwarf expression opcode 0xf3 ) at src/table/io/InputArchive.cc:184 #7 0x00007fff9577f65e in get<lsst::afw::detection::Psf> (this=0xb09260, fitsfile=Unhandled dwarf expression opcode 0xf3 ) at include/lsst/afw/table/io/InputArchive.h:57 #8 lsst::afw::image::ExposureInfo::_readFits (this=0xb09260, fitsfile=Unhandled dwarf expression opcode 0xf3 ) at src/image/ExposureInfo.cc:275 #9 0x00007fff956e8504 in lsst::afw::image::Exposure<float, unsigned short, float>::_readFits (this=0x1ac6b30, fitsfile=..., bbox=Unhandled dwarf expression opcode 0xf3 ) at src/image/Exposure.cc:219 The problem appears to be that the introduction of VisitInfo didn't preserve backwards compatibility (I'm not making any statement about the desirability of such backwards compatibility).
            Hide
            rowen Russell Owen added a comment - - edited

            There is code and a test for backwards compatibility with old versions of ExposureTable, but it relies on reading a persisted table, not a coadd. Clearly there are differences.

            I implemented a fix but it may need more work. I added simple coadds to tests/data, one with a v1 coadd inputs table and the other with v2 and updated testExposureRecord.py to read and check both of them. The new afw code can read old and new coadds, but there is a remaining issue: each visit in coadd.getInfo().getCoaddInputs().visits returns None for getCalib(), getApCorrMap() and getVisitInfo(). This is only expected for getVisitInfo() when reading a v1 coadd.

            I also see this problem for Calib and ValidPolygon when reading Hsin-Fang Chiang's coadd /lsst8/hchiang2/coadd/7,7.fits using older afw code, and any coaddTempExp ought to at least have a Calib. So this appears to be an existing bug, though well worth fixing on this ticket, if practical.

            Show
            rowen Russell Owen added a comment - - edited There is code and a test for backwards compatibility with old versions of ExposureTable , but it relies on reading a persisted table, not a coadd. Clearly there are differences. I implemented a fix but it may need more work. I added simple coadds to tests/data, one with a v1 coadd inputs table and the other with v2 and updated testExposureRecord.py to read and check both of them. The new afw code can read old and new coadds, but there is a remaining issue: each visit in coadd.getInfo().getCoaddInputs().visits returns None for getCalib() , getApCorrMap() and getVisitInfo() . This is only expected for getVisitInfo() when reading a v1 coadd. I also see this problem for Calib and ValidPolygon when reading Hsin-Fang Chiang 's coadd /lsst8/hchiang2/coadd/7,7.fits using older afw code, and any coaddTempExp ought to at least have a Calib . So this appears to be an existing bug, though well worth fixing on this ticket, if practical.
            Hide
            rowen Russell Owen added a comment -

            I found and fixed two problems:

            • The problem stated in the original description was fixed in afw with a minor update to table/Exposure.cc
            • Many fields of ExposureInfo were not being saved to CoaddInputs due to a bug in CoaddInputRecorderTask in pipe_tasks.

            It is difficult to test the first problem in afw, because generating coadds uses CoaddInputRecorderTask, so I put a new test tests/testCoaddInputs.py in pipe_tasks to test both problems. This test generates a simple coadd, writes it to a FITS file and reads it back in, checking that the information in CoaddInputs is preserved. That is not possible for coadds with version 1 CoaddInputs data, so I used the new test with an older version of the stack to generate a version 1 coadd and saved that. Instructions are in the new test.

            Show
            rowen Russell Owen added a comment - I found and fixed two problems: The problem stated in the original description was fixed in afw with a minor update to table/Exposure.cc Many fields of ExposureInfo were not being saved to CoaddInputs due to a bug in CoaddInputRecorderTask in pipe_tasks . It is difficult to test the first problem in afw , because generating coadds uses CoaddInputRecorderTask , so I put a new test tests/testCoaddInputs.py in pipe_tasks to test both problems. This test generates a simple coadd, writes it to a FITS file and reads it back in, checking that the information in CoaddInputs is preserved. That is not possible for coadds with version 1 CoaddInputs data, so I used the new test with an older version of the stack to generate a version 1 coadd and saved that. Instructions are in the new test.
            Hide
            rowen Russell Owen added a comment - - edited

            I described my fixes in my previous comment on this ticket.

            Two caveats:

            • The new test does not test CoaddInputs.ccds. I felt it was out of scope for this ticket, but will file a new one if desired.
            • The new test is not intended as a full test of CoaddInputRecorderTask, and such a test would be desirable but I'm not even sure a ticket is in order as so many of our tasks lack tests and this is a known issue.
            Show
            rowen Russell Owen added a comment - - edited I described my fixes in my previous comment on this ticket. Two caveats: The new test does not test CoaddInputs.ccds . I felt it was out of scope for this ticket, but will file a new one if desired. The new test is not intended as a full test of CoaddInputRecorderTask , and such a test would be desirable but I'm not even sure a ticket is in order as so many of our tasks lack tests and this is a known issue.
            Hide
            jbosch Jim Bosch added a comment -

            Comments on the GitHub PRs.

            Show
            jbosch Jim Bosch added a comment - Comments on the GitHub PRs.
            Hide
            rowen Russell Owen added a comment -

            Thank you for the quick and helpful review. I have implemented all your suggested changes and will merge after a Jenkins build passes (preferably both py2 and py3, but last time I started the py3 build lsst_py3 was still having problems with the obs_base rename, so it wasn't usable to test afw).

            Show
            rowen Russell Owen added a comment - Thank you for the quick and helpful review. I have implemented all your suggested changes and will merge after a Jenkins build passes (preferably both py2 and py3, but last time I started the py3 build lsst_py3 was still having problems with the obs_base rename, so it wasn't usable to test afw).

              People

              Assignee:
              rowen Russell Owen
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Jim Bosch
              Watchers:
              Hsin-Fang Chiang, Jim Bosch, John Swinbank, Meredith Rawls, Paul Price, Russell Owen, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.