The problem is seen when loading old coadds. New coadds, obtained by running ci_hsc with the current code, can be loaded without problems.

The coadd from validation_data_hsc rerun/20160805 as in the description is copied to lsst-dev:/lsst8/hchiang2/coadd/7,7.fits

Backtrace:

#0  lsst::afw::table::BaseRecord::getElement<int> (this=0x1ba52a0, key=...)

    at include/lsst/afw/table/BaseRecord.h:92

#1  0x00007fff9559c46e in get<int> (this=0x7fffffff2db0, input=..., output=

    ..., mapper=Unhandled dwarf expression opcode 0xf3

) at include/lsst/afw/table/BaseRecord.h:133

#2  lsst::afw::table::(anonymous namespace)::PersistenceHelper::readRecord (

    this=0x7fffffff2db0, input=..., output=..., mapper=Unhandled dwarf expression opcode 0xf3

)

    at src/table/Exposure.cc:132

#3  0x00007fff955a1d3e in lsst::afw::table::ExposureCatalogT<lsst::afw::table::ExposureRecord>::readFromArchive (archive=..., catalog=...)

    at src/table/Exposure.cc:446

#4  0x00007fffd752900e in lsst::meas::algorithms::CoaddPsf::Factory::read (

    this=Unhandled dwarf expression opcode 0xf3

) at src/CoaddPsf.cc:369

#5  0x00007fff9566a284 in lsst::afw::table::io::InputArchive::Impl::get (this=

    0x1c82860, id=132, self=...) at src/table/io/InputArchive.cc:101

#6  0x00007fff95667275 in lsst::afw::table::io::InputArchive::get (this=Unhandled dwarf expression opcode 0xf3

)

    at src/table/io/InputArchive.cc:184

#7  0x00007fff9577f65e in get<lsst::afw::detection::Psf> (this=0xb09260, 

    fitsfile=Unhandled dwarf expression opcode 0xf3

) at include/lsst/afw/table/io/InputArchive.h:57

#8  lsst::afw::image::ExposureInfo::_readFits (this=0xb09260, fitsfile=Unhandled dwarf expression opcode 0xf3

)

    at src/image/ExposureInfo.cc:275

#9  0x00007fff956e8504 in lsst::afw::image::Exposure<float, unsigned short, float>::_readFits (this=0x1ac6b30, fitsfile=..., bbox=Unhandled dwarf expression opcode 0xf3

)

    at src/image/Exposure.cc:219

The problem appears to be that the introduction of VisitInfo didn't preserve backwards compatibility (I'm not making any statement about the desirability of such backwards compatibility).

There is code and a test for backwards compatibility with old versions of ExposureTable, but it relies on reading a persisted table, not a coadd. Clearly there are differences.

I implemented a fix but it may need more work. I added simple coadds to tests/data, one with a v1 coadd inputs table and the other with v2 and updated testExposureRecord.py to read and check both of them. The new afw code can read old and new coadds, but there is a remaining issue: each visit in coadd.getInfo().getCoaddInputs().visits returns None for getCalib(), getApCorrMap() and getVisitInfo(). This is only expected for getVisitInfo() when reading a v1 coadd.

I also see this problem for Calib and ValidPolygon when reading Hsin-Fang Chiang's coadd /lsst8/hchiang2/coadd/7,7.fits using older afw code, and any coaddTempExp ought to at least have a Calib. So this appears to be an existing bug, though well worth fixing on this ticket, if practical.

I found and fixed two problems:

• The problem stated in the original description was fixed in afw with a minor update to table/Exposure.cc
• Many fields of ExposureInfo were not being saved to CoaddInputs due to a bug in CoaddInputRecorderTask in pipe_tasks.

It is difficult to test the first problem in afw, because generating coadds uses CoaddInputRecorderTask, so I put a new test tests/testCoaddInputs.py in pipe_tasks to test both problems. This test generates a simple coadd, writes it to a FITS file and reads it back in, checking that the information in CoaddInputs is preserved. That is not possible for coadds with version 1 CoaddInputs data, so I used the new test with an older version of the stack to generate a version 1 coadd and saved that. Instructions are in the new test.

I described my fixes in my previous comment on this ticket.

Two caveats:

• The new test does not test CoaddInputs.ccds. I felt it was out of scope for this ticket, but will file a new one if desired.
• The new test is not intended as a full test of CoaddInputRecorderTask, and such a test would be desirable but I'm not even sure a ticket is in order as so many of our tasks lack tests and this is a known issue.

Comments on the GitHub PRs.

Thank you for the quick and helpful review. I have implemented all your suggested changes and will merge after a Jenkins build passes (preferably both py2 and py3, but last time I started the py3 build lsst_py3 was still having problems with the obs_base rename, so it wasn't usable to test afw).