Details

    • Type: Story
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw, meas_algorithms
    • Labels:
      None
    • Templates:
    • Story Points:
      1
    • Epic Link:
    • Sprint:
      DRP F16-6
    • Team:
      Data Release Production

      Description

      dax_imgserv experiences an error while loading lsst-qserv-dax01:/datasets/gapon/data/DC_2013/coadd/deepCoadd/r/0/321,4.fits.
      This same file should be available as:
      lsst-dev01.ncsa.illinois.edu:/datasets/gapon/data/DC_2013/coadd/deepCoadd/r/0/321,4.fits.
      The error is:

        MalformedArchiveError: 
        File "src/CoaddPsf.cc", line 358, in virtual std::shared_ptr<lsst::afw::table::io::Persistable> lsst::meas::algorithms::CoaddPsf::Factory::read(const InputArchive&, const CatalogVector&) const
          Archive assertion failed: catalogs.size() == 2u {0}
        File "src/table/io/InputArchive.cc", line 105, in std::shared_ptr<lsst::afw::table::io::Persistable> lsst::afw::table::io::InputArchive::Impl::get(int, const lsst::afw::table::io::InputArchive&)
      

      On lsst-qserv-dax01.ncsa.illinois.edu this can be reproduced by

      $ source /software/lsstsw/stack/loadLSST.bash
      $ setup meas_algorithms
      python
      >>> import lsst.afw.image
      >>> e = lsst.afw.image.ExposureF("/datasets/gapon/data/DC_2013/coadd/deepCoadd/r/0/321,4/coadd-r-0-321,4.fits")
      

        Issue Links

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          I've found the problem; it's this commit from July 2013:

          commit 53e61faef61c9e5ee47d56b89e41f18325d96295
          Author: Jim Bosch <jbosch@astro.princeton.edu>
          Date: Wed Jul 10 13:16:53 2013 -0400

          Overhaul CoaddPsf persistence to make sure everything round-trips (#2948)

          This change breaks compatibility with previously-saved CoaddPsfs.

          Previously, we were recomputing the average position, using slightly different
          inputs to the algorithm when we unpersisted relative to when we originally
          constructed the CoaddPsf. In addition, we weren't saving the warping
          control parameters at all.

          Unfortunately, it doesn't lend itself to an easy solution; it looks like we weren't saving enough information to fully reconstruct our PSF models back when this file was written. I could probably add some backwards compatibility code that would let us read the file in with a slightly incorrect PSF model, since I'm guessing rebuilding the coadds is out of scope for the PDAC, but we should probably not rely on this generation of data products any longer than we have to, since we won't ever be able to reconstruct the original PSF models exactly.

          Show
          jbosch Jim Bosch added a comment - I've found the problem; it's this commit from July 2013: commit 53e61faef61c9e5ee47d56b89e41f18325d96295 Author: Jim Bosch <jbosch@astro.princeton.edu> Date: Wed Jul 10 13:16:53 2013 -0400 Overhaul CoaddPsf persistence to make sure everything round-trips (#2948) This change breaks compatibility with previously-saved CoaddPsfs. Previously, we were recomputing the average position, using slightly different inputs to the algorithm when we unpersisted relative to when we originally constructed the CoaddPsf. In addition, we weren't saving the warping control parameters at all. Unfortunately, it doesn't lend itself to an easy solution; it looks like we weren't saving enough information to fully reconstruct our PSF models back when this file was written. I could probably add some backwards compatibility code that would let us read the file in with a slightly incorrect PSF model, since I'm guessing rebuilding the coadds is out of scope for the PDAC, but we should probably not rely on this generation of data products any longer than we have to, since we won't ever be able to reconstruct the original PSF models exactly.
          Hide
          jbosch Jim Bosch added a comment -

          Russell Owen, could you take a look at this change to add more backwards compatibility to CoaddPsf and the ExposureCatalog it relies on? Some of the ExposureCatalog code should be quite familiar to you, though in order to support even older {{ExposureCatalog}}s I did switch some things around.

          Changes are in afw and meas_algorithms; PRs have already been created.

          Show
          jbosch Jim Bosch added a comment - Russell Owen , could you take a look at this change to add more backwards compatibility to CoaddPsf and the ExposureCatalog it relies on? Some of the ExposureCatalog code should be quite familiar to you, though in order to support even older {{ExposureCatalog}}s I did switch some things around. Changes are in afw and meas_algorithms; PRs have already been created.
          Hide
          rowen Russell Owen added a comment -

          This looks very nice (especially the changes to afw). I requested a few trivial comment tweaks on github.

          Show
          rowen Russell Owen added a comment - This looks very nice (especially the changes to afw). I requested a few trivial comment tweaks on github.
          Hide
          jbosch Jim Bosch added a comment -

          Merged to master.

          Show
          jbosch Jim Bosch added a comment - Merged to master.

            People

            • Assignee:
              jbosch Jim Bosch
              Reporter:
              jgates John Gates
              Reviewers:
              Russell Owen
              Watchers:
              Jim Bosch, John Gates, John Swinbank, Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development

                  Agile