Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11138

Convert meas_mosaic wcs output to a format directly readable by the butler

    Details

    • Story Points:
      2
    • Sprint:
      DRP S18-3
    • Team:
      Data Release Production

      Description

      meas_mosaic writes its WCS as FITS header with no attached image, which requires loading code to use the following pattern:

      md = butler.get("wcs_md", ...)
      wcs = lsst.afw.image.makeWcs(md)
      

      instead of simply

      wcs = butler.get("wcs", ...)
      

      Using a header-only format also limits us to FITS-standard WCS mappings.

      Because Wcs inherits from afw::table::io::Persistable it already has writeFits and readFits methods that utilize our FITS binary table format, which will be able to save more complex WCS solutions. It's also compatible (or will be soon, on DM-10728) with the "FitsCatalogStorage" butler storage type, so we should be able to fix this by:

      • Redefining "wcs" to be a "FitsCatalogStorage" dataset, instead of a "FitsStorage" exposure in all mappers;
      • modifying meas_mosaic (and possibly jointcal, if needed) to use butler.put directly.
      • modifying any code that consumes the wcs dataset to use butler.get directly.

      In addition, this issue should include creating a simple command-line script that can be used to convert a data repository from the old format to the new one.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Oh, and I think pipe_analysis is fine as is (it only checks on datasetExists, the "getting" is delegated to applyMosaicResultsCatalog in lsst.meas.mosaic.updateExposure).

            Show
            lauren Lauren MacArthur added a comment - Oh, and I think pipe_analysis is fine as is (it only checks on datasetExists, the "getting" is delegated to applyMosaicResultsCatalog in lsst.meas.mosaic.updateExposure ).
            Hide
            jbosch Jim Bosch added a comment -

            Yes, I think the backwards-compatibility code in meas_mosaic removes the need for the script to convert data repositories.

             

            In addition to Jenkins, I'll rerun meas_mosaic on the next weekly outputs and remake at least one coadd patch; I think if that goes through, it should be sufficient to say that this works (hard to imagine this change breaking something without it resulting in a hard failure).

            Show
            jbosch Jim Bosch added a comment - Yes, I think the backwards-compatibility code in meas_mosaic removes the need for the script to convert data repositories.   In addition to Jenkins, I'll rerun meas_mosaic on the next weekly outputs and remake at least one coadd patch; I think if that goes through, it should be sufficient to say that this works (hard to imagine this change breaking something without it resulting in a hard failure).
            Hide
            jbosch Jim Bosch added a comment -

            I'm going to defer the final test and merge of this to next week - since the last weekly, DM-12447 introduced an ABI change in afw, and that means that I can't do the meas_mosaic tests without either rebuilding essentially the full stack on lsst-dev or just waiting for the next weekly.  The latter is a lot easier.

            Show
            jbosch Jim Bosch added a comment - I'm going to defer the final test and merge of this to next week - since the last weekly, DM-12447 introduced an ABI change in afw, and that means that I can't do the meas_mosaic tests without either rebuilding essentially the full stack on lsst-dev or just waiting for the next weekly.  The latter is a lot easier.
            Hide
            jbosch Jim Bosch added a comment -

            meas_mosaic tests mentioned have been completed; I re-ran meas_mosaic and then rebuilt a single-patch coadd.  The pixel values were identical to the same patch in the w_2018_08/DM-13532 run, and I think that's sufficient.

            Running one more Jenkins job, as I think I needed to rebase some packages since the last one.

            Show
            jbosch Jim Bosch added a comment - meas_mosaic tests mentioned have been completed; I re-ran meas_mosaic and then rebuilt a single-patch coadd.  The pixel values were identical to the same patch in the w_2018_08/ DM-13532 run, and I think that's sufficient. Running one more Jenkins job, as I think I needed to rebase some packages since the last one.
            Hide
            jbosch Jim Bosch added a comment -

            Merged to master.

            Show
            jbosch Jim Bosch added a comment - Merged to master.

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Lauren MacArthur
                Watchers:
                Hsin-Fang Chiang, Jim Bosch, John Parejko, Lauren MacArthur, Paul Price, Russell Owen
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel