Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9036

butler PickleStorage is not py2/py3 cross-compatible

    Details

    • Templates:
    • Team:
      Data Access and Database

      Description

      The butler has a "PickleStorage" type, which is used when persisting an lsst.skymap. It defaults to pickle.HIGHEST_PROTOCOL (posixStorage.py:203), which means that if a skymap is generated in python3, it will be unusable in python2.

      Possible solutions:

      • Replace pickle.HIGHEST_PROTOCOL with 2 (easy, but we're still saving pickle files, which aren't really meant for this sort of persistence).
      • Write a new persistence type for skymap and use it (and deprecate/remove PickleStorage in the butler); possibilities include YAML or HDF5.
      • Go all-in on python3, so we can use protocol=4 without trouble.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ktl Kian-Tat Lim
                Reporter:
                Parejkoj John Parejko
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Kian-Tat Lim, Nate Pease, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel