Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17950

Make Schema picklable

    Details

      Description

      In order to easily parallelize validate_drp's data loading, we need, at minimum, to be able to pickle/unpickle afw.table.Schema objects. This may or may not be made more difficult by the SchemaImpl design: I suspect a pybind11 master would have some ideas for how to do this relatively quickly.

      Pickling support pybind11 docs for reference:
      https://pybind11.readthedocs.io/en/stable/advanced/classes.html#pickling-support

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Krzysztof Findeisen: I'd love to try to pair code this with you some time.

            Show
            Parejkoj John Parejko added a comment - Krzysztof Findeisen : I'd love to try to pair code this with you some time.
            Hide
            price Paul Price added a comment -

            I suggest you use Schema.__reduce__ to make a dummy Catalog using the Schema, as that will pickle.

            Show
            price Paul Price added a comment - I suggest you use Schema.__reduce__ to make a dummy Catalog using the Schema , as that will pickle.
            Hide
            krzys Krzysztof Findeisen added a comment -

            John Parejko sure, if you can provide the pickling expertise. I don't see any reason why the use of the pImpl idiom should matter.

            Show
            krzys Krzysztof Findeisen added a comment - John Parejko sure, if you can provide the pickling expertise. I don't see any reason why the use of the pImpl idiom should matter.
            Hide
            Parejkoj John Parejko added a comment -

            Russell Owen: are you able to tackle this medium-sized review (~250 lines)? I thought you might have some insights into pickling support.

            afw PR: https://github.com/lsst/afw/pull/440/files

            Big thanks to Krzysztof Findeisen for all the pair coding help putting this together. No way I could have done it solo.

            Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29477/pipeline

            Show
            Parejkoj John Parejko added a comment - Russell Owen : are you able to tackle this medium-sized review (~250 lines)? I thought you might have some insights into pickling support. afw PR: https://github.com/lsst/afw/pull/440/files Big thanks to Krzysztof Findeisen for all the pair coding help putting this together. No way I could have done it solo. Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29477/pipeline
            Hide
            rowen Russell Owen added a comment -

            Looks great. I had a few small requests on github.

            Show
            rowen Russell Owen added a comment - Looks great. I had a few small requests on github.
            Hide
            Parejkoj John Parejko added a comment -

            New jenkins run, since I added some tests and reworked the pybind11 post-review: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29503/pipeline

            Show
            Parejkoj John Parejko added a comment - New jenkins run, since I added some tests and reworked the pybind11 post-review: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29503/pipeline
            Hide
            Parejkoj John Parejko added a comment -

            Thank you for the review: I incorporated your suggestions and Jenkins is happy.

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - Thank you for the review: I incorporated your suggestions and Jenkins is happy. Merged and done.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Russell Owen
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Michael Wood-Vasey, Paul Price, Russell Owen, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel