Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15653

Add native yaml serialization support to daf_base

    XMLWordPrintable

    Details

      Description

      In DM-14503 support for YAML serialization of PropertySet and PropertyList was added to daf_persistence. To allow gen3 butler to serialize these using YAML it makes sense to move the YAML support directly into daf_base. This would require a pyyaml dependency on daf_base but we already use yaml in many places and it's in the core python dependency list. YAML support could be optional in the sense that it is only enabled if yaml can be found.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15676 Make PropertySet/List more dict-like

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14503 Finish implementing YAML storage for PropertyList/PropertySet

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            I've moved the code and added some tests. I was a bit disconcerted by the lack of _eq or len_ support.

            Show
            tjenness Tim Jenness added a comment - I've moved the code and added some tests. I was a bit disconcerted by the lack of _ eq or len _ support.
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            Minor suggestions for improvement, primarily to docstrings.

            Show
            ktl Kian-Tat Lim added a comment - Minor suggestions for improvement, primarily to docstrings.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            After discussion on Slack we have decided that this ticket should also remove the YAML property set/list files from daf_persistence, added in DM-14503, since they are now superfluous and ultimately confusing.

            Show
            tjenness Tim Jenness added a comment - After discussion on Slack we have decided that this ticket should also remove the YAML property set/list files from daf_persistence, added in DM-14503 , since they are now superfluous and ultimately confusing.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Kian-Tat Lim I've removed YAML daf_base from daf_persistence and all tests pass on ci_hsc and lsst_ci. Russell Owen noted that nested serialization might not work so I tested that and it does work except the class of the nested objects is lost during pickling/yamlification. Fixing that would require a change to the pickling code and I've decided that should be done on another ticket because we have to consider compatibility with instances that are already serialized.

            Show
            tjenness Tim Jenness added a comment - Kian-Tat Lim I've removed YAML daf_base from daf_persistence and all tests pass on ci_hsc and lsst_ci. Russell Owen noted that nested serialization might not work so I tested that and it does work except the class of the nested objects is lost during pickling/yamlification. Fixing that would require a change to the pickling code and I've decided that should be done on another ticket because we have to consider compatibility with instances that are already serialized.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Jim Bosch, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.