Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25377

Update base.Packages serialization to support bytes

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: base
    • Labels:
      None
    • Story Points:
      1
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      In DM-25325 support was added for YAML serialization but the serialization still requires a file name to be given.

      It would be helpful for butler gen3 s3 datastore to support a serialization to from bytes by updating the PackagesFormatter (created in DM-25327).

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            This is a minor reorganizing to allow the byte stream to be accessed.

            Show
            tjenness Tim Jenness added a comment - This is a minor reorganizing to allow the byte stream to be accessed.
            Hide
            ktl Kian-Tat Lim added a comment -

            For a list of packages there's no worry about size and scalability, but overall this "read-everything-into-memory-and-translate" paradigm is somewhat dangerous.

            Is there a particular reason to use classmethod over staticmethod?

            Show
            ktl Kian-Tat Lim added a comment - For a list of packages there's no worry about size and scalability, but overall this "read-everything-into-memory-and-translate" paradigm is somewhat dangerous. Is there a particular reason to use classmethod over staticmethod ?
            Hide
            tjenness Tim Jenness added a comment -

            I used classmethod because it was classmethod before. I imagine I could do an isinstance check in there as well to make sure that what we get back is the right type.

            The paradigm is to allow gen3 butler to use the s3 put/get without going through a temporary file. Whether this happens or not can be controlled by the formatter itself. The YAML/pickle/JSON gen3 formatters already support this mode. For Packages it should never be a problem. I get your point that it might be better for a Formatter to not reuse the code path of always going through bytes before writing to files so that the formatter can be configured to dump to disk without blowing up memory. That's downstream from this ticket though.

            Show
            tjenness Tim Jenness added a comment - I used classmethod because it was classmethod before. I imagine I could do an isinstance check in there as well to make sure that what we get back is the right type. The paradigm is to allow gen3 butler to use the s3 put/get without going through a temporary file. Whether this happens or not can be controlled by the formatter itself. The YAML/pickle/JSON gen3 formatters already support this mode. For Packages it should never be a problem. I get your point that it might be better for a Formatter to not reuse the code path of always going through bytes before writing to files so that the formatter can be configured to dump to disk without blowing up memory. That's downstream from this ticket though.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.