Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4555

Add initial butler support for remote GET

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None

      Description

      For Get:

      • If the mapper returns a URL:
        • retrieve the URL contents into a file
        • return the path to the file.

      This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.

      Need to solve the cleanup problem of when to delete the file that was downloaded.

      For Put:

      • serialize the object to a temporary file
      • transfer the file to the URL

        Attachments

          Issue Links

            Activity

            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            I assume the mention of "files" in the description is really talking about what an initial implementation might look like, not something that would be visible in the interfaces. It would be good to have an interface that could support in-memory work and, potentially, streaming.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - I assume the mention of "files" in the description is really talking about what an initial implementation might look like, not something that would be visible in the interfaces. It would be good to have an interface that could support in-memory work and, potentially, streaming.
            Hide
            npease Nate Pease added a comment -

            this DM-4548 might relate, or this might even depend on it, I'm not sure.

            Show
            npease Nate Pease added a comment - this DM-4548 might relate, or this might even depend on it, I'm not sure.
            Hide
            npease Nate Pease added a comment - - edited

            Gregory Dubois-Felsmann, in-memory and streaming dataset falls under a different story; DM-4542. This story represents the degenerate case for getting non-local files; copy to local and provide python object to user.

            Show
            npease Nate Pease added a comment - - edited Gregory Dubois-Felsmann , in-memory and streaming dataset falls under a different story; DM-4542 . This story represents the degenerate case for getting non-local files; copy to local and provide python object to user.
            Hide
            npease Nate Pease added a comment - - edited

            Brian Van Klaveren I took a few minutes to think about this. I think this is the way to proceed, and I'll keep thinking about it too. Let's continue talking about it, eh?

            for remote get:

            1. define and implement Access interfaces needed by Mapper (actually CameraMapper), Butler, and possibly Repository.
            2. Access will have to return one or more ButlerLocation instances, with enough information that it can be used later to retrieve the file. - refactor BulterLocation as needed to contain proper URL or whatever.
            3. create Access backend to support the remote connection, file transfer, temp file storage & management (e.g. when to delete the file?)
            4. one of:
              • create test mapper to use the remote
              • refactor camera mapper to use Access and use it to test (this will be a sizable amount of work, est. 10-20 story points), maybe this is something I could do in parallel)
            5. refactor butler _read to use Access instead of reading files from the local filesystem

            cameraMapper for non-direct access only (remove all os.path… foo) includes:

            • setting up the output root
            • finding parent
            • finding & loading registry
            • doing a backup
            • looking up defects
            • finding the camera (_makeCamera)
            Show
            npease Nate Pease added a comment - - edited Brian Van Klaveren I took a few minutes to think about this. I think this is the way to proceed, and I'll keep thinking about it too. Let's continue talking about it, eh? for remote get: define and implement Access interfaces needed by Mapper (actually CameraMapper), Butler, and possibly Repository. Access will have to return one or more ButlerLocation instances, with enough information that it can be used later to retrieve the file. - refactor BulterLocation as needed to contain proper URL or whatever. create Access backend to support the remote connection, file transfer, temp file storage & management (e.g. when to delete the file?) one of: create test mapper to use the remote refactor camera mapper to use Access and use it to test (this will be a sizable amount of work, est. 10-20 story points), maybe this is something I could do in parallel) refactor butler _read to use Access instead of reading files from the local filesystem cameraMapper for non-direct access only (remove all os.path… foo) includes: setting up the output root finding parent finding & loading registry doing a backup looking up defects finding the camera (_makeCamera)

              People

              • Assignee:
                bvan Brian Van Klaveren
                Reporter:
                npease Nate Pease
                Watchers:
                Gregory Dubois-Felsmann, Nate Pease
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel