Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4555

Add initial butler support for remote GET

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None

      Description

      For Get:

      • If the mapper returns a URL:
        • retrieve the URL contents into a file
        • return the path to the file.

      This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.

      Need to solve the cleanup problem of when to delete the file that was downloaded.

      For Put:

      • serialize the object to a temporary file
      • transfer the file to the URL

        Attachments

          Issue Links

            Activity

            No builds found.
            npease Nate Pease [X] (Inactive) created issue -
            npease Nate Pease [X] (Inactive) made changes -
            Field Original Value New Value
            Summary Add initial butler support for remote get Add initial butler support for remote get & put
            npease Nate Pease [X] (Inactive) made changes -
            Description If the mapper returns a URL:
            retrieve the URL contents into a file
            return the path to the file.

            This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.

            Need to solve the cleanup problem of when to delete the file that was downloaded.
            For Get:
            If the mapper returns a URL:
            retrieve the URL contents into a file
            return the path to the file.

            This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.

            Need to solve the cleanup problem of when to delete the file that was downloaded.

            For Put:
            serialize the object to a temporary file
            transfer the file to the URL
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            jbecla Jacek Becla made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13255 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13255 ] This issue links to "Page (Confluence)" [ 13255 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13255 ] This issue links to "Page (Confluence)" [ 13255 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13255 ] This issue links to "Page (Confluence)" [ 13255 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13255 ] This issue links to "Page (Confluence)" [ 13255 ]
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            I assume the mention of "files" in the description is really talking about what an initial implementation might look like, not something that would be visible in the interfaces. It would be good to have an interface that could support in-memory work and, potentially, streaming.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - I assume the mention of "files" in the description is really talking about what an initial implementation might look like, not something that would be visible in the interfaces. It would be good to have an interface that could support in-memory work and, potentially, streaming.
            xiuqin Xiuqin Wu [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            xiuqin Xiuqin Wu [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Story Points 20
            jbecla Jacek Becla made changes -
            Team Data Access and Database [ 10204 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            jbecla Jacek Becla made changes -
            Epic Link DM-4341 [ 21422 ]
            jbecla Jacek Becla made changes -
            Assignee Brian Van Klaveren [ bvan ]
            jbecla Jacek Becla made changes -
            Sprint DB_W16_01 [ 178 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Link This issue is cloned by DM-4698 [ DM-4698 ]
            jbecla Jacek Becla made changes -
            Link This issue relates to DM-4698 [ DM-4698 ]
            jbecla Jacek Becla made changes -
            Link This issue is cloned by DM-4698 [ DM-4698 ]
            jbecla Jacek Becla made changes -
            Summary Add initial butler support for remote get & put Add initial butler support for remote GET
            jbecla Jacek Becla made changes -
            Story Points 20 10
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            this DM-4548 might relate, or this might even depend on it, I'm not sure.

            Show
            npease Nate Pease [X] (Inactive) added a comment - this DM-4548 might relate, or this might even depend on it, I'm not sure.
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-4548 [ DM-4548 ]
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue is duplicated by DM-4554 [ DM-4554 ]
            bvan Brian Van Klaveren made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            npease Nate Pease [X] (Inactive) added a comment - - edited

            Gregory Dubois-Felsmann, in-memory and streaming dataset falls under a different story; DM-4542. This story represents the degenerate case for getting non-local files; copy to local and provide python object to user.

            Show
            npease Nate Pease [X] (Inactive) added a comment - - edited Gregory Dubois-Felsmann , in-memory and streaming dataset falls under a different story; DM-4542 . This story represents the degenerate case for getting non-local files; copy to local and provide python object to user.
            Hide
            npease Nate Pease [X] (Inactive) added a comment - - edited

            Brian Van Klaveren I took a few minutes to think about this. I think this is the way to proceed, and I'll keep thinking about it too. Let's continue talking about it, eh?

            for remote get:

            1. define and implement Access interfaces needed by Mapper (actually CameraMapper), Butler, and possibly Repository.
            2. Access will have to return one or more ButlerLocation instances, with enough information that it can be used later to retrieve the file. - refactor BulterLocation as needed to contain proper URL or whatever.
            3. create Access backend to support the remote connection, file transfer, temp file storage & management (e.g. when to delete the file?)
            4. one of:
              • create test mapper to use the remote
              • refactor camera mapper to use Access and use it to test (this will be a sizable amount of work, est. 10-20 story points), maybe this is something I could do in parallel)
            5. refactor butler _read to use Access instead of reading files from the local filesystem

            cameraMapper for non-direct access only (remove all os.path… foo) includes:

            • setting up the output root
            • finding parent
            • finding & loading registry
            • doing a backup
            • looking up defects
            • finding the camera (_makeCamera)
            Show
            npease Nate Pease [X] (Inactive) added a comment - - edited Brian Van Klaveren I took a few minutes to think about this. I think this is the way to proceed, and I'll keep thinking about it too. Let's continue talking about it, eh? for remote get: define and implement Access interfaces needed by Mapper (actually CameraMapper), Butler, and possibly Repository. Access will have to return one or more ButlerLocation instances, with enough information that it can be used later to retrieve the file. - refactor BulterLocation as needed to contain proper URL or whatever. create Access backend to support the remote connection, file transfer, temp file storage & management (e.g. when to delete the file?) one of: create test mapper to use the remote refactor camera mapper to use Access and use it to test (this will be a sizable amount of work, est. 10-20 story points), maybe this is something I could do in parallel) refactor butler _read to use Access instead of reading files from the local filesystem cameraMapper for non-direct access only (remove all os.path… foo) includes: setting up the output root finding parent finding & loading registry doing a backup looking up defects finding the camera (_makeCamera)
            jbecla Jacek Becla made changes -
            Sprint DB_W16_01 [ 178 ] DB_W16_01, DB_W16_02 [ 178, 179 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Sprint DB_W16_01, DB_W16_02 [ 178, 179 ] DB_W16_01, DB_W16_02, DB_S16_03 [ 178, 179, 199 ]
            jbecla Jacek Becla made changes -
            Sprint DB_W16_01, DB_W16_02, DB_W16_03 [ 178, 179, 199 ] DB_W16_01, DB_W16_02, DB_X16_03 [ 178, 179, 204 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13252 ] This issue links to "Page (Confluence)" [ 13252 ]
            fritzm Fritz Mueller made changes -
            Rank Ranked higher
            fritzm Fritz Mueller made changes -
            Sprint DB_W16_01, DB_W16_02, DB_X16_03 [ 178, 179, 204 ] DB_W16_01, DB_W16_02 [ 178, 179 ]
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-4341 [ 21422 ] DM-6046 [ 24357 ]
            tjenness Tim Jenness made changes -
            Description For Get:
            If the mapper returns a URL:
            retrieve the URL contents into a file
            return the path to the file.

            This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.

            Need to solve the cleanup problem of when to delete the file that was downloaded.

            For Put:
            serialize the object to a temporary file
            transfer the file to the URL
            For Get:
            * If the mapper returns a URL:
            * retrieve the URL contents into a file
            return the path to the file.

            This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.

            Need to solve the cleanup problem of when to delete the file that was downloaded.

            For Put:
            serialize the object to a temporary file
            transfer the file to the URL
            tjenness Tim Jenness made changes -
            Description For Get:
            * If the mapper returns a URL:
            * retrieve the URL contents into a file
            return the path to the file.

            This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.

            Need to solve the cleanup problem of when to delete the file that was downloaded.

            For Put:
            serialize the object to a temporary file
            transfer the file to the URL
            For Get:
            * If the mapper returns a URL:
            ** retrieve the URL contents into a file
            ** return the path to the file.

            This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.

            Need to solve the cleanup problem of when to delete the file that was downloaded.

            For Put:
            * serialize the object to a temporary file
            * transfer the file to the URL
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            bvan Brian Van Klaveren made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Won't Fix [ 10405 ]

              People

              Assignee:
              bvan Brian Van Klaveren
              Reporter:
              npease Nate Pease [X] (Inactive)
              Watchers:
              Gregory Dubois-Felsmann, Nate Pease [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.