Details
-
Type:
Story
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: butler
-
Labels:None
-
Story Points:10
-
Epic Link:
-
Sprint:DB_W16_01, DB_W16_02
-
Team:Data Access and Database
Description
For Get:
- If the mapper returns a URL:
- retrieve the URL contents into a file
- return the path to the file.
This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol.
Need to solve the cleanup problem of when to delete the file that was downloaded.
For Put:
- serialize the object to a temporary file
- transfer the file to the URL
Attachments
Issue Links
Activity
| Field | Original Value | New Value |
|---|---|---|
| Summary | Add initial butler support for remote get | Add initial butler support for remote get & put |
| Description |
If the mapper returns a URL:
retrieve the URL contents into a file return the path to the file. This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol. Need to solve the cleanup problem of when to delete the file that was downloaded. |
For Get:
If the mapper returns a URL: retrieve the URL contents into a file return the path to the file. This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol. Need to solve the cleanup problem of when to delete the file that was downloaded. For Put: serialize the object to a temporary file transfer the file to the URL |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13255 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13255 ] | This issue links to "Page (Confluence)" [ 13255 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13255 ] | This issue links to "Page (Confluence)" [ 13255 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13255 ] | This issue links to "Page (Confluence)" [ 13255 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13255 ] | This issue links to "Page (Confluence)" [ 13255 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Story Points | 20 |
| Team | Data Access and Database [ 10204 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Epic Link | DM-4341 [ 21422 ] |
| Assignee | Brian Van Klaveren [ bvan ] |
| Sprint | DB_W16_01 [ 178 ] |
| Rank | Ranked higher |
| Summary | Add initial butler support for remote get & put | Add initial butler support for remote GET |
| Story Points | 20 | 10 |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
this DM-4548 might relate, or this might even depend on it, I'm not sure.
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
Gregory Dubois-Felsmann, in-memory and streaming dataset falls under a different story; DM-4542. This story represents the degenerate case for getting non-local files; copy to local and provide python object to user.
Brian Van Klaveren I took a few minutes to think about this. I think this is the way to proceed, and I'll keep thinking about it too. Let's continue talking about it, eh?
for remote get:
- define and implement Access interfaces needed by Mapper (actually CameraMapper), Butler, and possibly Repository.
- Access will have to return one or more ButlerLocation instances, with enough information that it can be used later to retrieve the file. - refactor BulterLocation as needed to contain proper URL or whatever.
- create Access backend to support the remote connection, file transfer, temp file storage & management (e.g. when to delete the file?)
- one of:
- create test mapper to use the remote
- refactor camera mapper to use Access and use it to test (this will be a sizable amount of work, est. 10-20 story points), maybe this is something I could do in parallel)
- refactor butler _read to use Access instead of reading files from the local filesystem
cameraMapper for non-direct access only (remove all os.path… foo) includes:
- setting up the output root
- finding parent
- finding & loading registry
- doing a backup
- looking up defects
- finding the camera (_makeCamera)
| Sprint | DB_W16_01 [ 178 ] | DB_W16_01, DB_W16_02 [ 178, 179 ] |
| Rank | Ranked higher |
| Sprint | DB_W16_01, DB_W16_02 [ 178, 179 ] | DB_W16_01, DB_W16_02, DB_S16_03 [ 178, 179, 199 ] |
| Sprint | DB_W16_01, DB_W16_02, DB_W16_03 [ 178, 179, 199 ] | DB_W16_01, DB_W16_02, DB_X16_03 [ 178, 179, 204 ] |
| Rank | Ranked higher |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 13252 ] | This issue links to "Page (Confluence)" [ 13252 ] |
| Rank | Ranked higher |
| Sprint | DB_W16_01, DB_W16_02, DB_X16_03 [ 178, 179, 204 ] | DB_W16_01, DB_W16_02 [ 178, 179 ] |
| Rank | Ranked lower |
| Rank | Ranked lower |
| Epic Link | DM-4341 [ 21422 ] |
|
| Description |
For Get:
If the mapper returns a URL: retrieve the URL contents into a file return the path to the file. This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol. Need to solve the cleanup problem of when to delete the file that was downloaded. For Put: serialize the object to a temporary file transfer the file to the URL |
For Get:
* If the mapper returns a URL: * retrieve the URL contents into a file return the path to the file. This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol. Need to solve the cleanup problem of when to delete the file that was downloaded. For Put: serialize the object to a temporary file transfer the file to the URL |
| Description |
For Get:
* If the mapper returns a URL: * retrieve the URL contents into a file return the path to the file. This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol. Need to solve the cleanup problem of when to delete the file that was downloaded. For Put: serialize the object to a temporary file transfer the file to the URL |
For Get:
* If the mapper returns a URL: ** retrieve the URL contents into a file ** return the path to the file. This will be optimized in subsequent stories by "add read support for various transports to the afw object readers". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol. Need to solve the cleanup problem of when to delete the file that was downloaded. For Put: * serialize the object to a temporary file * transfer the file to the URL |
| Rank | Ranked lower |
| Resolution | Done [ 10000 ] | |
| Status | In Progress [ 3 ] | Won't Fix [ 10405 ] |
I assume the mention of "files" in the description is really talking about what an initial implementation might look like, not something that would be visible in the interfaces. It would be good to have an interface that could support in-memory work and, potentially, streaming.