Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4180

Butler: provide API so that a task can define the output dataset type

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None

      Description

      The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

      Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
      Another option is to hard-code some some of the policy in the butler itself:

      • the path template (it could be assembled out of the data id components)
        • if it's hard coded the task must pass the component dataId keys
        • if it's not hard coded the task must provide a template
      • python type
      • storage type

      there could be user overrides too.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-4170 Butler: move configuration (.paf) file into repository

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-4553 Move butler from daf_persistence to daf_butler

          • Won't Fix

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-5518 Create proposal & RFC for Butler API to define output dataset type

          • Won't Fix
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            npease Nate Pease [X] (Inactive) created issue -
            npease Nate Pease [X] (Inactive) made changes -
            Field Original Value New Value
            Description The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

            Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
            Another option is to hard-code some some of the policy in the butler itself:
            * the path template (it could be assembled out of the data id components)
             * if it's hard coded the task must pass the component dataId keys
             * if it's not hard coded the task must provide a template
            * python type
            * storage type             		The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

            Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
            Another option is to hard-code some some of the policy in the butler itself:
            * the path template (it could be assembled out of the data id components)
            ** if it's hard coded the task must pass the component dataId keys
            ** if it's not hard coded the task must provide a template
            * python type
            * storage type
            jbecla Jacek Becla made changes -
            Component/s butler [ 12317 ]
            Team Data Access and Database [ 10204 ]
            rowen Russell Owen made changes -
            Link This issue is blocked by DM-4170 [ DM-4170 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            rhl Robert Lupton made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            Hide
            Permalink
            npease Nate Pease [X] (Inactive) added a comment -

            Since we think many people will want to refactor to use this feature, it might be good to do this and DM-4553 at the same time so that they can do that refactor at the same time.

            Show
            npease Nate Pease [X] (Inactive) added a comment - Since we think many people will want to refactor to use this feature, it might be good to do this and DM-4553 at the same time so that they can do that refactor at the same time.
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-4553 [ DM-4553 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            jbecla Jacek Becla made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ] This issue links to "Page (Confluence)" [ 13257 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ] This issue links to "Page (Confluence)" [ 13257 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ] This issue links to "Page (Confluence)" [ 13257 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ] This issue links to "Page (Confluence)" [ 13257 ]
            Hide
            Permalink
            gpdf Gregory Dubois-Felsmann added a comment -

            I think where "task" is being used in the description of this story something more like SuperTask might be meant? That would be more consistent with the Apps groups' existing concept that Tasks should concern themselves with Python-domain objects and not with I/O (even through the Butler).

            Show
            gpdf Gregory Dubois-Felsmann added a comment - I think where "task" is being used in the description of this story something more like SuperTask might be meant? That would be more consistent with the Apps groups' existing concept that Tasks should concern themselves with Python-domain objects and not with I/O (even through the Butler).
            xiuqin Xiuqin Wu [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            xiuqin Xiuqin Wu [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            jbecla Jacek Becla made changes -
            Epic Link DM-4341 [ 21422 ]
            jbecla Jacek Becla made changes -
            Story Points 12
            npease Nate Pease [X] (Inactive) made changes -
            Description The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

            Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
            Another option is to hard-code some some of the policy in the butler itself:
            * the path template (it could be assembled out of the data id components)
            ** if it's hard coded the task must pass the component dataId keys
            ** if it's not hard coded the task must provide a template
            * python type
            * storage type             		The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

            Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
            Another option is to hard-code some some of the policy in the butler itself:
            * the path template (it could be assembled out of the data id components)
            ** if it's hard coded the task must pass the component dataId keys
            ** if it's not hard coded the task must provide a template
            * python type
            * storage type

            there could be user overrides too.
            npease Nate Pease [X] (Inactive) made changes -
            Story Points 12 20
            Hide
            Permalink
            npease Nate Pease [X] (Inactive) added a comment -

            in conversation with KT, changed story points to 20, there is a fair bit of design to get the genre and persistent storage features right, and we will need to take the time to do that (plus do the work)

            Show
            npease Nate Pease [X] (Inactive) added a comment - in conversation with KT, changed story points to 20, there is a fair bit of design to get the genre and persistent storage features right, and we will need to take the time to do that (plus do the work)
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-4341 [ 21422 ]
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-5262 [ 22985 ]
            npease Nate Pease [X] (Inactive) made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Sprint DB_X16_03 [ 204 ]
            npease Nate Pease [X] (Inactive) made changes -
            Rank Ranked higher
            fritzm Fritz Mueller made changes -
            Assignee Nate Pease [ npease ]
            fritzm Fritz Mueller made changes -
            Epic Link DM-5262 [ 22985 ] DM-4341 [ 21422 ]
            fritzm Fritz Mueller made changes -
            Sprint DB_X16_03 [ 204 ]
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-5518 [ DM-5518 ]
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-4341 [ 21422 ] DM-6035 [ 24345 ]
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            fritzm Fritz Mueller made changes -
            Assignee Nate Pease [ npease ]
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Closing this since is has been overtaken by PipelineTask + Gen 3

            Show
            tjenness Tim Jenness added a comment - Closing this since is has been overtaken by PipelineTask + Gen 3
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Won't Fix [ 10405 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              npease Nate Pease [X] (Inactive)
              Watchers:
              Gregory Dubois-Felsmann, Hsin-Fang Chiang, Nate Pease [X] (Inactive), Russell Owen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.