Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4180

Butler: provide API so that a task can define the output dataset type

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None

      Description

      The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

      Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
      Another option is to hard-code some some of the policy in the butler itself:

      • the path template (it could be assembled out of the data id components)
        • if it's hard coded the task must pass the component dataId keys
        • if it's not hard coded the task must provide a template
      • python type
      • storage type

      there could be user overrides too.

        Attachments

          Issue Links

            Activity

            npease Nate Pease [X] (Inactive) created issue -
            npease Nate Pease [X] (Inactive) made changes -
            Field Original Value New Value
            Description The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

            Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
            Another option is to hard-code some some of the policy in the butler itself:
            * the path template (it could be assembled out of the data id components)
             * if it's hard coded the task must pass the component dataId keys
             * if it's not hard coded the task must provide a template
            * python type
            * storage type
            The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

            Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
            Another option is to hard-code some some of the policy in the butler itself:
            * the path template (it could be assembled out of the data id components)
            ** if it's hard coded the task must pass the component dataId keys
            ** if it's not hard coded the task must provide a template
            * python type
            * storage type
            jbecla Jacek Becla made changes -
            Component/s butler [ 12317 ]
            Team Data Access and Database [ 10204 ]
            rowen Russell Owen made changes -
            Link This issue is blocked by DM-4170 [ DM-4170 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            rhl Robert Lupton made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13233 ] This issue links to "Page (Confluence)" [ 13233 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-4553 [ DM-4553 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            jbecla Jacek Becla made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ] This issue links to "Page (Confluence)" [ 13257 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ] This issue links to "Page (Confluence)" [ 13257 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ] This issue links to "Page (Confluence)" [ 13257 ]
            ktl Kian-Tat Lim made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13257 ] This issue links to "Page (Confluence)" [ 13257 ]
            xiuqin Xiuqin Wu [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            xiuqin Xiuqin Wu [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            jbecla Jacek Becla made changes -
            Epic Link DM-4341 [ 21422 ]
            jbecla Jacek Becla made changes -
            Story Points 12
            npease Nate Pease [X] (Inactive) made changes -
            Description The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

            Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
            Another option is to hard-code some some of the policy in the butler itself:
            * the path template (it could be assembled out of the data id components)
            ** if it's hard coded the task must pass the component dataId keys
            ** if it's not hard coded the task must provide a template
            * python type
            * storage type
            The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

            Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
            Another option is to hard-code some some of the policy in the butler itself:
            * the path template (it could be assembled out of the data id components)
            ** if it's hard coded the task must pass the component dataId keys
            ** if it's not hard coded the task must provide a template
            * python type
            * storage type

            there could be user overrides too.
            npease Nate Pease [X] (Inactive) made changes -
            Story Points 12 20
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-4341 [ 21422 ]
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-5262 [ 22985 ]
            npease Nate Pease [X] (Inactive) made changes -
            Rank Ranked higher
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 13239 ] This issue links to "Page (Confluence)" [ 13239 ]
            npease Nate Pease [X] (Inactive) made changes -
            Sprint DB_X16_03 [ 204 ]
            npease Nate Pease [X] (Inactive) made changes -
            Rank Ranked higher
            fritzm Fritz Mueller made changes -
            Assignee Nate Pease [ npease ]
            fritzm Fritz Mueller made changes -
            Epic Link DM-5262 [ 22985 ] DM-4341 [ 21422 ]
            fritzm Fritz Mueller made changes -
            Sprint DB_X16_03 [ 204 ]
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-5518 [ DM-5518 ]
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-4341 [ 21422 ] DM-6035 [ 24345 ]
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            fritzm Fritz Mueller made changes -
            Rank Ranked lower
            fritzm Fritz Mueller made changes -
            Assignee Nate Pease [ npease ]
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Won't Fix [ 10405 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              npease Nate Pease [X] (Inactive)
              Watchers:
              Gregory Dubois-Felsmann, Hsin-Fang Chiang, Nate Pease [X] (Inactive), Russell Owen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.