Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4180

Butler: provide API so that a task can define the output dataset type

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None

      Description

      The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

      Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
      Another option is to hard-code some some of the policy in the butler itself:

      • the path template (it could be assembled out of the data id components)
        • if it's hard coded the task must pass the component dataId keys
        • if it's not hard coded the task must provide a template
      • python type
      • storage type

      there could be user overrides too.

        Attachments

          Issue Links

            Activity

            Hide
            npease Nate Pease added a comment -

            Since we think many people will want to refactor to use this feature, it might be good to do this and DM-4553 at the same time so that they can do that refactor at the same time.

            Show
            npease Nate Pease added a comment - Since we think many people will want to refactor to use this feature, it might be good to do this and DM-4553 at the same time so that they can do that refactor at the same time.
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            I think where "task" is being used in the description of this story something more like SuperTask might be meant? That would be more consistent with the Apps groups' existing concept that Tasks should concern themselves with Python-domain objects and not with I/O (even through the Butler).

            Show
            gpdf Gregory Dubois-Felsmann added a comment - I think where "task" is being used in the description of this story something more like SuperTask might be meant? That would be more consistent with the Apps groups' existing concept that Tasks should concern themselves with Python-domain objects and not with I/O (even through the Butler).
            Hide
            npease Nate Pease added a comment -

            in conversation with KT, changed story points to 20, there is a fair bit of design to get the genre and persistent storage features right, and we will need to take the time to do that (plus do the work)

            Show
            npease Nate Pease added a comment - in conversation with KT, changed story points to 20, there is a fair bit of design to get the genre and persistent storage features right, and we will need to take the time to do that (plus do the work)
            Hide
            tjenness Tim Jenness added a comment -

            Closing this since is has been overtaken by PipelineTask + Gen 3

            Show
            tjenness Tim Jenness added a comment - Closing this since is has been overtaken by PipelineTask + Gen 3

              People

              • Assignee:
                Unassigned
                Reporter:
                npease Nate Pease
                Watchers:
                Gregory Dubois-Felsmann, Hsin-Fang Chiang, Nate Pease, Russell Owen, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel