Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4180

Butler: provide API so that a task can define the output dataset type

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None

      Description

      The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type.

      Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the "genre"), this potentially reduces the amount the task has to specify.
      Another option is to hard-code some some of the policy in the butler itself:

      • the path template (it could be assembled out of the data id components)
        • if it's hard coded the task must pass the component dataId keys
        • if it's not hard coded the task must provide a template
      • python type
      • storage type

      there could be user overrides too.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-4170 Butler: move configuration (.paf) file into repository

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-4553 Move butler from daf_persistence to daf_butler

          • Won't Fix

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-5518 Create proposal & RFC for Butler API to define output dataset type

          • Won't Fix
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            npease Nate Pease [X] (Inactive) added a comment -

            Since we think many people will want to refactor to use this feature, it might be good to do this and DM-4553 at the same time so that they can do that refactor at the same time.

            Show
            npease Nate Pease [X] (Inactive) added a comment - Since we think many people will want to refactor to use this feature, it might be good to do this and DM-4553 at the same time so that they can do that refactor at the same time.
            Hide
            Permalink
            gpdf Gregory Dubois-Felsmann added a comment -

            I think where "task" is being used in the description of this story something more like SuperTask might be meant? That would be more consistent with the Apps groups' existing concept that Tasks should concern themselves with Python-domain objects and not with I/O (even through the Butler).

            Show
            gpdf Gregory Dubois-Felsmann added a comment - I think where "task" is being used in the description of this story something more like SuperTask might be meant? That would be more consistent with the Apps groups' existing concept that Tasks should concern themselves with Python-domain objects and not with I/O (even through the Butler).
            Hide
            Permalink
            npease Nate Pease [X] (Inactive) added a comment -

            in conversation with KT, changed story points to 20, there is a fair bit of design to get the genre and persistent storage features right, and we will need to take the time to do that (plus do the work)

            Show
            npease Nate Pease [X] (Inactive) added a comment - in conversation with KT, changed story points to 20, there is a fair bit of design to get the genre and persistent storage features right, and we will need to take the time to do that (plus do the work)
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Closing this since is has been overtaken by PipelineTask + Gen 3

            Show
            tjenness Tim Jenness added a comment - Closing this since is has been overtaken by PipelineTask + Gen 3

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              npease Nate Pease [X] (Inactive)
              Watchers:
              Gregory Dubois-Felsmann, Hsin-Fang Chiang, Nate Pease [X] (Inactive), Russell Owen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.