Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13391

Develop Gen3 Butler sufficient to support CmdLineTask execution

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Epic Name:
      s18-mw-butler-1
    • Team:
      Architecture

      Description

      Develop the Gen3 Butler components as needed to support running all CmdLineTasks included in ci_hsc, via a testing script that calls their run methods directly (instead of using parseAndRun.

      This will require (at least):

      • A Butler class capable of supporting at least the get, put, and datasetExists operations. Expected to be developed by a combination of Pim Schellart [X] or Tim Jenness.
      • A SQLite-based Registry implementation capable of supporting those operations. Expected to be developed primarily by Pim Schellart [X].
      • A local POSIX Datastore implementation capable of supporting those operations. Expected to be developed primarily by Tim Jenness.
      • A shim layer that allows these operations to be done with the Gen3 Butler using the Gen2 Butler API (particularly a temporary DataRef class). Expected to be developed primarily by Pim Schellart [X].
      • A functional prototype definition of the Registry schema. Initial structure will be developed initially by Jim Bosch according to the general approach agreed at the January Middleware Meeting, with review support and additional content provided by others (especially Michelle Gower).
      • A script to allow Gen2 Data Repositories to be loaded into Gen3 Registry databases and accessed by Gen3 Datastores. Expected to be developed primarily by Jim Bosch.
      • A functional prototype for the Gen3 Butler's camera-customization interface (i.e. obs* package interface). Expected to be developed primarily by Jim Bosch, in consultation with Simon Krughoff and the obs redesign working group.
      • A functional implementation of the Gen3 Butler's camera-customization interface for HSC. Expected to be developed primarily by Jim Bosch.
      • Additional Registry/Datastore/shim functionality or minor CmdLineTask changes to support any use of methods beyond get, set, and datasetExists in CmdLineTasks (e.g. use of Butler.subset).

      The completion of this epic is the first of two S18 Middleware milestones, with completion criteria the ability to run a custom version of the ci_hsc test package that uses the Gen3 Butler, with minimally modified versions of CmdLineTasks it executes. Our goal is to complete these tests by April 13.

      This Epic should be used instead of the existing cycle-wide per-Team Epics by middleware developers members to organize the stories within its scope. Fritz Mueller will handle filling in the per-Team Epics as needed for loading PMCS.

        Attachments

          Issue Links

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          DM-14563

          Fix ci_hsc repo creation

          0.5 Jim Bosch Done
           
          DM-13358

          Add support for composites to Datastore prototype

          10 Tim Jenness Done
           
          DM-13350

          Move DatasetType template to Datastore configuration (override)

          6 Tim Jenness Done
           
          DM-13349

          Make StorageClass configurable in Butler Datastore

          4 Tim Jenness Done
           
          DM-14658

          Fix Gen3 SkyMap registration after polygon API changes

          0.5 Jim Bosch Done
           
          DM-13814

          Reorganize composites handling in datastore/butler

          10 Tim Jenness Done
           
          DM-13402

          Deal with comments from review of LDM-592 from butler gen3 review

          6 Tim Jenness Done
           
          DM-13217

          Deal with LDM-556 butler review comments

          6 Tim Jenness Done
           
          DM-13599

          Update copyright info following RFC-45

          0.5 Tim Jenness Done
           
          DM-13600

          Add YAML formatter

          1 Tim Jenness Done
           
          DM-13374

          Deconstruct Butler prototype for redesign

          2 Pim Schellart [X] (Inactive) Done
           
          DM-13371

          Enable flake8 testing in daf_butler

          0.5 Tim Jenness Done
           
          DM-13364

          Attend Butler/SuperTask Meeting at Princeton

          8 Tim Jenness Done
           
          DM-14428

          Add skypix and spatial relationships to Gen2->Gen3 conversion

          4 Jim Bosch Done
           
          DM-14378

          Add Gen3 conversion scripting and tests to ci_hsc

          2 Jim Bosch Done
           
          DM-13753

          Enable sphinx doc building in daf_butler

          2 Tim Jenness Done
           
          DM-13342

          Refactor Datastore prototype to improve orthogonality

          0.5 Pim Schellart [X] (Inactive) Done
           
          DM-13507

          Add stable hash to SkyMap objects

          2 Jim Bosch Done
           
          DM-13370

          Rebrand butler_prototype as daf_butler

          0.5 Tim Jenness Done
           
          DM-14407

          Finish implementing concrete composite component-writing

          4 Jim Bosch Invalid
           
          DM-14408

          Implement efficient Exposure component access and slicing

          2 Jim Bosch Invalid
           
          DM-14666

          Store Python types with PosixDatastore and remove redundant StorageClasses

          2 Unassigned Invalid

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Let's set the team on this epic to DAX, on the basis that all epics are required to have a team, and Fritz Mueller is coordinating this work.

            Show
            swinbank John Swinbank added a comment - Let's set the team on this epic to DAX, on the basis that all epics are required to have a team, and Fritz Mueller is coordinating this work.
            Hide
            jbosch Jim Bosch added a comment -

            The original stated goal for this informal Epic was never actually done, but that's because we decided there was a better way to get to the next step (DM-13392), which is now done, too.

            Show
            jbosch Jim Bosch added a comment - The original stated goal for this informal Epic was never actually done, but that's because we decided there was a better way to get to the next step ( DM-13392 ), which is now done, too.
            Hide
            swinbank John Swinbank added a comment -

            We can't close this epic while it contains DM-14408 which is still marked as “to do”. Please either mark that ticket as “done” (if it is) or move it to another epic (if it isn't) before closing this.

            Show
            swinbank John Swinbank added a comment - We can't close this epic while it contains DM-14408 which is still marked as “to do”. Please either mark that ticket as “done” (if it is) or move it to another epic (if it isn't) before closing this.
            Hide
            jbosch Jim Bosch added a comment -

            Closing again.  DM-14408 has been marked as invalid; the work was done on another ticket (DM-15189).

            Show
            jbosch Jim Bosch added a comment - Closing again.  DM-14408 has been marked as invalid; the work was done on another ticket ( DM-15189 ).

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Fritz Mueller, Jim Bosch, John Swinbank, Kian-Tat Lim, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Due:
                Created:
                Updated:
                Resolved:

                  Summary Panel