Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13391

Develop Gen3 Butler sufficient to support CmdLineTask execution

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Templates:
    • Epic Name:
      s18-mw-butler-1
    • Team:
      Data Access and Database

      Description

      Develop the Gen3 Butler components as needed to support running all CmdLineTasks included in ci_hsc, via a testing script that calls their run methods directly (instead of using parseAndRun.

      This will require (at least):

      • A Butler class capable of supporting at least the get, put, and datasetExists¬†operations. Expected to be developed by a combination of Pim Schellart [X] or Tim Jenness.
      • A SQLite-based Registry implementation capable of supporting those operations. Expected to be developed primarily by Pim Schellart [X].
      • A local POSIX Datastore implementation capable of supporting those operations. Expected to be developed primarily by Tim Jenness.
      • A shim layer that allows these operations to be done with the Gen3 Butler using the Gen2 Butler API (particularly a temporary DataRef class). Expected to be developed primarily by Pim Schellart [X].
      • A functional prototype definition of the Registry schema. Initial structure will be developed initially by Jim Bosch according to the general approach agreed at the January Middleware Meeting, with review support and additional content provided by others (especially Michelle Gower).
      • A script to allow Gen2 Data Repositories to be loaded into Gen3 Registry databases and accessed by Gen3 Datastores. Expected to be developed primarily by Jim Bosch.
      • A functional prototype for the Gen3 Butler's camera-customization interface (i.e. obs* package interface). Expected to be developed primarily by Jim Bosch, in consultation with Simon Krughoff and the obs redesign working group.
      • A functional implementation of the Gen3 Butler's camera-customization interface for HSC. Expected to be developed primarily by Jim Bosch.
      • Additional Registry/Datastore/shim functionality or minor CmdLineTask changes to support any use of methods beyond get, set, and datasetExists in CmdLineTasks (e.g. use of Butler.subset).

      The completion of this epic is the first of two S18 Middleware milestones, with completion criteria the ability to run a custom version of the ci_hsc test package that uses the Gen3 Butler, with minimally modified versions of CmdLineTasks it executes. Our goal is to complete these tests by April 13.

      This Epic should be used instead of the existing cycle-wide per-Team Epics by middleware developers members to organize the stories within its scope. Fritz Mueller will handle filling in the per-Team Epics as needed for loading PMCS.

        Attachments

          Issue Links

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          DM-14428

          Add skypix and spatial relationships to Gen2->Gen3 conversion

          4 Jim Bosch Done
           
          DM-14378

          Add Gen3 conversion scripting and tests to ci_hsc

          2 Jim Bosch Done
           
          DM-14563

          Fix ci_hsc repo creation

          0.5 Jim Bosch Done
           
          DM-13358

          Add support for composites to Datastore prototype

          10 Tim Jenness Done
           
          DM-13350

          Move DatasetType template to Datastore configuration (override)

          6 Tim Jenness Done
           
          DM-13349

          Make StorageClass configurable in Butler Datastore

          4 Tim Jenness Done
           
          DM-13342

          Refactor Datastore prototype to improve orthogonality

          0.5 Pim Schellart [X] (Inactive) Done
           
          DM-14658

          Fix Gen3 SkyMap registration after polygon API changes

          0.5 Jim Bosch Done
           
          DM-13600

          Add YAML formatter

          1 Tim Jenness Done
           
          DM-13599

          Update copyright info following RFC-45

          0.5 Tim Jenness Done
           
          DM-13753

          Enable sphinx doc building in daf_butler

          2 Tim Jenness Done
           
          DM-13402

          Deal with comments from review of LDM-592 from butler gen3 review

          6 Tim Jenness Done
           
          DM-13374

          Deconstruct Butler prototype for redesign

          2 Pim Schellart [X] (Inactive) Done
           
          DM-13371

          Enable flake8 testing in daf_butler

          0.5 Tim Jenness Done
           
          DM-13507

          Add stable hash to SkyMap objects

          2 Jim Bosch Done
           
          DM-13217

          Deal with LDM-556 butler review comments

          6 Tim Jenness Done
           
          DM-13370

          Rebrand butler_prototype as daf_butler

          0.5 Tim Jenness Done
           
          DM-13364

          Attend Butler/SuperTask Meeting at Princeton

          8 Tim Jenness Done
           
          DM-13814

          Reorganize composites handling in datastore/butler

          10 Tim Jenness Done
           
          DM-14408

          Implement efficient Exposure component access and slicing

          2 Jim Bosch Invalid
           
          DM-14407

          Finish implementing concrete composite component-writing

          4 Jim Bosch Invalid
           
          DM-14666

          Store Python types with PosixDatastore and remove redundant StorageClasses

          2 Unassigned Invalid

            Activity

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Fritz Mueller, Jim Bosch, John Swinbank, Kian-Tat Lim, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Summary Panel