Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14448

Butler and SuperTask Conversion during F18

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Epic Name:
      f18-drp-supertask
    • Story Points:
      200
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Fall 2018

      Description

      DRP team contribution to the cross-subsystem "third generation butler" project. This project will be managed during F18 by Fritz Mueller

      This epic will include:

      • continuing work on the Gen 3 butler and converting the stack to use the new butler
      • work on SuperTask and converting existing Tasks to use the new SuperTask framework

        Attachments

          Container Issues

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          DM-15770

          Learn how to run pipeline tasks from command line

          4 Nate Lust Done
           
          DM-15767

          Remove pex_policy and Persistable usage from daf_persistence, obs_*, and Filter

          6 Jim Bosch Done
           
          DM-15663

          Start pipeline conversion process for DetectCoaddSources

          10 Nate Lust Done
           
          DM-15537

          Rename Sensor to Detector in Gen3 schema

          2 Jim Bosch Done
           
          DM-15528

          Make cameraGeom::Camera round-trip persistable

          6 Jim Bosch Done
           
          DM-15500

          Add FITS image, catalog readers that infer types from file

          2 Jim Bosch Done
           
          DM-15459

          Make dependencies optional in sets-of-DataUnits lookups

          2 Jim Bosch Done
           
          DM-15424

          Revisit LimitedRegistry concept

          3 Jim Bosch Done
           
          DM-16436

          Fix broken inheritance in WriteObjectTableTask

          2 Nate Lust Done
           
          DM-16413

          Remove more paf files from obs_base

          0.5 Jim Bosch Done
           
          DM-16159

          Revert DM-15686 to un-break obs_subaru master

          0.5 Jim Bosch Done
           
          DM-16026

          Split apart inheritance relations of MergeDetectionsTask and MergeMeasurementsTask

          4 Nate Lust Done
           
          DM-15915

          Spend time becoming familiar interacting with butler through sql

          4 Nate Lust Done
           
          DM-15913

          Make daf_butler put handle all identifier types

          4 Nate Lust Done
           
          DM-15897

          Pipelinetask init should take kwargs

          1 Nate Lust Done
           
          DM-15836

          Add helper code for invoking C++ templates from Python dtype arguments

          1 Jim Bosch Done
           
          DM-15871

          Move daf_butler's doImport function to utils

          0.5 Jim Bosch Done
           
          DM-15098

          Add Registry.getRegion(DataId)

          8 Pim Schellart [X] (Inactive) Done
           
          DM-15085

          Fix gen3-middleware ci_hsc SConscript

          1 Jim Bosch Done
           
          DM-15082

          Switch to YamlStorage instead of BoostStorage in all obs packages

          2 Jim Bosch Done
           
          DM-15189

          Write initial raw-data ingest system for Gen3 Butler

          6 Jim Bosch Done
           
          DM-14811

          Update Butler makeView to use DDLEvents

          0.5 Pim Schellart [X] (Inactive) Done
           
          DM-15365

          Move SqlRegistry API back down into Registry

          1 Pim Schellart [X] (Inactive) Done
           
          DM-15364

          Use double quotes wherever possible in daf_butler

          0.5 Pim Schellart [X] (Inactive) Done
           
          DM-15336

          Decouple DataUnitRegistry from Schema

          10 Nate Lust Done
           
          DM-15311

          Refactor MeasureCrosstalkTask per RFC-352

          2 Christopher Waters Done
           
          DM-15310

          Refactor ForcedPhotImageTask (and children) per RFC-352

          2 Christopher Waters Done
           
          DM-15350

          Fix segfault in sphgeom::HtmPixelization::pixel

          1 Pim Schellart [X] (Inactive) Done
           
          DM-15276

          Unify DataUnitRegion and DataUnitJoin

          2 Pim Schellart [X] (Inactive) Done
           
          DM-15268

          Merge gen3-middleware branches to master

          1 Jim Bosch Done
           
          DM-15256

          Include calibration repositories in Gen2->Gen3 conversion

          12 Christopher Waters Done
           
          DM-15217

          Rename sha1 to general hash in Butler schema and API

          0.5 Pim Schellart [X] (Inactive) Done
           
          DM-15210

          Update registry schema (and documentation thereof) following review

          2 Jim Bosch Done
           
          DM-15207

          Useful __str__ for Butler

          3 Christopher Waters Done
           
          DM-15212

          Improved transactions and ingest-with-transfer for datastores

          2 Jim Bosch Done
           
          DM-14822

          Gen3 get/put with DatasetRef only

          2 Pim Schellart [X] (Inactive) Done
           
          DM-14824

          Add syntactic sugar for ConfigFields of *DatasetConfigs

          2 Nate Lust Done
           
          DM-14771

          Add support for views to Schema

          2 Pim Schellart [X] (Inactive) Done
           
          DM-13690

          Write up Gen3 Butler / obs_* package interface sketch

          2 Jim Bosch Done
           
          DM-12619

          Define Butler DataUnits for Calibration Products

          2 Jim Bosch Done
           
          DM-2639

          Standardize primary method names, run/runDataRef, across PipeTasks

          12 Christopher Waters Done
           
          DM-14980

          Reinstate the ability of a Detector to find the Camera in which it lives

          6 Jim Bosch Done
           
          DM-15679

          Rename DataUnit to DataUnitDimension

          1 Jim Bosch Won't Fix
           
          DM-14831

          Update Butler makeView to trigger view creation only on related tables

          2 Unassigned Won't Fix
           
          DM-14003

          Create minimal ShimButler for v2 on top of v3

          12 Unassigned Won't Fix
           
          DM-13992

          Modify concrete CmdLineTasks and Gen3 Butler as necessary to work together

          6 Jim Bosch Won't Fix
           
          DM-12649

          Write Test Cases for makeDataGraph and DataUnitMap

          4 Jim Bosch Won't Fix
           
          DM-12642

          Implement Butler Registry initialization from YAML file

          2 Unassigned Won't Fix
           
          DM-12634

          Write SQL-YAML Converters for Butler Registry Schema

          8 Unassigned Won't Fix
           
          DM-15918

          Make Camera round-trip persistable via afw::table::io

          4 Unassigned Invalid
           
          DM-14504

          Remove daf_persistence hooks in PropertySet, PropertyList, and afw objects

          6 Jim Bosch Invalid
           
          DM-12648

          Implement DataUnit Inserts in Registries

          8 Unassigned Invalid
           
          DM-12643

          Add unit tests for composite Datasets to initial Butler Registry

          2 Unassigned Invalid
           
          DM-12631

          Implement Butler QuantumGraph

          4 Unassigned Invalid
           
          DM-12621

          RFC and adjust metadata schema for Butler

          2 Unassigned Invalid

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                yusra Yusra AlSayyad
                Watchers:
                John Swinbank, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel