Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14448

Butler and SuperTask Conversion during F18

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Epic Name:
      f18-drp-supertask
    • Story Points:
      200
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Fall 2018

      Description

      DRP team contribution to the cross-subsystem "third generation butler" project. This project will be managed during F18 by Fritz Mueller

      This epic will include:

      • continuing work on the Gen 3 butler and converting the stack to use the new butler
      • work on SuperTask and converting existing Tasks to use the new SuperTask framework

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        DM-16436

        Fix broken inheritance in WriteObjectTableTask

        2 Nate Lust Done
         
        DM-16413

        Remove more paf files from obs_base

        0.5 Jim Bosch Done
         
        DM-15915

        Spend time becoming familiar interacting with butler through sql

        4 Nate Lust Done
         
        DM-15913

        Make daf_butler put handle all identifier types

        4 Nate Lust Done
         
        DM-15770

        Learn how to run pipeline tasks from command line

        4 Nate Lust Done
         
        DM-15767

        Remove pex_policy and Persistable usage from daf_persistence, obs_*, and Filter

        6 Jim Bosch Done
         
        DM-15528

        Make cameraGeom::Camera round-trip persistable

        6 Jim Bosch Done
         
        DM-15500

        Add FITS image, catalog readers that infer types from file

        2 Jim Bosch Done
         
        DM-16026

        Split apart inheritance relations of MergeDetectionsTask and MergeMeasurementsTask

        4 Nate Lust Done
         
        DM-15871

        Move daf_butler's doImport function to utils

        0.5 Jim Bosch Done
         
        DM-15836

        Add helper code for invoking C++ templates from Python dtype arguments

        1 Jim Bosch Done
         
        DM-15537

        Rename Sensor to Detector in Gen3 schema

        2 Jim Bosch Done
         
        DM-15663

        Start pipeline conversion process for DetectCoaddSources

        10 Nate Lust Done
         
        DM-16159

        Revert DM-15686 to un-break obs_subaru master

        0.5 Jim Bosch Done
         
        DM-15897

        Pipelinetask init should take kwargs

        1 Nate Lust Done
         
        DM-15459

        Make dependencies optional in sets-of-DataUnits lookups

        2 Jim Bosch Done
         
        DM-15336

        Decouple DataUnitRegistry from Schema

        10 Nate Lust Done
         
        DM-15311

        Refactor MeasureCrosstalkTask per RFC-352

        2 Christopher Waters Done
         
        DM-15350

        Fix segfault in sphgeom::HtmPixelization::pixel

        1 Pim Schellart [X] (Inactive) Done
         
        DM-15310

        Refactor ForcedPhotImageTask (and children) per RFC-352

        2 Christopher Waters Done
         
        DM-15085

        Fix gen3-middleware ci_hsc SConscript

        1 Jim Bosch Done
         
        DM-15082

        Switch to YamlStorage instead of BoostStorage in all obs packages

        2 Jim Bosch Done
         
        DM-15098

        Add Registry.getRegion(DataId)

        8 Pim Schellart [X] (Inactive) Done
         
        DM-14824

        Add syntactic sugar for ConfigFields of *DatasetConfigs

        2 Nate Lust Done
         
        DM-14822

        Gen3 get/put with DatasetRef only

        2 Pim Schellart [X] (Inactive) Done
         
        DM-14811

        Update Butler makeView to use DDLEvents

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-14771

        Add support for views to Schema

        2 Pim Schellart [X] (Inactive) Done
         
        DM-12619

        Define Butler DataUnits for Calibration Products

        2 Jim Bosch Done
         
        DM-15424

        Revisit LimitedRegistry concept

        3 Jim Bosch Done
         
        DM-15217

        Rename sha1 to general hash in Butler schema and API

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-15212

        Improved transactions and ingest-with-transfer for datastores

        2 Jim Bosch Done
         
        DM-15210

        Update registry schema (and documentation thereof) following review

        2 Jim Bosch Done
         
        DM-15207

        Useful __str__ for Butler

        3 Christopher Waters Done
         
        DM-15189

        Write initial raw-data ingest system for Gen3 Butler

        6 Jim Bosch Done
         
        DM-15365

        Move SqlRegistry API back down into Registry

        1 Pim Schellart [X] (Inactive) Done
         
        DM-15364

        Use double quotes wherever possible in daf_butler

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-15268

        Merge gen3-middleware branches to master

        1 Jim Bosch Done
         
        DM-15256

        Include calibration repositories in Gen2->Gen3 conversion

        12 Christopher Waters Done
         
        DM-15276

        Unify DataUnitRegion and DataUnitJoin

        2 Pim Schellart [X] (Inactive) Done
         
        DM-14980

        Reinstate the ability of a Detector to find the Camera in which it lives

        6 Jim Bosch Done
         
        DM-13690

        Write up Gen3 Butler / obs_* package interface sketch

        2 Jim Bosch Done
         
        DM-2639

        Standardize primary method names, run/runDataRef, across PipeTasks

        12 Christopher Waters Done
         
        DM-15679

        Rename DataUnit to DataUnitDimension

        1 Jim Bosch Won't Fix
         
        DM-12649

        Write Test Cases for makeDataGraph and DataUnitMap

        4 Jim Bosch Won't Fix
         
        DM-12642

        Implement Butler Registry initialization from YAML file

        2 Unassigned Won't Fix
         
        DM-12634

        Write SQL-YAML Converters for Butler Registry Schema

        8 Unassigned Won't Fix
         
        DM-13992

        Modify concrete CmdLineTasks and Gen3 Butler as necessary to work together

        6 Jim Bosch Won't Fix
         
        DM-14003

        Create minimal ShimButler for v2 on top of v3

        12 Unassigned Won't Fix
         
        DM-14831

        Update Butler makeView to trigger view creation only on related tables

        2 Unassigned Won't Fix
         
        DM-15918

        Make Camera round-trip persistable via afw::table::io

        4 Unassigned Invalid
         
        DM-12648

        Implement DataUnit Inserts in Registries

        8 Unassigned Invalid
         
        DM-12643

        Add unit tests for composite Datasets to initial Butler Registry

        2 Unassigned Invalid
         
        DM-12631

        Implement Butler QuantumGraph

        4 Unassigned Invalid
         
        DM-14504

        Remove daf_persistence hooks in PropertySet, PropertyList, and afw objects

        6 Jim Bosch Invalid
         
        DM-12621

        RFC and adjust metadata schema for Butler

        2 Unassigned Invalid

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              yusra Yusra AlSayyad
              Watchers:
              John Swinbank, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel