Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14448

Butler and SuperTask Conversion during F18

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      f18-drp-supertask
    • Story Points:
      200
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Fall 2018

      Description

      DRP team contribution to the cross-subsystem "third generation butler" project. This project will be managed during F18 by Fritz Mueller

      This epic will include:

      • continuing work on the Gen 3 butler and converting the stack to use the new butler
      • work on SuperTask and converting existing Tasks to use the new SuperTask framework

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        		DM-16159

        Revert DM-15686 to un-break obs_subaru master

        		 0.5 Jim Bosch Done
         
        		DM-16436

        Fix broken inheritance in WriteObjectTableTask

        		 2 Nate Lust Done
         
        		DM-16413

        Remove more paf files from obs_base

        		 0.5 Jim Bosch Done
         
        		DM-16026

        Split apart inheritance relations of MergeDetectionsTask and MergeMeasurementsTask

        		 4 Nate Lust Done
         
        		DM-15915

        Spend time becoming familiar interacting with butler through sql

        		 4 Nate Lust Done
         
        		DM-15913

        Make daf_butler put handle all identifier types

        		 4 Nate Lust Done
         
        		DM-15871

        Move daf_butler's doImport function to utils

        		 0.5 Jim Bosch Done
         
        		DM-15836

        Add helper code for invoking C++ templates from Python dtype arguments

        		 1 Jim Bosch Done
         
        		DM-15897

        Pipelinetask init should take kwargs

        		 1 Nate Lust Done
         
        		DM-13690

        Write up Gen3 Butler / obs_* package interface sketch

        		 2 Jim Bosch Done
         
        		DM-15663

        Start pipeline conversion process for DetectCoaddSources

        		 10 Nate Lust Done
         
        		DM-15537

        Rename Sensor to Detector in Gen3 schema

        		 2 Jim Bosch Done
         
        		DM-15528

        Make cameraGeom::Camera round-trip persistable

        		 6 Jim Bosch Done
         
        		DM-15500

        Add FITS image, catalog readers that infer types from file

        		 2 Jim Bosch Done
         
        		DM-15770

        Learn how to run pipeline tasks from command line

        		 4 Nate Lust Done
         
        		DM-15767

        Remove pex_policy and Persistable usage from daf_persistence, obs_*, and Filter

        		 6 Jim Bosch Done
         
        		DM-14980

        Reinstate the ability of a Detector to find the Camera in which it lives

        		 6 Jim Bosch Done
         
        		DM-15098

        Add Registry.getRegion(DataId)

        		 8 Pim Schellart [X] (Inactive) Done
         
        		DM-15085

        Fix gen3-middleware ci_hsc SConscript

        		 1 Jim Bosch Done
         
        		DM-15082

        Switch to YamlStorage instead of BoostStorage in all obs packages

        		 2 Jim Bosch Done
         
        		DM-12619

        Define Butler DataUnits for Calibration Products

        		 2 Jim Bosch Done
         
        		DM-14811

        Update Butler makeView to use DDLEvents

        		 0.5 Pim Schellart [X] (Inactive) Done
         
        		DM-14771

        Add support for views to Schema

        		 2 Pim Schellart [X] (Inactive) Done
         
        		DM-14824

        Add syntactic sugar for ConfigFields of *DatasetConfigs

        		 2 Nate Lust Done
         
        		DM-14822

        Gen3 get/put with DatasetRef only

        		 2 Pim Schellart [X] (Inactive) Done
         
        		DM-15311

        Refactor MeasureCrosstalkTask per RFC-352

        		 2 Christopher Waters Done
         
        		DM-15310

        Refactor ForcedPhotImageTask (and children) per RFC-352

        		 2 Christopher Waters Done
         
        		DM-15276

        Unify DataUnitRegion and DataUnitJoin

        		 2 Pim Schellart [X] (Inactive) Done
         
        		DM-15268

        Merge gen3-middleware branches to master

        		 1 Jim Bosch Done
         
        		DM-15256

        Include calibration repositories in Gen2->Gen3 conversion

        		 12 Christopher Waters Done
         
        		DM-15217

        Rename sha1 to general hash in Butler schema and API

        		 0.5 Pim Schellart [X] (Inactive) Done
         
        		DM-15212

        Improved transactions and ingest-with-transfer for datastores

        		 2 Jim Bosch Done
         
        		DM-15210

        Update registry schema (and documentation thereof) following review

        		 2 Jim Bosch Done
         
        		DM-15207

        Useful __str__ for Butler

        		 3 Christopher Waters Done
         
        		DM-15189

        Write initial raw-data ingest system for Gen3 Butler

        		 6 Jim Bosch Done
         
        		DM-15459

        Make dependencies optional in sets-of-DataUnits lookups

        		 2 Jim Bosch Done
         
        		DM-15424

        Revisit LimitedRegistry concept

        		 3 Jim Bosch Done
         
        		DM-15365

        Move SqlRegistry API back down into Registry

        		 1 Pim Schellart [X] (Inactive) Done
         
        		DM-15364

        Use double quotes wherever possible in daf_butler

        		 0.5 Pim Schellart [X] (Inactive) Done
         
        		DM-15350

        Fix segfault in sphgeom::HtmPixelization::pixel

        		 1 Pim Schellart [X] (Inactive) Done
         
        		DM-15336

        Decouple DataUnitRegistry from Schema

        		 10 Nate Lust Done
         
        		DM-2639

        Standardize primary method names, run/runDataRef, across PipeTasks

        		 12 Christopher Waters Done
         
        		DM-15679

        Rename DataUnit to DataUnitDimension

        		 1 Jim Bosch Won't Fix
         
        		DM-12649

        Write Test Cases for makeDataGraph and DataUnitMap

        		 4 Jim Bosch Won't Fix
         
        		DM-12642

        Implement Butler Registry initialization from YAML file

        		 2 Unassigned Won't Fix
         
        		DM-12634

        Write SQL-YAML Converters for Butler Registry Schema

        		 8 Unassigned Won't Fix
         
        		DM-14831

        Update Butler makeView to trigger view creation only on related tables

        		 2 Unassigned Won't Fix
         
        		DM-14003

        Create minimal ShimButler for v2 on top of v3

        		 12 Unassigned Won't Fix
         
        		DM-13992

        Modify concrete CmdLineTasks and Gen3 Butler as necessary to work together

        		 6 Jim Bosch Won't Fix
         
        		DM-15918

        Make Camera round-trip persistable via afw::table::io

        		 4 Unassigned Invalid
         
        		DM-12648

        Implement DataUnit Inserts in Registries

        		 8 Unassigned Invalid
         
        		DM-12643

        Add unit tests for composite Datasets to initial Butler Registry

        		 2 Unassigned Invalid
         
        		DM-12631

        Implement Butler QuantumGraph

        		 4 Unassigned Invalid
         
        		DM-12621

        RFC and adjust metadata schema for Butler

        		 2 Unassigned Invalid
         
        		DM-14504

        Remove daf_persistence hooks in PropertySet, PropertyList, and afw objects

        		 6 Jim Bosch Invalid

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            yusra Yusra AlSayyad
            Watchers:
            John Swinbank, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.