Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14448

Butler and SuperTask Conversion during F18

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      f18-drp-supertask
    • Story Points:
      200
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Fall 2018

      Description

      DRP team contribution to the cross-subsystem "third generation butler" project. This project will be managed during F18 by Fritz Mueller

      This epic will include:

      • continuing work on the Gen 3 butler and converting the stack to use the new butler
      • work on SuperTask and converting existing Tasks to use the new SuperTask framework

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        DM-16436

        Fix broken inheritance in WriteObjectTableTask

        2 Nate Lust Done
         
        DM-16413

        Remove more paf files from obs_base

        0.5 Jim Bosch Done
         
        DM-16159

        Revert DM-15686 to un-break obs_subaru master

        0.5 Jim Bosch Done
         
        DM-15897

        Pipelinetask init should take kwargs

        1 Nate Lust Done
         
        DM-15871

        Move daf_butler's doImport function to utils

        0.5 Jim Bosch Done
         
        DM-16026

        Split apart inheritance relations of MergeDetectionsTask and MergeMeasurementsTask

        4 Nate Lust Done
         
        DM-15915

        Spend time becoming familiar interacting with butler through sql

        4 Nate Lust Done
         
        DM-15913

        Make daf_butler put handle all identifier types

        4 Nate Lust Done
         
        DM-15836

        Add helper code for invoking C++ templates from Python dtype arguments

        1 Jim Bosch Done
         
        DM-13690

        Write up Gen3 Butler / obs_* package interface sketch

        2 Jim Bosch Done
         
        DM-15537

        Rename Sensor to Detector in Gen3 schema

        2 Jim Bosch Done
         
        DM-15528

        Make cameraGeom::Camera round-trip persistable

        6 Jim Bosch Done
         
        DM-15500

        Add FITS image, catalog readers that infer types from file

        2 Jim Bosch Done
         
        DM-15424

        Revisit LimitedRegistry concept

        3 Jim Bosch Done
         
        DM-12619

        Define Butler DataUnits for Calibration Products

        2 Jim Bosch Done
         
        DM-15770

        Learn how to run pipeline tasks from command line

        4 Nate Lust Done
         
        DM-15767

        Remove pex_policy and Persistable usage from daf_persistence, obs_*, and Filter

        6 Jim Bosch Done
         
        DM-15350

        Fix segfault in sphgeom::HtmPixelization::pixel

        1 Pim Schellart [X] (Inactive) Done
         
        DM-15336

        Decouple DataUnitRegistry from Schema

        10 Nate Lust Done
         
        DM-15311

        Refactor MeasureCrosstalkTask per RFC-352

        2 Christopher Waters Done
         
        DM-15310

        Refactor ForcedPhotImageTask (and children) per RFC-352

        2 Christopher Waters Done
         
        DM-15098

        Add Registry.getRegion(DataId)

        8 Pim Schellart [X] (Inactive) Done
         
        DM-15085

        Fix gen3-middleware ci_hsc SConscript

        1 Jim Bosch Done
         
        DM-15082

        Switch to YamlStorage instead of BoostStorage in all obs packages

        2 Jim Bosch Done
         
        DM-14771

        Add support for views to Schema

        2 Pim Schellart [X] (Inactive) Done
         
        DM-15212

        Improved transactions and ingest-with-transfer for datastores

        2 Jim Bosch Done
         
        DM-15210

        Update registry schema (and documentation thereof) following review

        2 Jim Bosch Done
         
        DM-15207

        Useful __str__ for Butler

        3 Christopher Waters Done
         
        DM-15189

        Write initial raw-data ingest system for Gen3 Butler

        6 Jim Bosch Done
         
        DM-14824

        Add syntactic sugar for ConfigFields of *DatasetConfigs

        2 Nate Lust Done
         
        DM-14822

        Gen3 get/put with DatasetRef only

        2 Pim Schellart [X] (Inactive) Done
         
        DM-14811

        Update Butler makeView to use DDLEvents

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-15459

        Make dependencies optional in sets-of-DataUnits lookups

        2 Jim Bosch Done
         
        DM-15663

        Start pipeline conversion process for DetectCoaddSources

        10 Nate Lust Done
         
        DM-15276

        Unify DataUnitRegion and DataUnitJoin

        2 Pim Schellart [X] (Inactive) Done
         
        DM-15268

        Merge gen3-middleware branches to master

        1 Jim Bosch Done
         
        DM-15256

        Include calibration repositories in Gen2->Gen3 conversion

        12 Christopher Waters Done
         
        DM-15365

        Move SqlRegistry API back down into Registry

        1 Pim Schellart [X] (Inactive) Done
         
        DM-15364

        Use double quotes wherever possible in daf_butler

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-15217

        Rename sha1 to general hash in Butler schema and API

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-14980

        Reinstate the ability of a Detector to find the Camera in which it lives

        6 Jim Bosch Done
         
        DM-2639

        Standardize primary method names, run/runDataRef, across PipeTasks

        12 Christopher Waters Done
         
        DM-14003

        Create minimal ShimButler for v2 on top of v3

        12 Unassigned Won't Fix
         
        DM-13992

        Modify concrete CmdLineTasks and Gen3 Butler as necessary to work together

        6 Jim Bosch Won't Fix
         
        DM-12649

        Write Test Cases for makeDataGraph and DataUnitMap

        4 Jim Bosch Won't Fix
         
        DM-12642

        Implement Butler Registry initialization from YAML file

        2 Unassigned Won't Fix
         
        DM-12634

        Write SQL-YAML Converters for Butler Registry Schema

        8 Unassigned Won't Fix
         
        DM-14831

        Update Butler makeView to trigger view creation only on related tables

        2 Unassigned Won't Fix
         
        DM-15679

        Rename DataUnit to DataUnitDimension

        1 Jim Bosch Won't Fix
         
        DM-15918

        Make Camera round-trip persistable via afw::table::io

        4 Unassigned Invalid
         
        DM-12648

        Implement DataUnit Inserts in Registries

        8 Unassigned Invalid
         
        DM-12643

        Add unit tests for composite Datasets to initial Butler Registry

        2 Unassigned Invalid
         
        DM-12631

        Implement Butler QuantumGraph

        4 Unassigned Invalid
         
        DM-12621

        RFC and adjust metadata schema for Butler

        2 Unassigned Invalid
         
        DM-14504

        Remove daf_persistence hooks in PropertySet, PropertyList, and afw objects

        6 Jim Bosch Invalid

          Activity

          No builds found.
          yusra Yusra AlSayyad created issue -
          yusra Yusra AlSayyad made changes -
          Field Original Value New Value
          Risk Score 0
          yusra Yusra AlSayyad made changes -
          Story Points 240 230
          yusra Yusra AlSayyad made changes -
          Story Points 230 220
          yusra Yusra AlSayyad made changes -
          Story Points 220 200
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14760 [ 119244 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14771 [ 120147 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-14811 [ 125918 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-14831 [ 126830 ]
          swinbank John Swinbank made changes -
          Epic Child DM-14822 [ 126111 ]
          swinbank John Swinbank made changes -
          Epic Child DM-14822 [ 126111 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13992 [ 39834 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12621 [ 36165 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12651 [ 36203 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12650 [ 36201 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13990 [ 39832 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12649 [ 36199 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12642 [ 36190 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12648 [ 36198 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12619 [ 36152 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12643 [ 36192 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12634 [ 36179 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12631 [ 36176 ]
          swinbank John Swinbank made changes -
          Epic Child DM-14591 [ 101884 ]
          swinbank John Swinbank made changes -
          Epic Child DM-14003 [ 39848 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13690 [ 39075 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14816 [ 126055 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14822 [ 126111 ]
          swinbank John Swinbank made changes -
          Epic Child DM-15082 [ 147600 ]
          swinbank John Swinbank made changes -
          Epic Child DM-15085 [ 148261 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14824 [ 126126 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15236 [ 158917 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15256 [ 161369 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15268 [ 161477 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14819 [ 126076 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-2639 [ 17392 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15212 [ 155824 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15310 [ 163424 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15311 [ 163432 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15336 [ 164333 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15411 [ 168517 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15412 [ 168522 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15424 [ 171585 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15459 [ 174993 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15500 [ 180184 ]
          tjenness Tim Jenness made changes -
          Epic Child DM-14760 [ 119244 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-14980 [ 136812 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15528 [ 182403 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15207 [ 155128 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15350 [ 165359 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15364 [ 166866 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15365 [ 166889 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15366 [ 166892 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15217 [ 155966 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15466 [ 178759 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15098 [ 148717 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15189 [ 152962 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15210 [ 155793 ]
          nlust Nate Lust made changes -
          Epic Child DM-15663 [ 195745 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15675 [ 195904 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15679 [ 195952 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15276 [ 161634 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15749 [ 201290 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15750 [ 201291 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-14504 [ 85512 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15767 [ 202885 ]
          nlust Nate Lust made changes -
          Epic Child DM-15770 [ 204102 ]
          swinbank John Swinbank made changes -
          Epic Child DM-15760 [ 202047 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15836 [ 210929 ]
          nlust Nate Lust made changes -
          Epic Child DM-15843 [ 211951 ]
          nlust Nate Lust made changes -
          Epic Child DM-15845 [ 212454 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15871 [ 213942 ]
          tjenness Tim Jenness made changes -
          Epic Child DM-15760 [ 202047 ]
          nlust Nate Lust made changes -
          Epic Child DM-15913 [ 218313 ]
          nlust Nate Lust made changes -
          Epic Child DM-15915 [ 218317 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15918 [ 218346 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15897 [ 215419 ]
          nlust Nate Lust made changes -
          Epic Child DM-16026 [ 221225 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-16056 [ 223676 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15537 [ 182652 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-16159 [ 225987 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16221 [ 229188 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-16297 [ 231570 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-16413 [ 234219 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16436 [ 234414 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-16539 [ 234780 ]
          nlust Nate Lust made changes -
          Epic Child DM-16797 [ 236159 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16797 [ 236159 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15843 [ 211951 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13990 [ 39832 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16221 [ 229188 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15749 [ 201290 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16056 [ 223676 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15845 [ 212454 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-12650 [ 36201 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15412 [ 168522 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15466 [ 178759 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15411 [ 168517 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16297 [ 231570 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-12651 [ 36203 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15366 [ 166892 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14591 [ 101884 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15675 [ 195904 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15236 [ 158917 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14819 [ 126076 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14816 [ 126055 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15750 [ 201291 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16539 [ 234780 ]
          yusra Yusra AlSayyad made changes -
          Resolution Done [ 10000 ]
          Status To Do [ 10001 ] Done [ 10002 ]

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            yusra Yusra AlSayyad
            Watchers:
            John Swinbank, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.