Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12738

Butler development during S18

    XMLWordPrintable

Details

    • Epic
    • Status: Done
    • Resolution: Done
    • None
    • None
    • s18-drp-butler
    • 140
    • 02C.04.01
    • Data Release Production
    • Spring 2018

    Description

      This represents the DRP team contribution to the cross-subsystem "third generation butler" project. This project will be managed during S18 by fritzm.

      Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        DM-13787

        afw::table::SchemaMapper::addMapping calls wrong overload with string literal

        0.1 Pim Schellart [X] (Inactive) Done
         
        DM-13700

        Discuss composites

        1 Pim Schellart [X] (Inactive) Done
         
        DM-13689

        Write up design proposal for Butler composites

        4 Jim Bosch Done
         
        DM-13688

        Write up design proposal for Registry/Datastore boundary

        2 Jim Bosch Done
         
        DM-13627

        Discuss Registry / Datastore boundary

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-12640

        Implement Initial Butler Registry Hierarchy

        6 Pim Schellart [X] (Inactive) Done
         
        DM-12635

        Write initial Gen2->Gen3 conversion script

        10 Jim Bosch Done
         
        DM-12627

        Implement Butler DatasetType

        4 Pim Schellart [X] (Inactive) Done
         
        DM-12624

        Implement Butler DataUnits

        Pim Schellart [X] (Inactive) Done
         
        DM-12620

        Write Initial Butler Metadata Schema Proposal

        12 Jim Bosch Done
         
        DM-12618

        Define Storage Classes for Butler

        2 Jim Bosch Done
         
        DM-14639

        Make Butler optionally transactional

        2 Pim Schellart [X] (Inactive) Done
         
        DM-14638

        Make Datastore optionally transactional

        3 Pim Schellart [X] (Inactive) Done
         
        DM-14520

        Re-enable Dataset->DataUnit foreign keys

        1 Jim Bosch Done
         
        DM-14545

        Add test for composite calexp to Butler

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-14544

        Add pickle support to Butler

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-14543

        Fix DatasetType registration

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-14012

        Add helper container class supporting topologically sorted iteration over elements

        1 Pim Schellart [X] (Inactive) Done
         
        DM-13963

        Enable foreign_key integrity checking on Butler SqlRegistry

        1 Pim Schellart [X] (Inactive) Done
         
        DM-13895

        Implement Butler provenance primitives

        4 Pim Schellart [X] (Inactive) Done
         
        DM-14225

        Make PosixDatastore's internal records persistent

        4 Jim Bosch Done
         
        DM-14467

        Make SqlRegistry more transactional

        6 Pim Schellart [X] (Inactive) Done
         
        DM-14321

        Make StorageClass pickleable (by making instances instead of subclasses)

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-14139

        Implement Registry.ensureRun on Gen3 Butler

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-14114

        Add interface for camera specialization to Gen3 Butler

        2 Pim Schellart [X] (Inactive) Done
         
        DM-14377

        calexp_camera is an ImageF

        0.1 Pim Schellart [X] (Inactive) Done
         
        DM-13496

        Prototype Butler Registry schema definition in YAML

        4 Pim Schellart [X] (Inactive) Done
         
        DM-13399

        Use cls for first argument to Python metaclass methods

        0.5 Pim Schellart [X] (Inactive) Done
         
        DM-13378

        Attend Gen3 Middleware Design/Hack Week

        8 Jim Bosch Done
         
        DM-13372

        Attend Butler/SuperTask Meeting at Princeton

        8 Pim Schellart [X] (Inactive) Done
         
        DM-13246

        Synchronize DMTN-056 with Butler prototype

        1.5 Pim Schellart [X] (Inactive) Done
         
        DM-12921

        Implement makePath in butler prototype

        1 Pim Schellart [X] (Inactive) Done
         
        DM-12920

        Prototype Butler configuration

        6 Pim Schellart [X] (Inactive) Done
         
        DM-13869

        Implement basic Registry.find and related functionality

        3 Pim Schellart [X] (Inactive) Done
         
        DM-13842

        Implement access to DatasetStorage table

        2 Pim Schellart [X] (Inactive) Done
         
        DM-13840

        Prepare Butler for composite work to begin

        8 Pim Schellart [X] (Inactive) Done
         
        DM-13197

        Explore design issues with DatasetRef hierarchy and PreFlight (J)

        4 Jim Bosch Done
         
        DM-13196

        Explore design issues with DatasetRef hierarchy and PreFlight (P)

        4 Pim Schellart [X] (Inactive) Done
         
        DM-12633

        Implement Direct SQL Butler Schema Creation

        2 Pim Schellart [X] (Inactive) Won't Fix
         
        DM-12632

        Implement Butler DataUnit Packing or Hashing

        10 Unassigned Won't Fix
         
        DM-12625

        Investigate machine generating/verifying Butler DataUnits

        4 Unassigned Won't Fix
         
        DM-12613

        Implement minimal butler Registry prototype with sqlalchemy orm

        6 Pim Schellart [X] (Inactive) Won't Fix
         
        DM-13287

        Redesign DataUnit class (and children) in Butler prototype

        4 Pim Schellart [X] (Inactive) Won't Fix
         
        DM-13171

        Convert butler Registry test data to use sqlalchemy

        1 Pim Schellart [X] (Inactive) Won't Fix
         
        DM-13063

        Find solution for ids in butler prototype

        2 Pim Schellart [X] (Inactive) Won't Fix
         
        DM-13820

        Factor out DataUnit description from Schema object

        2 Pim Schellart [X] (Inactive) Invalid
         
        DM-12647

        Implement Basic Collection Management in Registries

        2 Pim Schellart [X] (Inactive) Invalid
         
        DM-12646

        Implement Provenance Methods in Registries

        6 Pim Schellart [X] (Inactive) Invalid
         
        DM-12645

        Implement addDataset in Registries

        6 Pim Schellart [X] (Inactive) Invalid
         
        DM-12644

        Implement registerDatasetType in Registries

        2 Pim Schellart [X] (Inactive) Invalid
         
        DM-12630

        Implement Butler Quantum

        2 Pim Schellart [X] (Inactive) Invalid
         
        DM-12629

        Implement Butler Run

        2 Pim Schellart [X] (Inactive) Invalid
         
        DM-12628

        Implement Butler DatasetRef

        6 Pim Schellart [X] (Inactive) Invalid
         
        DM-12626

        Implement Butler DataUnitSet

        4 Pim Schellart [X] (Inactive) Invalid
         
        DM-13991

        Add Gen3 Butler support to CmdLineTask

        4 Jim Bosch Invalid
         
        DM-13033

        Implement support for composite datasets in butler Registry

        6 Pim Schellart [X] (Inactive) Invalid

        Activity

          No builds found.
          swinbank John Swinbank created issue -
          swinbank John Swinbank made changes -
          Field Original Value New Value
          Epic Child DM-12618 [ 36151 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12649 [ 36199 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12619 [ 36152 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12644 [ 36193 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12631 [ 36176 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12617 [ 36150 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12640 [ 36187 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12646 [ 36196 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12627 [ 36171 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12642 [ 36190 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12620 [ 36153 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12621 [ 36165 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12628 [ 36172 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12635 [ 36180 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12629 [ 36173 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12634 [ 36179 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12645 [ 36194 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12633 [ 36178 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12626 [ 36170 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12622 [ 36166 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12651 [ 36203 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12630 [ 36175 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12648 [ 36198 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12647 [ 36197 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12624 [ 36168 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12625 [ 36169 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12623 [ 36167 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12643 [ 36192 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12632 [ 36177 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12650 [ 36201 ]
          yusra Yusra AlSayyad made changes -
          Cycle Spring 2018 [ 10806 ]
          swinbank John Swinbank made changes -
          Labels gen3-middleware
          yusra Yusra AlSayyad made changes -
          Watchers John Swinbank [ John Swinbank ] John Swinbank, Yusra AlSayyad [ John Swinbank, Yusra AlSayyad ]
          yusra Yusra AlSayyad made changes -
          Watchers John Swinbank, Yusra AlSayyad [ John Swinbank, Yusra AlSayyad ] John Swinbank, Pim Schellart, Yusra AlSayyad [ John Swinbank, Pim Schellart, Yusra AlSayyad ]
          swinbank John Swinbank made changes -
          Story Points 130 140
          yusra Yusra AlSayyad made changes -
          Epic Child DM-12613 [ 36141 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12921 [ 36703 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12920 [ 36701 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-12617 [ 36150 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-12622 [ 36166 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-12623 [ 36167 ]
          swinbank John Swinbank made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13378 [ 38310 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13197 [ 37737 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13196 [ 37736 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13171 [ 37509 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13372 [ 38301 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13246 [ 38000 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-13627 [ 38963 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-13688 [ 39073 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-13689 [ 39074 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-13690 [ 39075 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-13700 [ 39107 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-13496 [ 38652 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-13063 [ 37005 ]
          pschella Pim Schellart [X] (Inactive) made changes -
          Epic Child DM-13033 [ 36945 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-13990 [ 39832 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-13991 [ 39833 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-13992 [ 39834 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-14225 [ 53179 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-14520 [ 86264 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14543 [ 91403 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14545 [ 91410 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13895 [ 39609 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14591 [ 101884 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13840 [ 39450 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14003 [ 39848 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14377 [ 72398 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13787 [ 39338 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14139 [ 42207 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13963 [ 39750 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13287 [ 38146 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14638 [ 106100 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13842 [ 39462 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14467 [ 81550 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14012 [ 39869 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14639 [ 106105 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14321 [ 65094 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13820 [ 39400 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14114 [ 40157 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14544 [ 91406 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13869 [ 39515 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13399 [ 38365 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13992 [ 39834 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12621 [ 36165 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12651 [ 36203 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12650 [ 36201 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13990 [ 39832 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12649 [ 36199 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12642 [ 36190 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12648 [ 36198 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12619 [ 36152 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12643 [ 36192 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12634 [ 36179 ]
          swinbank John Swinbank made changes -
          Epic Child DM-12631 [ 36176 ]
          swinbank John Swinbank made changes -
          Epic Child DM-14591 [ 101884 ]
          swinbank John Swinbank made changes -
          Epic Child DM-14003 [ 39848 ]
          swinbank John Swinbank made changes -
          Epic Child DM-13690 [ 39075 ]
          swinbank John Swinbank made changes -
          Resolution Done [ 10000 ]
          Status In Progress [ 3 ] Done [ 10002 ]
          swinbank John Swinbank made changes -
          Risk Score 0
          jbosch Jim Bosch made changes -
          Epic Child DM-15082 [ 147600 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-15085 [ 148261 ]
          swinbank John Swinbank made changes -
          Epic Child DM-15082 [ 147600 ]
          swinbank John Swinbank made changes -
          Epic Child DM-15085 [ 148261 ]

          People

            Unassigned Unassigned
            swinbank John Swinbank
            John Swinbank, Pim Schellart [X] (Inactive), Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.