Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16771

DRP efforts toward Gen 3 Middleware during second half of S19

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      s19-drp-pipelinetask
    • Story Points:
      158
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Spring 2019

      Description

      This is a continuation of the Gen3 middleware and butler work being done by the DRP team on DM-16675. This epic includes work done after the push to get a DRP runnning with a Gen 3 butler. This work is scheduled for February 1st - May 31st. It includes:

      • Ongoing removal of the shims and broken-rules after Jan 31st.¬†
      • Ongoing conversion of additional tasks after Jan 31st.
      • Schema changes
      • Science Pipelines modification so to support new workflow system development
      • Get off Gen2 and pipe_drivers:
        • Repo converter
        • ¬†Implement a ctrl_pool activator
        • Gen3 views into Gen2 repos

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        DM-19871

        Fix validity range end in Gen3 calibration bootstrapping

        1 Jim Bosch Done
         
        DM-19531

        Add option to include regions when ingesting gen3 images

        4 Nate Lust Done
         
        DM-19272

        Make script for creating skymaps in gen3 bulter

        5 Nate Lust Done
         
        DM-19271

        Add transactions to gen3 skymap registry

        4 Nate Lust Done
         
        DM-18863

        Investigate and fix warnings in daf_butler

        2 Nate Lust Done
         
        DM-19731

        Investigate inaccuracy of initial HSC WCSs

        2 Jim Bosch Done
         
        DM-19818

        Fix Gen3 Butler pickling broken on DM-19638

        1 Jim Bosch Done
         
        DM-19808

        Avoid duplication in deferred prelight follow-up queries

        6 Jim Bosch Done
         
        DM-19387

        Vectorize dimension entry addition to gen3 registry

        4 Nate Lust Done
         
        DM-19971

        Fixup for tickets/DM-19272

        0.25 Nate Lust Done
         
        DM-19638

        Create parent task/script for bootstrapping Gen3 repos

        4 Jim Bosch Done
         
        DM-19623

        Change sqlalchemy syntax in addDimensionEntryList

        2 Nate Lust Done
         
        DM-19622

        Make PosixDatastore's internal table lowercase

        0.5 Jim Bosch Done
         
        DM-19615

        Change raw storage class in isr task

        1 Nate Lust Done
         
        DM-19614

        Write transmission curves in writeCuratedCalibrations

        3 Nate Lust Done
         
        DM-18839

        Remove explicit registry close in the butler

        8 Nate Lust Done
         
        DM-18838

        Fix butlerRoot in OracleRegistry

        1 Nate Lust Done
         
        DM-18531

        convert skyCorrection to gen3 pipelineTask

        6 Christopher Waters Done
         
        DM-18367

        Get butler working with Oracle

        14 Nate Lust Done
         
        DM-17912

        gen3 ip_isr can attempt to remove a non-existant dataset, causing KeyError

        1 Christopher Waters Done
         
        DM-17838

        Spend time post-demo reviewing and preparing next stages

        7 Nate Lust Done
         
        DM-17757

        Key error when trying to use pipetask to output pipeline/quantum graph and then read it.

        8 Nate Lust Done
         
        DM-17679

        Butler sqlite data issues

        2 Jim Bosch Done
         
        DM-17611

        Performance optimizations to data ID code

        1 Jim Bosch Done
         
        DM-17608

        Fix performance regression from SQLite transaction changes

        0.5 Jim Bosch Done
         
        DM-17663

        Make Registry table names lowercase

        1 Jim Bosch Done
         
        DM-17496

        QuantumGraph generation hits SQLite join limit

        6 Jim Bosch Done
         
        DM-17491

        Implement Butler deletion APIs

        2 Jim Bosch Done
         
        DM-16539

        Add level of indirection betwen calib identifier and exposure range in schema

        4 Jim Bosch Done
         
        DM-15843

        Clean up details of pipeline task conversion process

        4 Jim Bosch Done
         
        DM-14821

        Provide packed integer versions of Gen3 data IDs

        6 Jim Bosch Done
         
        DM-18027

        gen2convert robustification and enhancements

        15 Christopher Waters Invalid

          Activity

          yusra Yusra AlSayyad created issue -
          yusra Yusra AlSayyad made changes -
          Field Original Value New Value
          Cycle Spring 2019 [ 10808 ]
          Story Points 158
          Team Data Release Production [ 10301 ]
          WBS 02C.04.01
          Description This is a continuation of the Gen3 middleware and butler work being done by the DRP team on DM-16675. This epic includes work done after the push to get a DRP runnning with a Gen 3 butler. This work is scheduled for February 1st - May 31st. It includes:
           * Ongoing removal of the shims and broken-rules after Jan 31st. 
           * Ongoing conversion of additional tasks after Jan 31st.
           * Schema changes
           * Science Pipelines modification so to support new workflow system development
           * Get off Gen2 and pipe_drivers:
           ** Repo converter
           **  Implement a ctrl_pool activator
           ** Gen3 views into Gen2 repos
          Summary DRP efforts toward Gen 3 Middleware during S19 DRP efforts toward Gen 3 Middleware during second half of S19
          nlust Nate Lust made changes -
          Epic Child DM-17390 [ 239192 ]
          nlust Nate Lust made changes -
          Epic Child DM-17395 [ 239202 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17390 [ 239192 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17395 [ 239202 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15749 [ 201290 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16297 [ 231570 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15412 [ 168522 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16056 [ 223676 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17689 [ 239741 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16876 [ 238171 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17062 [ 238688 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15411 [ 168517 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17491 [ 239366 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17154 [ 238910 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-12650 [ 36201 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15750 [ 201291 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13990 [ 39832 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17494 [ 239369 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17498 [ 239376 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-12651 [ 36203 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17017 [ 238492 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14816 [ 126055 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16539 [ 234780 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15466 [ 178759 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17663 [ 239673 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16803 [ 236849 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17611 [ 239582 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17608 [ 239579 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17496 [ 239371 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15843 [ 211951 ]
          yusra Yusra AlSayyad made changes -
          Assignee Yusra AlSayyad [ yusra ]
          Status To Do [ 10001 ] In Progress [ 3 ]
          yusra Yusra AlSayyad made changes -
          Epic Name s19b-drp-pipelineTask s19-drp-pipelinetask
          swinbank John Swinbank made changes -
          Epic Child DM-17679 [ 239693 ]
          swinbank John Swinbank made changes -
          Epic Child DM-14821 [ 126106 ]
          frossie Frossie Economou made changes -
          Status Admin Review [ 3 ] In Progress [ 11605 ]
          frossie Frossie Economou made changes -
          Status Review [ 11605 ] In Progress [ 3 ]
          nlust Nate Lust made changes -
          Epic Child DM-17838 [ 240095 ]
          swinbank John Swinbank made changes -
          Epic Child DM-17912 [ 240361 ]
          swinbank John Swinbank made changes -
          Epic Child DM-17757 [ 239863 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-18027 [ 240610 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-18367 [ 242429 ]
          nlust Nate Lust made changes -
          Epic Child DM-18839 [ 251816 ]
          nlust Nate Lust made changes -
          Epic Child DM-18838 [ 251812 ]
          nlust Nate Lust made changes -
          Epic Child DM-18863 [ 252363 ]
          nlust Nate Lust made changes -
          Epic Child DM-19271 [ 259139 ]
          nlust Nate Lust made changes -
          Epic Child DM-19272 [ 259145 ]
          nlust Nate Lust made changes -
          Epic Child DM-19387 [ 261495 ]
          nlust Nate Lust made changes -
          Epic Child DM-19531 [ 268703 ]
          nlust Nate Lust made changes -
          Epic Child DM-19614 [ 274734 ]
          nlust Nate Lust made changes -
          Epic Child DM-19615 [ 274763 ]
          nlust Nate Lust made changes -
          Epic Child DM-19623 [ 275028 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19638 [ 276264 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19731 [ 286279 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19807 [ 293954 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19808 [ 293955 ]
          nlust Nate Lust made changes -
          Epic Child DM-19817 [ 294937 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19818 [ 294938 ]
          salnikov Andy Salnikov made changes -
          Epic Child DM-19851 [ 297340 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19871 [ 298554 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19622 [ 275027 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19942 [ 305955 ]
          jbosch Jim Bosch made changes -
          Epic Child DM-19961 [ 306451 ]
          nlust Nate Lust made changes -
          Epic Child DM-19971 [ 307335 ]
          swinbank John Swinbank made changes -
          Epic Child DM-20014 [ 311524 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-18531 [ 244227 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16803 [ 236849 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-12650 [ 36201 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17494 [ 239369 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15749 [ 201290 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16297 [ 231570 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17062 [ 238688 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17689 [ 239741 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-14816 [ 126055 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-19851 [ 297340 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16876 [ 238171 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-12651 [ 36203 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15466 [ 178759 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-16056 [ 223676 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-19961 [ 306451 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15412 [ 168522 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17498 [ 239376 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17154 [ 238910 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-13990 [ 39832 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-19807 [ 293954 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-17017 [ 238492 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-20014 [ 311524 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-19817 [ 294937 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15411 [ 168517 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-15750 [ 201291 ]
          yusra Yusra AlSayyad made changes -
          Epic Child DM-19942 [ 305955 ]
          yusra Yusra AlSayyad made changes -
          Resolution Done [ 10000 ]
          Status In Progress [ 3 ] Done [ 10002 ]
          Hide
          yusra Yusra AlSayyad added a comment -

          See: [ conclusion presented to the DMLT June 4-6 2019|https://confluence.lsstcorp.org/download/attachments/102467909/Gen3%20Middleware%20Progress%20Report%2C%202019-06.pdf?version=1&modificationDate=1559673286000&api=v2]

          Show
          yusra Yusra AlSayyad added a comment - See: [ conclusion presented to the DMLT June 4-6 2019|https://confluence.lsstcorp.org/download/attachments/102467909/Gen3%20Middleware%20Progress%20Report%2C%202019-06.pdf?version=1&modificationDate=1559673286000&api=v2]

            People

            • Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Watchers:
              Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel