Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16771

DRP efforts toward Gen 3 Middleware during second half of S19

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      s19-drp-pipelinetask
    • Story Points:
      158
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Spring 2019

      Description

      This is a continuation of the Gen3 middleware and butler work being done by the DRP team on DM-16675. This epic includes work done after the push to get a DRP runnning with a Gen 3 butler. This work is scheduled for February 1st - May 31st. It includes:

      • Ongoing removal of the shims and broken-rules after Jan 31st.¬†
      • Ongoing conversion of additional tasks after Jan 31st.
      • Schema changes
      • Science Pipelines modification so to support new workflow system development
      • Get off Gen2 and pipe_drivers:
        • Repo converter
        • ¬†Implement a ctrl_pool activator
        • Gen3 views into Gen2 repos

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        DM-19871

        Fix validity range end in Gen3 calibration bootstrapping

        1 Jim Bosch Done
         
        DM-19531

        Add option to include regions when ingesting gen3 images

        4 Nate Lust Done
         
        DM-19272

        Make script for creating skymaps in gen3 bulter

        5 Nate Lust Done
         
        DM-19271

        Add transactions to gen3 skymap registry

        4 Nate Lust Done
         
        DM-18863

        Investigate and fix warnings in daf_butler

        2 Nate Lust Done
         
        DM-19731

        Investigate inaccuracy of initial HSC WCSs

        2 Jim Bosch Done
         
        DM-19818

        Fix Gen3 Butler pickling broken on DM-19638

        1 Jim Bosch Done
         
        DM-19808

        Avoid duplication in deferred prelight follow-up queries

        6 Jim Bosch Done
         
        DM-19387

        Vectorize dimension entry addition to gen3 registry

        4 Nate Lust Done
         
        DM-19971

        Fixup for tickets/DM-19272

        0.25 Nate Lust Done
         
        DM-19638

        Create parent task/script for bootstrapping Gen3 repos

        4 Jim Bosch Done
         
        DM-19623

        Change sqlalchemy syntax in addDimensionEntryList

        2 Nate Lust Done
         
        DM-19622

        Make PosixDatastore's internal table lowercase

        0.5 Jim Bosch Done
         
        DM-19615

        Change raw storage class in isr task

        1 Nate Lust Done
         
        DM-19614

        Write transmission curves in writeCuratedCalibrations

        3 Nate Lust Done
         
        DM-18839

        Remove explicit registry close in the butler

        8 Nate Lust Done
         
        DM-18838

        Fix butlerRoot in OracleRegistry

        1 Nate Lust Done
         
        DM-18531

        convert skyCorrection to gen3 pipelineTask

        6 Christopher Waters Done
         
        DM-18367

        Get butler working with Oracle

        14 Nate Lust Done
         
        DM-17912

        gen3 ip_isr can attempt to remove a non-existant dataset, causing KeyError

        1 Christopher Waters Done
         
        DM-17838

        Spend time post-demo reviewing and preparing next stages

        7 Nate Lust Done
         
        DM-17757

        Key error when trying to use pipetask to output pipeline/quantum graph and then read it.

        8 Nate Lust Done
         
        DM-17679

        Butler sqlite data issues

        2 Jim Bosch Done
         
        DM-17611

        Performance optimizations to data ID code

        1 Jim Bosch Done
         
        DM-17608

        Fix performance regression from SQLite transaction changes

        0.5 Jim Bosch Done
         
        DM-17663

        Make Registry table names lowercase

        1 Jim Bosch Done
         
        DM-17496

        QuantumGraph generation hits SQLite join limit

        6 Jim Bosch Done
         
        DM-17491

        Implement Butler deletion APIs

        2 Jim Bosch Done
         
        DM-16539

        Add level of indirection betwen calib identifier and exposure range in schema

        4 Jim Bosch Done
         
        DM-15843

        Clean up details of pipeline task conversion process

        4 Jim Bosch Done
         
        DM-14821

        Provide packed integer versions of Gen3 data IDs

        6 Jim Bosch Done
         
        DM-18027

        gen2convert robustification and enhancements

        15 Christopher Waters Invalid

          Activity

          Hide
          yusra Yusra AlSayyad added a comment -

          See: [ conclusion presented to the DMLT June 4-6 2019|https://confluence.lsstcorp.org/download/attachments/102467909/Gen3%20Middleware%20Progress%20Report%2C%202019-06.pdf?version=1&modificationDate=1559673286000&api=v2]

          Show
          yusra Yusra AlSayyad added a comment - See: [ conclusion presented to the DMLT June 4-6 2019|https://confluence.lsstcorp.org/download/attachments/102467909/Gen3%20Middleware%20Progress%20Report%2C%202019-06.pdf?version=1&modificationDate=1559673286000&api=v2]

            People

            • Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Watchers:
              Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel