Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16771

DRP efforts toward Gen 3 Middleware during second half of S19

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      s19-drp-pipelinetask
    • Story Points:
      158
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Spring 2019

      Description

      This is a continuation of the Gen3 middleware and butler work being done by the DRP team on DM-16675. This epic includes work done after the push to get a DRP runnning with a Gen 3 butler. This work is scheduled for February 1st - May 31st. It includes:

      • Ongoing removal of the shims and broken-rules after Jan 31st. 
      • Ongoing conversion of additional tasks after Jan 31st.
      • Schema changes
      • Science Pipelines modification so to support new workflow system development
      • Get off Gen2 and pipe_drivers:
        • Repo converter
        •  Implement a ctrl_pool activator
        • Gen3 views into Gen2 repos

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        		DM-18531

        convert skyCorrection to gen3 pipelineTask

        		 6 Christopher Waters Done
         
        		DM-17496

        QuantumGraph generation hits SQLite join limit

        		 6 Jim Bosch Done
         
        		DM-17491

        Implement Butler deletion APIs

        		 2 Jim Bosch Done
         
        		DM-19871

        Fix validity range end in Gen3 calibration bootstrapping

        		 1 Jim Bosch Done
         
        		DM-19818

        Fix Gen3 Butler pickling broken on DM-19638

        		 1 Jim Bosch Done
         
        		DM-19808

        Avoid duplication in deferred prelight follow-up queries

        		 6 Jim Bosch Done
         
        		DM-19731

        Investigate inaccuracy of initial HSC WCSs

        		 2 Jim Bosch Done
         
        		DM-19971

        Fixup for tickets/DM-19272

        		 0.25 Nate Lust Done
         
        		DM-19531

        Add option to include regions when ingesting gen3 images

        		 4 Nate Lust Done
         
        		DM-19387

        Vectorize dimension entry addition to gen3 registry

        		 4 Nate Lust Done
         
        		DM-19638

        Create parent task/script for bootstrapping Gen3 repos

        		 4 Jim Bosch Done
         
        		DM-19623

        Change sqlalchemy syntax in addDimensionEntryList

        		 2 Nate Lust Done
         
        		DM-19622

        Make PosixDatastore's internal table lowercase

        		 0.5 Jim Bosch Done
         
        		DM-19615

        Change raw storage class in isr task

        		 1 Nate Lust Done
         
        		DM-19614

        Write transmission curves in writeCuratedCalibrations

        		 3 Nate Lust Done
         
        		DM-18367

        Get butler working with Oracle

        		 14 Nate Lust Done
         
        		DM-17912

        gen3 ip_isr can attempt to remove a non-existant dataset, causing KeyError

        		 1 Christopher Waters Done
         
        		DM-17838

        Spend time post-demo reviewing and preparing next stages

        		 7 Nate Lust Done
         
        		DM-17679

        Butler sqlite data issues

        		 2 Jim Bosch Done
         
        		DM-17663

        Make Registry table names lowercase

        		 1 Jim Bosch Done
         
        		DM-17611

        Performance optimizations to data ID code

        		 1 Jim Bosch Done
         
        		DM-17608

        Fix performance regression from SQLite transaction changes

        		 0.5 Jim Bosch Done
         
        		DM-17757

        Key error when trying to use pipetask to output pipeline/quantum graph and then read it.

        		 8 Nate Lust Done
         
        		DM-18839

        Remove explicit registry close in the butler

        		 8 Nate Lust Done
         
        		DM-18838

        Fix butlerRoot in OracleRegistry

        		 1 Nate Lust Done
         
        		DM-18863

        Investigate and fix warnings in daf_butler

        		 2 Nate Lust Done
         
        		DM-19272

        Make script for creating skymaps in gen3 bulter

        		 5 Nate Lust Done
         
        		DM-19271

        Add transactions to gen3 skymap registry

        		 4 Nate Lust Done
         
        		DM-15843

        Clean up details of pipeline task conversion process

        		 4 Jim Bosch Done
         
        		DM-16539

        Add level of indirection betwen calib identifier and exposure range in schema

        		 4 Jim Bosch Done
         
        		DM-14821

        Provide packed integer versions of Gen3 data IDs

        		 6 Jim Bosch Done
         
        		DM-18027

        gen2convert robustification and enhancements

        		 15 Christopher Waters Invalid

          Activity

          Hide
          Permalink
          yusra Yusra AlSayyad added a comment -

          See: [ conclusion presented to the DMLT June 4-6 2019|https://confluence.lsstcorp.org/download/attachments/102467909/Gen3%20Middleware%20Progress%20Report%2C%202019-06.pdf?version=1&modificationDate=1559673286000&api=v2]

          Show
          yusra Yusra AlSayyad added a comment - See: [ conclusion presented to the DMLT June 4-6 2019|https://confluence.lsstcorp.org/download/attachments/102467909/Gen3%20Middleware%20Progress%20Report%2C%202019-06.pdf?version=1&modificationDate=1559673286000&api=v2]

            People

            Assignee:
            yusra Yusra AlSayyad
            Reporter:
            yusra Yusra AlSayyad
            Watchers:
            Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.