Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16675

DRP efforts toward Gen 3 Middleware: PipelineTask Conversion Dec/Jan

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Epic Name:
      s19a-drp-pipelinetask
    • Story Points:
      100
    • Sprint:
      DRP S19-3
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Spring 2019

      Description

      This epic will capture DRP's efforts toward Gen 3 middleware, primarily Nate Lust, Chris Waters, and Jim Bosch with some initial effort from Yusra AlSayyad. This includes work done toward the January 31st deadline to demo a DRP with the Gen3 butler:

      • Quickly converting the Tasks run in an HSC DRP (except meas_mosaic or sky correction) by Jan 31st to demonstrate that a DRP can be run with a Gen 3 butler.  These conversions will "break the rules" and add shims where needed. 
      • Gen3 Butler and PipelineTask features and bugfixes as needed. 

      Work this cycle to be continued on: https://jira.lsstcorp.org/browse/DM-16771

        Attachments

          Container Issues

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          DM-17665

          Integrate and test PipelineTask systems and prepare demo presentation

          6 Nate Lust Done
           
          DM-17395

          Write all outputs from CharacterizeTask in ci_hsc

          1 Nate Lust Done
           
          DM-17301

          Update DetectCoaddSources for new PipelineTask utilities

          2 Nate Lust Done
           
          DM-17028

          Write PipelineTask to Make Warps (MakeCoaddTempExp conversion)

          8 Yusra AlSayyad Done
           
          DM-16931

          Activator should register dataset types before trying to run pipeline

          1 Nate Lust Done
           
          DM-16844

          Convert DeblendCoaddSourcesTask to PipelineTask framework

          6 Nate Lust Done
           
          DM-16843

          Fix typo in commit to address review comments for init(inputs/outputs)

          0.25 Nate Lust Done
           
          DM-17197

          Support PipelineTask execution without writing init output datasets

          2 Jim Bosch Done
           
          DM-16873

          Convert MeasureMergedCoaddSources to Pipeline Task

          8 Nate Lust Done
           
          DM-17545

          Fix MeasureCoaddSources regression

          1 Nate Lust Done
           
          DM-17505

          Fix bug in associate implementation

          0.5 Jim Bosch Done
           
          DM-17474

          Convert forcedPhotCoadd/Ccd to pipelineTask

          6 Christopher Waters Done
           
          DM-17447

          Update ci_hsc preSfm command

          1 Nate Lust Done
           
          DM-17300

          Add brightObjectMasks to gen2convert

          1 Jim Bosch Done
           
          DM-17048

          Fix handling of ref_cat in gen2convert

          0.5 Jim Bosch Done
           
          DM-17493

          Include Filter in Gen3 HSC raw formatter

          1 Jim Bosch Done
           
          DM-17495

          Running PipelineTasks in parallel can lead to aborting on locks instead of blocking

          4 Jim Bosch Done
           
          DM-17390

          Convert CalibrateTask into a PipelineTask

          8 Nate Lust Done
           
          DM-17146

          Fix storageClass for DetectCoaddSources PipelineTask

          0.5 Nate Lust Done
           
          DM-17149

          MergeDetectionTask PipelineTask mode should export peak schema

          0.5 Nate Lust Done
           
          DM-17489

          Initial prototype of Gen3 interfaces for human-curated master calibrations

          2 Jim Bosch Done
           
          DM-17492

          Fix problems in PipelineTasks uncovered by end to end tests

          6 Nate Lust Done
           
          DM-17060

          Fix non-merged ticket

          1 Nate Lust Done
           
          DM-17045

          Convert AssembleCoaddTasks to PipelineTasks with Shims

          8 Yusra AlSayyad Done
           
          DM-16904

          Pass butler object to adaptArgsAndRun in PipelineTask

          1 Nate Lust Done
           
          DM-17416

          Fix origin parameter name in Gen2->Gen3 Butler shim

          0.5 Jim Bosch Done
           
          DM-17412

          Make MergeMeasurementsTask a valid pipelineTask

          2 Christopher Waters Done
           
          DM-16837

          Let PipelineTasks accept config parameters that are lists from the command line

          1 Nate Lust Done
           
          DM-16801

          Add method to turn sequential indexes to pairs in skymap

          2 Nate Lust Done
           
          DM-16819

          Make minimal Gen3 shim for Gen2 DataRef, ButlerSubset, and Butler

          2 Jim Bosch Done
           
          DM-16821

          Add better debug output to butler queries

          1 Nate Lust Done
           
          DM-16810

          Butler schema changes to run on Oracle

          1 Jim Bosch Done
           
          DM-16811

          Add initinputarguments to quantum graph

          4 Nate Lust Done
           
          DM-16834

          Fix skymap Dimension setting for converted Gen2 SkyMap Datasets

          0.5 Jim Bosch Done
           
          DM-16797

          Add template string names and formatters to PipelineTask configs

          2 Nate Lust Done
           
          DM-16467

          isrTask conversion to pipelineTask

          8 Christopher Waters Done
           
          DM-15845

          Convert MergeDetectionsTask into a pipelinetask

          6 Nate Lust Done
           
          DM-15675

          Make sure data IDs are expanded when adding Datasets and filling templates

          6 Jim Bosch Done
           
          DM-15503

          Improve raw data handling in gen2convert

          2 Jim Bosch Done
           
          DM-15366

          Attempt to create Butler schema in NCSA Oracle database

          2 Pim Schellart [X] (Inactive) Done
           
          DM-15034

          Custom classes for DataUnit tuples/sets and Data IDs

          10 Jim Bosch Done
           
          DM-14819

          Refactor LoadReferenceObjectsTask for SuperTask compatibility

          8 Nate Lust Done
           
          DM-16221

          Avoid full-job transactions in Gen3 ingest

          2 Jim Bosch Won't Fix
           
          DM-17302

          Support brightObjectMask datasets in Gen3 butler and gen2convert

          1 Jim Bosch Invalid
           
          DM-15236

          Refactor MakeCoaddTempExp s.t. run doesn't touch disk

          Yusra AlSayyad Invalid
           
          DM-14591

          Fix handling of filter in templates

          1 Jim Bosch Invalid

            Activity

              People

              • Assignee:
                yusra Yusra AlSayyad
                Reporter:
                yusra Yusra AlSayyad
                Watchers:
                Fritz Mueller, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel