Data ManagementIcon indicating the project type

Data Management

Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16675

DRP efforts toward Gen 3 Middleware: PipelineTask Conversion Dec/Jan

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      s19a-drp-pipelinetask
    • Story Points:
      100
    • Sprint:
      DRP S19-3
    • WBS:
      02C.04.01
    • Team:
      Data Release Production
    • Cycle:
      Spring 2019

      Description

      This epic will capture DRP's efforts toward Gen 3 middleware, primarily Nate Lust, Chris Waters, and Jim Bosch with some initial effort from Yusra AlSayyad. This includes work done toward the January 31st deadline to demo a DRP with the Gen3 butler:

      • Quickly converting the Tasks run in an HSC DRP (except meas_mosaic or sky correction) by Jan 31st to demonstrate that a DRP can be run with a Gen 3 butler.  These conversions will "break the rules" and add shims where needed. 
      • Gen3 Butler and PipelineTask features and bugfixes as needed. 

      Work this cycle to be continued on: https://jira.lsstcorp.org/browse/DM-16771

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        		DM-17665

        Integrate and test PipelineTask systems and prepare demo presentation

        		6 Nate Lust Done
         
        		DM-17545

        Fix MeasureCoaddSources regression

        		1 Nate Lust Done
         
        		DM-17416

        Fix origin parameter name in Gen2->Gen3 Butler shim

        		0.5 Jim Bosch Done
         
        		DM-17412

        Make MergeMeasurementsTask a valid pipelineTask

        		2 Christopher Waters Done
         
        		DM-17395

        Write all outputs from CharacterizeTask in ci_hsc

        		1 Nate Lust Done
         
        		DM-17390

        Convert CalibrateTask into a PipelineTask

        		8 Nate Lust Done
         
        		DM-17474

        Convert forcedPhotCoadd/Ccd to pipelineTask

        		6 Christopher Waters Done
         
        		DM-17447

        Update ci_hsc preSfm command

        		1 Nate Lust Done
         
        		DM-17505

        Fix bug in associate implementation

        		0.5 Jim Bosch Done
         
        		DM-17495

        Running PipelineTasks in parallel can lead to aborting on locks instead of blocking

        		4 Jim Bosch Done
         
        		DM-17493

        Include Filter in Gen3 HSC raw formatter

        		1 Jim Bosch Done
         
        		DM-17492

        Fix problems in PipelineTasks uncovered by end to end tests

        		6 Nate Lust Done
         
        		DM-17489

        Initial prototype of Gen3 interfaces for human-curated master calibrations

        		2 Jim Bosch Done
         
        		DM-17301

        Update DetectCoaddSources for new PipelineTask utilities

        		2 Nate Lust Done
         
        		DM-17300

        Add brightObjectMasks to gen2convert

        		1 Jim Bosch Done
         
        		DM-17149

        MergeDetectionTask PipelineTask mode should export peak schema

        		0.5 Nate Lust Done
         
        		DM-17146

        Fix storageClass for DetectCoaddSources PipelineTask

        		0.5 Nate Lust Done
         
        		DM-17197

        Support PipelineTask execution without writing init output datasets

        		2 Jim Bosch Done
         
        		DM-17060

        Fix non-merged ticket

        		1 Nate Lust Done
         
        		DM-17048

        Fix handling of ref_cat in gen2convert

        		0.5 Jim Bosch Done
         
        		DM-17045

        Convert AssembleCoaddTasks to PipelineTasks with Shims

        		8 Yusra AlSayyad Done
         
        		DM-16931

        Activator should register dataset types before trying to run pipeline

        		1 Nate Lust Done
         
        		DM-16904

        Pass butler object to adaptArgsAndRun in PipelineTask

        		1 Nate Lust Done
         
        		DM-17028

        Write PipelineTask to Make Warps (MakeCoaddTempExp conversion)

        		8 Yusra AlSayyad Done
         
        		DM-16873

        Convert MeasureMergedCoaddSources to Pipeline Task

        		8 Nate Lust Done
         
        		DM-16844

        Convert DeblendCoaddSourcesTask to PipelineTask framework

        		6 Nate Lust Done
         
        		DM-16843

        Fix typo in commit to address review comments for init(inputs/outputs)

        		0.25 Nate Lust Done
         
        		DM-16821

        Add better debug output to butler queries

        		1 Nate Lust Done
         
        		DM-16837

        Let PipelineTasks accept config parameters that are lists from the command line

        		1 Nate Lust Done
         
        		DM-16834

        Fix skymap Dimension setting for converted Gen2 SkyMap Datasets

        		0.5 Jim Bosch Done
         
        		DM-16819

        Make minimal Gen3 shim for Gen2 DataRef, ButlerSubset, and Butler

        		2 Jim Bosch Done
         
        		DM-16811

        Add initinputarguments to quantum graph

        		4 Nate Lust Done
         
        		DM-16810

        Butler schema changes to run on Oracle

        		1 Jim Bosch Done
         
        		DM-15675

        Make sure data IDs are expanded when adding Datasets and filling templates

        		6 Jim Bosch Done
         
        		DM-16801

        Add method to turn sequential indexes to pairs in skymap

        		2 Nate Lust Done
         
        		DM-16797

        Add template string names and formatters to PipelineTask configs

        		2 Nate Lust Done
         
        		DM-16467

        isrTask conversion to pipelineTask

        		8 Christopher Waters Done
         
        		DM-15845

        Convert MergeDetectionsTask into a pipelinetask

        		6 Nate Lust Done
         
        		DM-14819

        Refactor LoadReferenceObjectsTask for SuperTask compatibility

        		8 Nate Lust Done
         
        		DM-15366

        Attempt to create Butler schema in NCSA Oracle database

        		2 Pim Schellart [X] (Inactive) Done
         
        		DM-15503

        Improve raw data handling in gen2convert

        		2 Jim Bosch Done
         
        		DM-15034

        Custom classes for DataUnit tuples/sets and Data IDs

        		10 Jim Bosch Done
         
        		DM-16221

        Avoid full-job transactions in Gen3 ingest

        		2 Jim Bosch Won't Fix
         
        		DM-17302

        Support brightObjectMask datasets in Gen3 butler and gen2convert

        		1 Jim Bosch Invalid
         
        		DM-14591

        Fix handling of filter in templates

        		1 Jim Bosch Invalid
         
        		DM-15236

        Refactor MakeCoaddTempExp s.t. run doesn't touch disk

        		Yusra AlSayyad Invalid

          Activity

          Hide
          Permalink
          yusra Yusra AlSayyad added a comment - - edited

          This epic culminated in the conversion of all* tasks run in ci_hsc, starting with IsrTask and ending with ForcedPhotCcdTask. 

          During the demo on Feb 1st, Nate presented the state of Pipeline Tasks from a Science Pipelines perspective in:

          https://lsst-dm.github.io/PipelineTask_demo_presentation/

          and demonstrated a full DRP being run on the ci_hsc dataset with Gen 3 middleware

           

          * The exception was SkyCorrectionTask which was an HSC-specific feature ported from NAOJ,  implemented as a pipe_driver rather than a regular task in pipe_tasks.  pipe_drivers will be tackled in DM-16771 

          Show
          yusra Yusra AlSayyad added a comment - - edited This epic culminated in the conversion of all* tasks run in ci_hsc, starting with IsrTask and ending with ForcedPhotCcdTask.  During the demo on Feb 1st, Nate presented the state of Pipeline Tasks from a Science Pipelines perspective in: https://lsst-dm.github.io/PipelineTask_demo_presentation/ and demonstrated a full DRP being run on the ci_hsc dataset with Gen 3 middleware   * The exception was SkyCorrectionTask which was an HSC-specific feature ported from NAOJ,  implemented as a pipe_driver rather than a regular task in pipe_tasks.  pipe_drivers will be tackled in  DM-16771  

            People

            • Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Watchers:
              Fritz Mueller, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel