Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13926

Dataset Reprocessing Campaigns (FY18b-1)

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Epic Name:
      Dataset Reprocessing Campaigns (FY18b-1)
    • Story Points:
      80
    • WBS:
      02C.07.06.01
    • Team:
      Data Facility
    • Cycle:
      Fall 2018

      Description

      Execute data reprocessing campaigns on test data. Processing campaigns are specified by a goal/purpose, pipeline(s) to be run, configurations, and input and output dataset(s). Pre-camera data campaigns support pipeline development and evolution of the batch production service. Work includes setup, testing, execution, debugging, verification of results, handling of data products, and liaison with the Pipelines teams. Campaigns anticipated for FY18b include:
      1) "Weeklies," which are run ~biweekly on a pre-defined subset of HSC data (RC2) using the latest tagged version of the LSST Stack.
      2) "PDR1 Reprocessing," which is run over the entire HSC PDR dataset using a stable version of the LSST Stack. PDR1 reprocessing campaigns are scheduled with the DRP group.
      Deliverables include data products delivered to the developer-accessible file system.

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        		DM-14547

        Reprocess RC2 with w_2018_22

        		 3 Hsin-Fang Chiang Done
         
        		DM-14546

        Run pipe_analysis and validate_drp with w_2018_20 HSC RC2 outputs

        		 1 Hsin-Fang Chiang Done
         
        		DM-14362

        summarize low-level processing details for the operations team

        		 1 Hsin-Fang Chiang Done
         
        		DM-14354

        Summarize node utilization for RC2 jobs for w_2018_15 w_2018_17 w_2018_18

        		 0.25 Samantha Thrush Done
         
        		DM-14288

        Create node-hour usage plots for S18 HSC PDR1 reprocessing

        		 1.5 Samantha Thrush Done
         
        		DM-14340

        Run pipe_analysis scripts with the RC2 data of w_2018_18

        		 0.5 Hsin-Fang Chiang Done
         
        		DM-14339

        Reprocess RC2 with w_2018_20

        		 3 Hsin-Fang Chiang Done
         
        		DM-14341

        Run validate_drp with w_2018_18 HSC RC2 outputs

        		 1 Hsin-Fang Chiang Done
         
        		DM-14880

        Run pipe_analysis and validate_drp with w_2018_24 HSC RC2 outputs

        		 1 Hsin-Fang Chiang Done
         
        		DM-14071

        Allow usage.py to account for failed jobs

        		 1 Samantha Thrush Done
         
        		DM-14070

        Modify usage.py to output the node-hours for each code

        		 0.15 Samantha Thrush Done
         
        		DM-14057

        Simplify dealing with task names mapping

        		 0.5 Samantha Thrush Done
         
        		DM-14056

        Summarize node utilization for RC2 jobs

        		 1 Samantha Thrush Done
         
        		DM-14055

        Reprocess RC2 with w_2018_17

        		 3 Hsin-Fang Chiang Done
         
        		DM-14054

        Add command line option for resolution in usage.py

        		 0.5 Samantha Thrush Done
         
        		DM-14048

        Run validate_drp with w_2018_14 HSC RC2 outputs

        		 0.5 Hsin-Fang Chiang Done
         
        		DM-14047

        Run pipe_analysis scripts with the RC2 data of w_2018_14

        		 1 Hsin-Fang Chiang Done
         
        		DM-14686

        Run pipe_analysis and validate_drp with w_2018_22 HSC RC2 outputs

        		 1 Hsin-Fang Chiang Done
         
        		DM-14688

        Reprocess RC2 with w_2018_24

        		 3 Hsin-Fang Chiang Done
         
        		DM-14245

        Run pipe_analysis scripts with the RC2 data of w_2018_17

        		 1 Hsin-Fang Chiang Done
         
        		DM-14243

        Reprocess RC2 with w_2018_18

        		 3 Hsin-Fang Chiang Done
         
        		DM-14202

        multiBandDriver of the HSC PDR1 dataset

        		 5 Hsin-Fang Chiang Done
         
        		DM-14201

        coaddDriver of the HSC PDR1 dataset

        		 4 Hsin-Fang Chiang Done
         
        		DM-14200

        mosaic of the HSC PDR1 dataset

        		 2 Hsin-Fang Chiang Done
         
        		DM-14199

        skyCorrection of the HSC PDR1 dataset

        		 1 Hsin-Fang Chiang Done
         
        		DM-14144

        Run validate_drp with w_2018_15 HSC RC2 outputs

        		 0.25 Hsin-Fang Chiang Done
         
        		DM-14137

        Run pipe_analysis scripts with the RC2 data of w_2018_15

        		 1 Hsin-Fang Chiang Done
         
        		DM-14123

        Reprocess RC2 with w_2018_15

        		 3 Hsin-Fang Chiang Done
         
        		DM-14256

        Run validate_drp with w_2018_17 HSC RC2 outputs

        		 1 Hsin-Fang Chiang Done
         
        		DM-14101

        Report the node-hour breakdown in S17B PDR1 reprocessing

        		 0.1 Samantha Thrush Done
         
        		DM-14113

        Divide usage.py functions into modules

        		 1 Samantha Thrush Done
         
        		DM-14112

        Reprocess RC with stack version 15.0 and compare to version 14.0 for the VVDS tract

        		 6 Samantha Thrush Done
         
        		DM-14111

        Overhaul usage.py Readme file and delete key_len variable

        		 1 Samantha Thrush Done
         
        		DM-13667

        singleFrameDriver of the HSC PDR1 dataset

        		 7 Hsin-Fang Chiang Done
         
        		DM-13665

        Finalize the stack version, step, and config for the S18 PDR1 reprocessing

        		 1 Hsin-Fang Chiang Done
         
        		DM-13890

        Reprocess RC2 with w_2018_14

        		 3 Hsin-Fang Chiang Done
         
        		DM-13816

        Modify usage.py to allow the user to specify SLURM job names.

        		 4 Samantha Thrush Done
         
        		DM-13819

        Create a top-level script to run both usage.py and usageplot.py

        		 2 Samantha Thrush Done
         
        		DM-13818

        Modify usage.py to output node-hours that take into account significant figures

        		 2 Samantha Thrush Done

          Activity

          Hide
          Permalink
          plutchak Joel Plutchak (Inactive) added a comment -

          Time-bounded epic. Resulting datasets and logs are in subdirectories of /datasets/hsc/repo/rerun/RC/.  Results and processes are described at https://confluence.lsstcorp.org/display/DM/Data+Processing+End+to+End+Testing

          Show
          plutchak Joel Plutchak (Inactive) added a comment - Time-bounded epic. Resulting datasets and logs are in subdirectories of /datasets/hsc/repo/rerun/RC/ .  Results and processes are described at https://confluence.lsstcorp.org/display/DM/Data+Processing+End+to+End+Testing

            People

            Assignee:
            hchiang2 Hsin-Fang Chiang
            Reporter:
            plutchak Joel Plutchak (Inactive)
            Watchers:
            Joel Plutchak (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Start date:
              End date:

                Jenkins

                No builds found.