Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14818

Add hooks for running converted SuperTasks in ci_hsc

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
    • Story Points:
      2
    • Team:
      Data Release Production

      Description

      On master, ci_hsc simply runs all CmdLineTasks in the prototype DRP pipeline, then runs custom validate code after each to check that outputs exist.

      On the gen3-middleware branch, we also create a Gen3 data repository containing all outputs and re-run all validation code.

      Once SuperTask conversion begins, we will want to have an easy way to include executing those SuperTasks in ci_hsc, regardless of whether preceding CmdLineTasks have been converted (we can do this by allowing SuperTasks to use the converted Gen2 outputs as inputs).  We'd then want to follow this up with validating those outputs as well.

      In the long-term, we will those validation steps to themselves be SuperTasks, and we'll want them to live outside ci_hsc.  We'll also want to replace ci_hsc's SCons control logic with a real SuperTask executor.  With those goals in mind, the end point of this ticket should just be a way to easily add a concrete SuperTask to ci_hsc and validate its outputs.

       

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            T

             

            Show
            jbosch Jim Bosch added a comment - T  

              People

              • Assignee:
                Unassigned
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Jim Bosch
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel