Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21892

Implement ctrl_pool / MPI / Slurm PipelineTask executor

    Details

    • Templates:
    • Story Points:
      10
    • Team:
      Data Release Production

      Description

      Implement a PipelineTask executor that submits to all batch systems currently supported by ctrl_pool and uses MPI to distribute Quantum executions to worker nodes as the QuantumGraph is walked in dependency order.

      I'm hoping the first version of this can get away with assuming all Quanta require the same number of cores and same amount of memory, as that will make things substantially easier (though that would require running jointcal and FGCM separately, and hence bifurcating the DRP pipeline around them).

      A moderately mature Batch Production Service with features needed by Science Pipelines developers could remove LSST's need for this executor entirely, though that would need to be accompanied by hardware configured to use HTCondor jobs instead of / in addition to Slurm jobs during the transition away from Gen2 (that may already exist; I don't know).  It is likely that HSC and PFS would continue to want this, however, and it might be that Paul Price would want to be the person to work on this for educational purposes.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Summary Panel