Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:10
-
Epic Link:
-
Team:Data Facility
Description
There are always overheads when scheduling jobs, commonly related to scheduling, data transfers, state notifications, and task book keeping. These overheads can be very noticeable for short jobs. The solution recommended by Pegasus team to minimize those overhead is to use task clustering. The goal is to explore this mechanism and access if it is flexible enough to address common operational requirements.
I read available documentation regarding job clustering and tried out available clustering mechanisms on simple workflows.