Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10991

Optimizing workflows

    XMLWordPrintable

    Details

      Description

      There are always overheads when scheduling jobs, commonly related to scheduling, data transfers, state notifications, and task book keeping. These overheads can be very noticeable for short jobs. The solution recommended by Pegasus team to minimize those overhead is to use task clustering. The goal is to explore this mechanism and access if it is flexible enough to address common operational requirements.

        Attachments

          Activity

          Hide
          mkowalik Mikolaj Kowalik added a comment -

          I read available documentation regarding job clustering and tried out available clustering mechanisms on simple workflows.

          Show
          mkowalik Mikolaj Kowalik added a comment - I read available documentation regarding job clustering and tried out available clustering mechanisms on simple workflows.

            People

            Assignee:
            mkowalik Mikolaj Kowalik
            Reporter:
            mkowalik Mikolaj Kowalik
            Watchers:
            Mikolaj Kowalik
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.