Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32675

Improvement PanDA plugin on using iDDS

    XMLWordPrintable

    Details

      Description

      There are few items could be (cumulatively) implemented:
      1) Since the PanDA plugin was created, iDDS was updating the API, the current becomes outdated. We should update the API calls
      2) iDDS now supporting propagation of the IAM authorization information into the workflow. The way of authentication and using PanDA as authorization proxy should be updated.
      3) The workflows with very large tasks (60k) jobs creates large memory footprints in the iDDS server, it would be more efficient if they get split into few less chunks of less size.

        Attachments

          Issue Links

            Activity

            Hide
            mgower Michelle Gower added a comment -

            A few things about the changes and then a couple higher level things:

            • Need a file in doc/changes
            • To understand new chunk function I had to look at code not in this PR. The docstring for parameter dependencies in split_dependencies_by_tasks doesn't match code.

            Approved for merging after addressing concerns.

            Show
            mgower Michelle Gower added a comment - A few things about the changes and then a couple higher level things: Need a file in doc/changes To understand new chunk function I had to look at code not in this PR. The docstring for parameter dependencies in split_dependencies_by_tasks doesn't match code. Approved for merging after addressing concerns.
            Hide
            podolsky Sergey Padolski added a comment -

            Thanks a lot, Michelle Gower for the review!

            Show
            podolsky Sergey Padolski added a comment - Thanks a lot, Michelle Gower  for the review!
            Hide
            mgower Michelle Gower added a comment -

            Tim Jenness Will v23 be getting the new iDDS versions?  If not, we're just backporting changes that aren't just due to the new API, right (e.g. the 60k job splitting)?

            Show
            mgower Michelle Gower added a comment - Tim Jenness Will v23 be getting the new iDDS versions?  If not, we're just backporting changes that aren't just due to the new API, right (e.g. the 60k job splitting)?

              People

              Assignee:
              podolsky Sergey Padolski
              Reporter:
              podolsky Sergey Padolski
              Reviewers:
              Michelle Gower
              Watchers:
              Hsin-Fang Chiang, Michelle Gower, Sergey Padolski, Shuwei Ye, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.