Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32675

Improvement PanDA plugin on using iDDS

    XMLWordPrintable

Details

    Description

      There are few items could be (cumulatively) implemented:
      1) Since the PanDA plugin was created, iDDS was updating the API, the current becomes outdated. We should update the API calls
      2) iDDS now supporting propagation of the IAM authorization information into the workflow. The way of authentication and using PanDA as authorization proxy should be updated.
      3) The workflows with very large tasks (60k) jobs creates large memory footprints in the iDDS server, it would be more efficient if they get split into few less chunks of less size.

      Attachments

        Issue Links

          Activity

            A few things about the changes and then a couple higher level things:

            • Need a file in doc/changes
            • To understand new chunk function I had to look at code not in this PR. The docstring for parameter dependencies in split_dependencies_by_tasks doesn't match code.

            Approved for merging after addressing concerns.

            mgower Michelle Gower added a comment - A few things about the changes and then a couple higher level things: Need a file in doc/changes To understand new chunk function I had to look at code not in this PR. The docstring for parameter dependencies in split_dependencies_by_tasks doesn't match code. Approved for merging after addressing concerns.

            Thanks a lot, mgower for the review!

            podolsky Sergey Padolski added a comment - Thanks a lot, mgower  for the review!

            tjenness Will v23 be getting the new iDDS versions?  If not, we're just backporting changes that aren't just due to the new API, right (e.g. the 60k job splitting)?

            mgower Michelle Gower added a comment - tjenness Will v23 be getting the new iDDS versions?  If not, we're just backporting changes that aren't just due to the new API, right (e.g. the 60k job splitting)?

            People

              podolsky Sergey Padolski
              podolsky Sergey Padolski
              Michelle Gower
              Hsin-Fang Chiang, Michelle Gower, Sergey Padolski, Shuwei Ye, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.