Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4397

Scale CommandLineTask multiprocessing timeout with workload

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: pipe_base
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Team:
      Data Facility

      Description

      The default timeout value for aborting a multiprocessing run in CommandLineTask is too short. Currently if no time length is supplied by the user, the default value gets set to 9999s. However if a processing task is quite large it is possible for the processing pool to take much longer to arrive at the result. Currently if the processing pool does not complete it's run within that time limit, python multiprocessing will throw a timeout error. The timeout value should be scaled such that the supplied value is assumed to be the timeout length for one processing task, and should be scaled by the number of tasks divided by the number of cpus available. The command line task documentation should be updated to reflect this change.

        Attachments

          Container Issues

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                nlust Nate Lust
                Watchers:
                John Swinbank, Nate Lust, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel