Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4397

Scale CommandLineTask multiprocessing timeout with workload

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_base
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Team:
      Data Facility

      Description

      The default timeout value for aborting a multiprocessing run in CommandLineTask is too short. Currently if no time length is supplied by the user, the default value gets set to 9999s. However if a processing task is quite large it is possible for the processing pool to take much longer to arrive at the result. Currently if the processing pool does not complete it's run within that time limit, python multiprocessing will throw a timeout error. The timeout value should be scaled such that the supplied value is assumed to be the timeout length for one processing task, and should be scaled by the number of tasks divided by the number of cpus available. The command line task documentation should be updated to reflect this change.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              nlust Nate Lust
              Watchers:
              John Swinbank, Michelle Butler, Nate Lust, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel