Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4397

Scale CommandLineTask multiprocessing timeout with workload

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_base
    • Labels:
      None
    • Story Points:
      2
    • Team:
      Data Facility

      Description

      The default timeout value for aborting a multiprocessing run in CommandLineTask is too short. Currently if no time length is supplied by the user, the default value gets set to 9999s. However if a processing task is quite large it is possible for the processing pool to take much longer to arrive at the result. Currently if the processing pool does not complete it's run within that time limit, python multiprocessing will throw a timeout error. The timeout value should be scaled such that the supplied value is assumed to be the timeout length for one processing task, and should be scaled by the number of tasks divided by the number of cpus available. The command line task documentation should be updated to reflect this change.

        Attachments

          Activity

          Hide
          swinbank John Swinbank added a comment -

          The solution described here – scaling based on the ratio of tasks to CPUs – doesn't seem to have reached a consensus in the discussion on Discourse.

          Show
          swinbank John Swinbank added a comment - The solution described here – scaling based on the ratio of tasks to CPUs – doesn't seem to have reached a consensus in the discussion on Discourse .
          Hide
          mbutler Michelle Butler added a comment -

          cleaning up old tickets assigned to the data facility.   11/20/2015...  pretty old.  If this is still an issue, please assign to someone and off the data facility.   

          Show
          mbutler Michelle Butler added a comment - cleaning up old tickets assigned to the data facility.   11/20/2015...  pretty old.  If this is still an issue, please assign to someone and off the data facility.   

            People

            • Assignee:
              Unassigned
              Reporter:
              nlust Nate Lust
              Watchers:
              John Swinbank, Michelle Butler, Nate Lust, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel