Details
-
Type:
Story
-
Status: To Do
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Epic Link:
-
Team:SQuaRE
Description
The cfht and decam datasets have timed out several times now after the pybind11 merge. Eg. https://ci.lsst.codes/job/validate_drp/851/
The runtime before failure for both datasets is ~2 hours 47mins. This isn't conclusively related to the pybind11 merge but the timing is coincidental.
There have been recent changes to both pipe_tasks and pipe_drivers.
https://github.com/lsst/pipe_tasks/commit/edc5e2ac3717660f1ff218ac0e43a61e2afcb794
https://github.com/lsst/pipe_drivers/commit/a2154f1044a73c9f569b26784e97744581503f61
Traceback (most recent call last):
|
File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_tasks/13.0-4-gedc5e2a/bin/processCcd.py", line 25, in <module> |
ProcessCcdTask.parseAndRun()
|
File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0+5/python/lsst/pipe/base/cmdLineTask.py", line 482, in parseAndRun |
resultList = taskRunner.run(parsedCmd)
|
File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0+5/python/lsst/pipe/base/cmdLineTask.py", line 209, in run |
resultList = list(mapFunc(self, targetList))
|
File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0+5/python/lsst/pipe/base/cmdLineTask.py", line 70, in _runPool |
return pool.map_async(functools.partial(_poolFunctionWrapper, function), iterable).get(timeout) |
File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/miniconda/lib/python2.7/multiprocessing/pool.py", line 563, in get |
raise TimeoutError
|
multiprocessing.TimeoutError
|
Attachments
Issue Links
Activity
The default timeout value in pipe_base is 9999s, which explains the ~ 2 hour 47 min runtime(s):
https://github.com/lsst/pipe_base/blob/master/python/lsst/pipe/base/cmdLineTask.py#L137
It appears that the timeout can be controlled via a --timeout argument to processCcd.py. It is probably worth adding support for this flag to examples/runExample.sh, as we know both of these datasets should have much shorter runtimes.
When implementing DM-9749, I tested running with NUMPROC set to 1, and the cfht dataset was able to run to completion.
Per discussion on slack with Michael Wood-Vasey, we are going to set NUMPROC = 1 in production for the cfht/decam datasets in order to get them working for the time being.
To clarify, the hsc dataset is working. It differs from the cfht/decam datasets in that it uses pipe_drivers.