Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9746

validate_drp cfht/decam datasets timing out post pybind11 merge

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The cfht and decam datasets have timed out several times now after the pybind11 merge. Eg. https://ci.lsst.codes/job/validate_drp/851/

      The runtime before failure for both datasets is ~2 hours 47mins. This isn't conclusively related to the pybind11 merge but the timing is coincidental.

      There have been recent changes to both pipe_tasks and pipe_drivers.

      https://github.com/lsst/pipe_tasks/commit/edc5e2ac3717660f1ff218ac0e43a61e2afcb794
      https://github.com/lsst/pipe_drivers/commit/a2154f1044a73c9f569b26784e97744581503f61

      Traceback (most recent call last):
        File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_tasks/13.0-4-gedc5e2a/bin/processCcd.py", line 25, in <module>
          ProcessCcdTask.parseAndRun()
        File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0+5/python/lsst/pipe/base/cmdLineTask.py", line 482, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0+5/python/lsst/pipe/base/cmdLineTask.py", line 209, in run
          resultList = list(mapFunc(self, targetList))
        File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0+5/python/lsst/pipe/base/cmdLineTask.py", line 70, in _runPool
          return pool.map_async(functools.partial(_poolFunctionWrapper, function), iterable).get(timeout)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/lsstsw/miniconda/lib/python2.7/multiprocessing/pool.py", line 563, in get
          raise TimeoutError
      multiprocessing.TimeoutError
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Angelo Fausti, John Parejko, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel