Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10183

Investigate why maxtasksperchild=1 causes mosaic.py to hang on pybind11 stack

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_mosaic
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Team:
      Data Release Production

      Description

      mosaic.py uses multiprocessing.Pool to read catalogs with multiple cores. When this pool is initialized with maxtasksperchild=1, mosaic.py hangs indefinitely at a consistent point in the running---that is, running with the same arguments multiple times will freeze up in the same place. This is only a problem with the pybind11 version of the stack, as this behavior does not occur in the HSC stack, which is currently still wrapped with swig. The underlying cause of this should be investigated to make sure that there is not some deeper issue that might cause problems with parallelization elsewhere.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                price Paul Price
                Reporter:
                tmorton Tim Morton
                Reviewers:
                Tim Morton
                Watchers:
                John Swinbank, Paul Price, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel