Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15406

mosaic.py timeout error in readCatalog

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_mosaic
    • Labels:
      None
    • Templates:
    • Team:
      External

      Description

      Jeffrey Carlin and I noticed that running mosaic.py with numCoresForReadSource >1 no longer works, it hangs forever, results in timeout like below:

      Traceback (most recent call last):
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_mosaic/16.0-6-ged8029f+4/bin/mosaic.py", line 5, in <module>
          MosaicTask.parseAndRun()
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-6-g44ca919+2/python/lsst/pipe/base/cmdLineTask.py", line 575, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-6-g44ca919+2/python/lsst/pipe/base/cmdLineTask.py", line 224, in run
          resultList = list(mapFunc(self, targetList))
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_mosaic/16.0-6-ged8029f+4/python/lsst/meas/mosaic/mosaicTask.py", line 80, in __call__
          result = task.run(*args)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_mosaic/16.0-6-ged8029f+4/python/lsst/meas/mosaic/mosaicTask.py", line 1054, in run
          numCoresForReadSource, readTimeout, verbose)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_mosaic/16.0-6-ged8029f+4/python/lsst/meas/mosaic/mosaicTask.py", line 871, in mosaic
          readTimeout, verbose)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_mosaic/16.0-6-ged8029f+4/python/lsst/meas/mosaic/mosaicTask.py", line 554, in readCatalog
          resultList = pool.map_async(worker, params).get(readTimeout)
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/multiprocessing/pool.py", line 640, in get
          raise TimeoutError
      

      Stack version w_2018_30 was used.

      FWIW, I was able to run the same thing using only 1 core for reading source catalog (--numCoresForReadSource=1).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                price Paul Price
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Jim Bosch
                Watchers:
                Hsin-Fang Chiang, Jeffrey Carlin, Jim Bosch, John Swinbank, Paul Price, Pim Schellart [X] (Inactive), Russell Owen, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel