Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16957

Multi-process option is broken in pipe_supertask

    Details

    • Story Points:
      1
    • Sprint:
      BG3_F18_11
    • Team:
      Data Access and Database

      Description

      Looks like another victim of Unit to Dimension switch - -j option stopped working in stac:

      $ stac -b ../ci_hsc/DATA/butler.yaml -i shared/ci_hsc -o coll-C -j2 run -t RawToCalexpTask
      Process ForkPoolWorker-1:
      Process ForkPoolWorker-2:
      Traceback (most recent call last):
        File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
          self.run()
      Traceback (most recent call last):
        File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/process.py", line 93, in run
          self._target(*self._args, **self._kwargs)
        File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/pool.py", line 108, in worker
          task = get()
        File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/queues.py", line 337, in get
          return _ForkingPickler.loads(res)
        File "/project/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/dimensions/dataId.py", line 134, in __new__
          raise ValueError(f"Cannot infer dimensions without universe.")
        File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
          self.run()
        File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/process.py", line 93, in run
          self._target(*self._args, **self._kwargs)
      ValueError: Cannot infer dimensions without universe.
        File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/pool.py", line 108, in worker
          task = get()
        File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/queues.py", line 337, in get
          return _ForkingPickler.loads(res)
        File "/project/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/dimensions/dataId.py", line 134, in __new__
          raise ValueError(f"Cannot infer dimensions without universe.")
      ValueError: Cannot infer dimensions without universe.
      

      I think DataId is not trivially pickled and may need special serialization override.

        Attachments

          Activity

          Hide
          salnikov Andy Salnikov added a comment -

          Traceback from single process:

          Traceback (most recent call last):
            File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
              self.run()
            File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/process.py", line 93, in run
              self._target(*self._args, **self._kwargs)
            File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/pool.py", line 108, in worker
              task = get()
            File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/queues.py", line 337, in get
              return _ForkingPickler.loads(res)
            File "/project/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/dimensions/dataId.py", line 134, in __new__
              raise ValueError(f"Cannot infer dimensions without universe.")
          ValueError: Cannot infer dimensions without universe.
          

          Show
          salnikov Andy Salnikov added a comment - Traceback from single process: Traceback (most recent call last): File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/pool.py", line 108, in worker task = get() File "/software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/multiprocessing/queues.py", line 337, in get return _ForkingPickler.loads(res) File "/project/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/dimensions/dataId.py", line 134, in __new__ raise ValueError(f"Cannot infer dimensions without universe.") ValueError: Cannot infer dimensions without universe.
          Hide
          salnikov Andy Salnikov added a comment -

          Jim, can you check my quick fix for DataId pickling? Jenkins build is success.

          Show
          salnikov Andy Salnikov added a comment - Jim, can you check my quick fix for DataId pickling? Jenkins build is success.
          Hide
          jbosch Jim Bosch added a comment -

          I'm pretty sure the logic is good, but a little hard to follow; hoping we can fix that with comment.

          Show
          jbosch Jim Bosch added a comment - I'm pretty sure the logic is good, but a little hard to follow; hoping we can fix that with comment.
          Hide
          salnikov Andy Salnikov added a comment -

          Thanks for review! I extended docstring with more info. Merged and done.

          Show
          salnikov Andy Salnikov added a comment - Thanks for review! I extended docstring with more info. Merged and done.

            People

            • Assignee:
              salnikov Andy Salnikov
              Reporter:
              salnikov Andy Salnikov
              Reviewers:
              Jim Bosch
              Watchers:
              Andy Salnikov, Jim Bosch
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel