Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16077

Fix multi-process setup for CmdLineFwk

    XMLWordPrintable

    Details

      Description

      I did not check multi-process option for some time and now it breaks:

      Traceback (most recent call last):
        File "/project/salnikov/DM-15686/pipe_supertask/bin/stac", line 25, in <module>
          sys.exit(CmdLineFwk().parseAndRun())
        File "/project/salnikov/DM-15686/pipe_supertask/python/lsst/pipe/supertask/cmdLineFwk.py", line 226, in parseAndRun
          return self.runPipeline(qgraph, butler, args)
        File "/project/salnikov/DM-15686/pipe_supertask/python/lsst/pipe/supertask/cmdLineFwk.py", line 372, in runPipeline
          mapFunc(self._executePipelineTask, target_list)
        File "/project/salnikov/DM-15686/pipe_supertask/python/lsst/pipe/supertask/cmdLineFwk.py", line 96, in __call__
          return result.get(self.timeout)
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/multiprocessing/pool.py", line 644, in get
          raise self._value
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
          put(task)
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/multiprocessing/connection.py", line 206, in send
          self._send_bytes(_ForkingPickler.dumps(obj))
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
          cls(buf, protocol).dump(obj)
      TypeError: can't pickle lsst.log.log.log.Log objects
      

      Need to fix it ASAP
       

        Attachments

          Issue Links

            Activity

            salnikov Andy Salnikov created issue -
            salnikov Andy Salnikov made changes -
            Field Original Value New Value
            Epic Link DM-14661 [ 106365 ]
            salnikov Andy Salnikov made changes -
            Link This issue is triggered by DM-15686 [ DM-15686 ]
            salnikov Andy Salnikov made changes -
            Risk Score 0
            salnikov Andy Salnikov made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Watchers Andy Salnikov, Vaikunth Thukral [ Andy Salnikov, Vaikunth Thukral ] Andy Salnikov, Jim Bosch, Tim Jenness, Vaikunth Thukral [ Andy Salnikov, Jim Bosch, Tim Jenness, Vaikunth Thukral ]
            salnikov Andy Salnikov made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            salnikov Andy Salnikov made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              salnikov Andy Salnikov
              Reviewers:
              Tim Jenness
              Watchers:
              Andy Salnikov, Jim Bosch, Tim Jenness, Vaikunth Thukral
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.