Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_association, dax_ppdb
-
Labels:None
-
Story Points:1
-
Team:Architecture
-
Urgent?:No
Description
In DM-29530 I modified CompositeFunctor.from_file to allow it to expand environment variables in the path and changed obs_lsst and obs_subaru configs to use them rather than getPackageDir.
This was needed to allow pipelines to be executed by PanDA where the submission node has the software in a different location to the execution nodes and so getPackageDir at graph building gives a different answer to pipeline running.
It seems that I missed a change to use the environment variable and ap_association is triggering the problem.
ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 166 quanta for 30 tasks, graph ID: '1625192742.6326392-1146'
|
transformSourceTable INFO: Loading tranform functor definitions from $OBS_SUBARU_DIR/policy/Source.yaml
|
transformObjectTable INFO: Loading tranform functor definitions from $OBS_SUBARU_DIR/policy/Object.yaml
|
transformDiaSourceCat INFO: Loading tranform functor definitions from /home/spadolski/wrk/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ap_association/21.0.0-16-g11ff3f0+61acd69820/data/DiaSource.yaml
|
Traceback (most recent call last):
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+6ca1d9376e/bin/pipetask", line 29, in <module>
|
sys.exit(main())
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+6ca1d9376e/python/lsst/ctrl/mpexec/cli/pipetask.py", line 43, in main
|
return cli()
|
File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 829, in __call__
|
return self.main(*args, **kwargs)
|
File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 782, in main
|
rv = self.invoke(ctx)
|
File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
|
return _process_result(sub_ctx.command.invoke(sub_ctx))
|
File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
|
return ctx.invoke(self.callback, **ctx.params)
|
File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 610, in invoke
|
return callback(*args, **kwargs)
|
File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
|
return f(get_current_context(), *args, **kwargs)
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+6ca1d9376e/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 103, in run
|
script.run(qgraphObj=qgraph, **kwargs)
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+6ca1d9376e/python/lsst/ctrl/mpexec/cli/script/run.py", line 168, in run
|
f.runPipeline(qgraphObj, taskFactory, args)
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+6ca1d9376e/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 624, in runPipeline
|
preExecInit.initialize(graph,
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+6ca1d9376e/python/lsst/ctrl/mpexec/preExecInit.py", line 95, in initialize
|
self.saveInitOutputs(graph)
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+6ca1d9376e/python/lsst/ctrl/mpexec/preExecInit.py", line 176, in saveInitOutputs
|
task = self.taskFactory.makeTask(taskDef.taskClass,
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+6ca1d9376e/python/lsst/ctrl/mpexec/taskFactory.py", line 96, in makeTask
|
task = taskClass(config=config, initInputs=initInputs, name=label)
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ap_association/21.0.0-16-g11ff3f0+61acd69820/python/lsst/ap/association/transformDiaSourceCatalog.py", line 105, in __init__
|
super().__init__(**kwargs)
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_tasks/21.0.0-115-g99380953+42c4f6a8e4/python/lsst/pipe/tasks/postprocess.py", line 630, in __init__
|
self.funcs = CompositeFunctor.from_file(self.config.functorFile)
|
File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_tasks/21.0.0-115-g99380953+42c4f6a8e4/python/lsst/pipe/tasks/functors.py", line 534, in from_file
|
with open(filename) as f:
|
FileNotFoundError: [Errno 2] No such file or directory: '/home/spadolski/wrk/stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ap_association/21.0.0-16-g11ff3f0+61acd69820/data/DiaSource.yaml'
|
Attachments
Issue Links
Activity
For completeness, Krzysztof Findeisen noticed that I also needed to fix dax_apdb. I also needed to fix a test in ap_pipe that was checking that "ap_association" was in the config entry but now it's AP_ASSOCIATION.
Krzysztof Findeisen sorry to ask again, but would you be able to review this so that we can get PanDA working? It's a handful of lines to defer the getPackageDir in the config and use a expandvars in the execution code instead.