Description
In order to support PanDA job submission in ctrl_bps four additional packages are needed from conda-forge: 3 idds- packages and panda-client.
Currently panda job submission happens on a special node curated by Sergey Padolski but that model is not sustainable. We would like to add these 4 packages to the base rubin-env to allow PanDA job submission to be done from a standard IDF notebook (shell) environment. Adding these to rubin-env directly (as opposed by adding them as afterburners in the IDF RSP deployment) seems justifiable given our assumption that we will be using PanDA as the primary workflow engine for the foreseeable future.
Attachments
Issue Links
Activity
If we primarily expect job submission to be happening from RSP instances, doesn't that mean that adding the packages just to the jupyter containers would be sufficient?
Colin Slater
added a comment - If we primarily expect job submission to be happening from RSP instances, doesn't that mean that adding the packages just to the jupyter containers would be sufficient? I expect that job submission in the future will be from development cluster nodes as well.
One question: how much extra space do these packages take?
docopt 0.6.2 py_1 conda-forge/noarch 14 KB
|
idds-client 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB
|
idds-common 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB
|
idds-doma 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 14 KB
|
idds-workflow 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 17 KB
|
panda-client 1.4.78 pyhd8ed1ab_0 conda-forge/noarch 156 KB
|
stomp.py 7.0.0 pyhd8ed1ab_0 conda-forge/noarch 35 KB
|
|
|
Summary:
|
|
|
Install: 7 packages
|
|
|
Total download: 266 KB
|
Sergey Padolski you reported that you have a problem with newer versions of curl refusing to use your panda certs. Do you have any details on that that would allow us to work out whether we need to pin curl in the env? Is there a Jira ticket for it?
The DM-CCB decided that this is OK for rubin-env at this time, but in the future when rubin-env-extras exists it will likely be moved there.
Shuwei Ye points out that we only need to add idds-doma and idds-client to the env since those two will pull in the other 3 packages. We will need v0.6.8 of esutil to fix a problem with stomp.py.