# Add PanDA packages to rubin-env

XMLWordPrintable

#### Details

• Type: RFC
• Status: Implemented
• Resolution: Done
• Component/s:
• Labels:
None

#### Description

In order to support PanDA job submission in ctrl_bps four additional packages are needed from conda-forge: 3 idds- packages and panda-client.

Currently panda job submission happens on a special node curated by Sergey Padolski but that model is not sustainable. We would like to add these 4 packages to the base rubin-env to allow PanDA job submission to be done from a standard IDF notebook (shell) environment. Adding these to rubin-env directly (as opposed by adding them as afterburners in the IDF RSP deployment) seems justifiable given our assumption that we will be using PanDA as the primary workflow engine for the foreseeable future.

#### Activity

Hide
Tim Jenness added a comment - - edited

Shuwei Ye points out that we only need to add idds-doma and idds-client to the env since those two will pull in the other 3 packages. We will need v0.6.8 of esutil to fix a problem with stomp.py.

Show
Tim Jenness added a comment - - edited Shuwei Ye points out that we only need to add idds-doma and idds-client to the env since those two will pull in the other 3 packages. We will need v0.6.8 of esutil to fix a problem with stomp.py.
Hide
Colin Slater added a comment -

If we primarily expect job submission to be happening from RSP instances, doesn't that mean that adding the packages just to the jupyter containers would be sufficient?

Show
Colin Slater added a comment - If we primarily expect job submission to be happening from RSP instances, doesn't that mean that adding the packages just to the jupyter containers would be sufficient?
Hide
Kian-Tat Lim added a comment -

I expect that job submission in the future will be from development cluster nodes as well.

Show
Kian-Tat Lim added a comment - I expect that job submission in the future will be from development cluster nodes as well.
Hide
Kian-Tat Lim added a comment -

One question: how much extra space do these packages take?

Show
Kian-Tat Lim added a comment - One question: how much extra space do these packages take?
Hide
Tim Jenness added a comment -

  docopt 0.6.2 py_1 conda-forge/noarch 14 KB  idds-client 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB  idds-common 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB  idds-doma 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 14 KB  idds-workflow 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 17 KB  panda-client 1.4.78 pyhd8ed1ab_0 conda-forge/noarch 156 KB  stomp.py 7.0.0 pyhd8ed1ab_0 conda-forge/noarch 35 KB    Summary:    Install: 7 packages    Total download: 266 KB 

Show
Tim Jenness added a comment - docopt 0.6.2 py_1 conda-forge/noarch 14 KB idds-client 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB idds-common 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB idds-doma 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 14 KB idds-workflow 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 17 KB panda-client 1.4.78 pyhd8ed1ab_0 conda-forge/noarch 156 KB stomp.py 7.0.0 pyhd8ed1ab_0 conda-forge/noarch 35 KB   Summary:   Install: 7 packages   Total download: 266 KB
Hide
Tim Jenness added a comment -

Sergey Padolski you reported that you have a problem with newer versions of curl refusing to use your panda certs. Do you have any details on that that would allow us to work out whether we need to pin curl in the env? Is there a Jira ticket for it?

Show
Tim Jenness added a comment - Sergey Padolski you reported that you have a problem with newer versions of curl refusing to use your panda certs. Do you have any details on that that would allow us to work out whether we need to pin curl in the env? Is there a Jira ticket for it?
Hide
Kian-Tat Lim added a comment -

The DM-CCB decided that this is OK for rubin-env at this time, but in the future when rubin-env-extras exists it will likely be moved there.

Show
Kian-Tat Lim added a comment - The DM-CCB decided that this is OK for rubin-env at this time, but in the future when rubin-env-extras exists it will likely be moved there.

#### People

Assignee:
Tim Jenness
Reporter:
Tim Jenness
Watchers:
Colin Slater, Kian-Tat Lim, Leanne Guy, Michelle Butler [X] (Inactive), Michelle Gower, Sergey Padolski, Shuwei Ye, Tim Jenness, Wil O'Mullane, Yusra AlSayyad
0 Vote for this issue
Watchers:
10 Start watching this issue

#### Dates

Created:
Updated:
Resolved:
Planned End:

#### Jenkins

No builds found.