Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-792

Add PanDA packages to rubin-env

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM, TCT
    • Labels:
      None

      Description

      In order to support PanDA job submission in ctrl_bps four additional packages are needed from conda-forge: 3 idds- packages and panda-client.

      Currently panda job submission happens on a special node curated by Sergey Padolski but that model is not sustainable. We would like to add these 4 packages to the base rubin-env to allow PanDA job submission to be done from a standard IDF notebook (shell) environment. Adding these to rubin-env directly (as opposed by adding them as afterburners in the IDF RSP deployment) seems justifiable given our assumption that we will be using PanDA as the primary workflow engine for the foreseeable future.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment - - edited

            Shuwei Ye points out that we only need to add idds-doma and idds-client to the env since those two will pull in the other 3 packages. We will need v0.6.8 of esutil to fix a problem with stomp.py.

            Show
            tjenness Tim Jenness added a comment - - edited Shuwei Ye points out that we only need to add idds-doma and idds-client to the env since those two will pull in the other 3 packages. We will need v0.6.8 of esutil to fix a problem with stomp.py.
            Hide
            ctslater Colin Slater added a comment -

            If we primarily expect job submission to be happening from RSP instances, doesn't that mean that adding the packages just to the jupyter containers would be sufficient?

            Show
            ctslater Colin Slater added a comment - If we primarily expect job submission to be happening from RSP instances, doesn't that mean that adding the packages just to the jupyter containers would be sufficient?
            Hide
            ktl Kian-Tat Lim added a comment -

            I expect that job submission in the future will be from development cluster nodes as well.

            Show
            ktl Kian-Tat Lim added a comment - I expect that job submission in the future will be from development cluster nodes as well.
            Hide
            ktl Kian-Tat Lim added a comment -

            One question: how much extra space do these packages take?

            Show
            ktl Kian-Tat Lim added a comment - One question: how much extra space do these packages take?
            Hide
            tjenness Tim Jenness added a comment -

              docopt           0.6.2  py_1          conda-forge/noarch      14 KB
              idds-client      0.5.2  pyhd8ed1ab_0  conda-forge/noarch      15 KB
              idds-common      0.5.2  pyhd8ed1ab_0  conda-forge/noarch      15 KB
              idds-doma        0.5.2  pyhd8ed1ab_0  conda-forge/noarch      14 KB
              idds-workflow    0.5.2  pyhd8ed1ab_0  conda-forge/noarch      17 KB
              panda-client    1.4.78  pyhd8ed1ab_0  conda-forge/noarch     156 KB
              stomp.py         7.0.0  pyhd8ed1ab_0  conda-forge/noarch      35 KB
             
              Summary:
             
              Install: 7 packages
             
              Total download: 266 KB
            

            Show
            tjenness Tim Jenness added a comment - docopt 0.6.2 py_1 conda-forge/noarch 14 KB idds-client 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB idds-common 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB idds-doma 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 14 KB idds-workflow 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 17 KB panda-client 1.4.78 pyhd8ed1ab_0 conda-forge/noarch 156 KB stomp.py 7.0.0 pyhd8ed1ab_0 conda-forge/noarch 35 KB   Summary:   Install: 7 packages   Total download: 266 KB
            Hide
            tjenness Tim Jenness added a comment -

            Sergey Padolski you reported that you have a problem with newer versions of curl refusing to use your panda certs. Do you have any details on that that would allow us to work out whether we need to pin curl in the env? Is there a Jira ticket for it?

            Show
            tjenness Tim Jenness added a comment - Sergey Padolski you reported that you have a problem with newer versions of curl refusing to use your panda certs. Do you have any details on that that would allow us to work out whether we need to pin curl in the env? Is there a Jira ticket for it?
            Hide
            ktl Kian-Tat Lim added a comment -

            The DM-CCB decided that this is OK for rubin-env at this time, but in the future when rubin-env-extras exists it will likely be moved there.

            Show
            ktl Kian-Tat Lim added a comment - The DM-CCB decided that this is OK for rubin-env at this time, but in the future when rubin-env-extras exists it will likely be moved there.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Colin Slater, Kian-Tat Lim, Leanne Guy, Michelle Butler [X] (Inactive), Michelle Gower, Sergey Padolski, Shuwei Ye, Tim Jenness, Wil O'Mullane, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.