Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-792

Add PanDA packages to rubin-env

    XMLWordPrintable

Details

    • RFC
    • Status: Implemented
    • Resolution: Done
    • DM, TCT
    • None

    Description

      In order to support PanDA job submission in ctrl_bps four additional packages are needed from conda-forge: 3 idds- packages and panda-client.

      Currently panda job submission happens on a special node curated by podolsky but that model is not sustainable. We would like to add these 4 packages to the base rubin-env to allow PanDA job submission to be done from a standard IDF notebook (shell) environment. Adding these to rubin-env directly (as opposed by adding them as afterburners in the IDF RSP deployment) seems justifiable given our assumption that we will be using PanDA as the primary workflow engine for the foreseeable future.

      Attachments

        Issue Links

          Activity

            tjenness Tim Jenness added a comment - - edited

            yesw points out that we only need to add idds-doma and idds-client to the env since those two will pull in the other 3 packages. We will need v0.6.8 of esutil to fix a problem with stomp.py.

            tjenness Tim Jenness added a comment - - edited yesw points out that we only need to add idds-doma and idds-client to the env since those two will pull in the other 3 packages. We will need v0.6.8 of esutil to fix a problem with stomp.py.

            If we primarily expect job submission to be happening from RSP instances, doesn't that mean that adding the packages just to the jupyter containers would be sufficient?

            ctslater Colin Slater added a comment - If we primarily expect job submission to be happening from RSP instances, doesn't that mean that adding the packages just to the jupyter containers would be sufficient?
            ktl Kian-Tat Lim added a comment -

            I expect that job submission in the future will be from development cluster nodes as well.

            ktl Kian-Tat Lim added a comment - I expect that job submission in the future will be from development cluster nodes as well.
            ktl Kian-Tat Lim added a comment -

            One question: how much extra space do these packages take?

            ktl Kian-Tat Lim added a comment - One question: how much extra space do these packages take?
            tjenness Tim Jenness added a comment -

              docopt           0.6.2  py_1          conda-forge/noarch      14 KB
              idds-client      0.5.2  pyhd8ed1ab_0  conda-forge/noarch      15 KB
              idds-common      0.5.2  pyhd8ed1ab_0  conda-forge/noarch      15 KB
              idds-doma        0.5.2  pyhd8ed1ab_0  conda-forge/noarch      14 KB
              idds-workflow    0.5.2  pyhd8ed1ab_0  conda-forge/noarch      17 KB
              panda-client    1.4.78  pyhd8ed1ab_0  conda-forge/noarch     156 KB
              stomp.py         7.0.0  pyhd8ed1ab_0  conda-forge/noarch      35 KB
             
              Summary:
             
              Install: 7 packages
             
              Total download: 266 KB
            

            tjenness Tim Jenness added a comment - docopt 0.6.2 py_1 conda-forge/noarch 14 KB idds-client 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB idds-common 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 15 KB idds-doma 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 14 KB idds-workflow 0.5.2 pyhd8ed1ab_0 conda-forge/noarch 17 KB panda-client 1.4.78 pyhd8ed1ab_0 conda-forge/noarch 156 KB stomp.py 7.0.0 pyhd8ed1ab_0 conda-forge/noarch 35 KB   Summary:   Install: 7 packages   Total download: 266 KB
            tjenness Tim Jenness added a comment -

            podolsky you reported that you have a problem with newer versions of curl refusing to use your panda certs. Do you have any details on that that would allow us to work out whether we need to pin curl in the env? Is there a Jira ticket for it?

            tjenness Tim Jenness added a comment - podolsky you reported that you have a problem with newer versions of curl refusing to use your panda certs. Do you have any details on that that would allow us to work out whether we need to pin curl in the env? Is there a Jira ticket for it?
            ktl Kian-Tat Lim added a comment -

            The DM-CCB decided that this is OK for rubin-env at this time, but in the future when rubin-env-extras exists it will likely be moved there.

            ktl Kian-Tat Lim added a comment - The DM-CCB decided that this is OK for rubin-env at this time, but in the future when rubin-env-extras exists it will likely be moved there.

            People

              tjenness Tim Jenness
              tjenness Tim Jenness
              Colin Slater, Kian-Tat Lim, Leanne Guy, Michelle Butler [X] (Inactive), Michelle Gower, Sergey Padolski, Shuwei Ye, Tim Jenness, Wil O'Mullane, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                Jenkins

                  No builds found.