Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36920

Include htcondor python bindings from conda-forge into 'stack'

    XMLWordPrintable

    Details

    • Team:
      Data Facility
    • Urgent?:
      No

      Description

      The BPS htcondor plugin will require the htcondor python bindings to function at S3DF (and elsewhere) . From examining
      https://anaconda.org/conda-forge/htcondor
      https://anaconda.org/conda-forge/htcondor/files
      it appears that a suitable distribution for Python 3.10 should be available.
      If this looks appropriate, requesting that the htcondor python bindings be included in 'the stack'.

        Attachments

          Activity

          Hide
          ktl Kian-Tat Lim added a comment -

          This should really be an RFC, but I will try to get it into rubin-env 5.0.0 anyway.

          My understanding is that this needs to be part of the user's conda environment for both submission and execution and as a result needs to be in the base rubin-env environment rather than rubin-env-rsp or rubin-env-developer. Is that correct?

          Show
          ktl Kian-Tat Lim added a comment - This should really be an RFC, but I will try to get it into rubin-env 5.0.0 anyway. My understanding is that this needs to be part of the user's conda environment for both submission and execution and as a result needs to be in the base rubin-env environment rather than rubin-env-rsp or rubin-env-developer. Is that correct?
          Hide
          gdaues Greg Daues added a comment -

          I do not think that the execute nodes actually need access to the htcondor python bindings from conda-forge.

          If the bindings are in the base rubin-env environment, that would help in some broad cases, like users running at other sites conveniently having them in place, or more exotic scaling scenarios where parts of workflow run on a compute node. But for the initial base case of running at s3df, I do not think access on the execute nodes is needed.

          Show
          gdaues Greg Daues added a comment - I do not think that the execute nodes actually need access to the htcondor python bindings from conda-forge. If the bindings are in the base rubin-env environment, that would help in some broad cases, like users running at other sites conveniently having them in place, or more exotic scaling scenarios where parts of workflow run on a compute node. But for the initial base case of running at s3df, I do not think access on the execute nodes is needed.
          Hide
          gdaues Greg Daues added a comment -

          This seems to be done.   Perhaps we can mark this ticket as done/complete. 

          [daues@sdfrome001 ~]$ source /sdf/group/rubin/sw/loadLSST.bash
          (lsst-scipipe-5.1.0) [daues@sdfrome001 ~]$ setup lsst_distrib -t w_2023_05
          (lsst-scipipe-5.1.0) [daues@sdfrome001 ~]$ python3
          Python 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:26:04) [GCC 10.4.0] on linux
          Type "help", "copyright", "credits" or "license" for more information.
          >>> import htcondor
          >>> 

          Show
          gdaues Greg Daues added a comment - This seems to be done.   Perhaps we can mark this ticket as done/complete.  [daues @sdfrome001 ~]$ source /sdf/group/rubin/sw/loadLSST.bash (lsst-scipipe- 5.1 . 0 ) [daues @sdfrome001 ~]$ setup lsst_distrib -t w_2023_05 (lsst-scipipe- 5.1 . 0 ) [daues @sdfrome001 ~]$ python3 Python 3.10 . 8 | packaged by conda-forge | (main, Nov 22 2022 , 08 : 26 : 04 ) [GCC 10.4 . 0 ] on linux Type "help" , "copyright" , "credits" or "license" for more information. >>> import htcondor >>>

            People

            Assignee:
            ktl Kian-Tat Lim
            Reporter:
            gdaues Greg Daues
            Watchers:
            Greg Daues, Kian-Tat Lim
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.