Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-871

Include ctrl_bps_parsl in lsst_bps_plugins

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      ctrl_bps_parsl is a BPS plugin implemented with Parsl. Originally written by James Chiang for DESC as gen3_workflow, it has been refactored for general use, and will be adopted by the Subaru Prime Focus Spectrograph (PFS) and the Merian survey. For more details, see the package README.

      I propose that ctrl_bps_parsl be included in the lsst_bps_plugins metapackage, which will make it readily available to everyone who installs lsst_distrib. The advantages are:
      1. ctrl_bps_parsl is currently the only plugin that supports typical HPC clusters with Slurm/PBS/LSF schedulers. Users of the Gen2 packages ctrl_pool and pipe_drivers will find ctrl_bps_parsl to be the appropriate tool under Gen3.
      2. ctrl_bps_parsl is currently the only plugin that supports operating BPS on the local machine. This allows developers to test BPS workflows without access to a cluster, and it opens the possibility of developing a simple continuous integration (CI) test suite for BPS.

      One concern has to do with support, since there are currently no LSST-DM developers with first-hand experience using it. I intend to support ctrl_bps_parsl since we need it for PFS. Beyond that, I believe the code is written to LSST-DM standards (including GHA checks with black and mypy, among others), and LSST-DM developers should have little difficulty finding their way around. DESC's Computing Infrastructure team includes a Parsl developer in case knowledge from or fixes to Parsl are required.

      Adding ctrl_bps_parsl to lsst_bps_plugins could be done either with or without adding the parsl package to the rubin-env conda environment. parsl is available from conda-forge. Given the possibility of using ctrl_bps_parsl for CI, I think it would be best to go ahead and add it now. This would make ctrl_bps_parsl useful without any additional effort, and therefore allow use of BPS to the regular LSST-DM user without any additional setup.

        Attachments

          Issue Links

            Activity

            Hide
            erykoff Eli Rykoff added a comment -

            This sounds like a good idea, I know that a lot of users have wanted first-class parsl support.

            As for adding parsl to rubin-env I note that this would bring in the following dependencies:

              + bcrypt        3.2.2  py310h02f21da_0  conda-forge/osx-arm64      41kB
              + dill        0.3.5.1  pyhd8ed1ab_0     conda-forge/noarch         73kB
              + globus-sdk   3.10.1  pyhd8ed1ab_0     conda-forge/noarch        117kB
              + libsodium    1.0.18  h27ca646_1       conda-forge/osx-arm64     325kB
              + paramiko     2.11.0  pyhd8ed1ab_0     conda-forge/noarch        143kB
              + parsl         1.2.0  pyhd8ed1ab_0     conda-forge/noarch        274kB
              + pynacl        1.5.0  py310hf8d0d8f_1  conda-forge/osx-arm64       1MB
              + pyzmq        23.2.0  py310hede2015_0  conda-forge/osx-arm64     478kB
              + tblib         1.7.0  pyhd8ed1ab_0     conda-forge/noarch         15kB
              + typeguard    2.13.0  pyhd8ed1ab_0     conda-forge/noarch         19kB
              + zeromq        4.3.4  hbdafb3b_1       conda-forge/osx-arm64     318kB
            

            Show
            erykoff Eli Rykoff added a comment - This sounds like a good idea, I know that a lot of users have wanted first-class parsl support. As for adding parsl to rubin-env I note that this would bring in the following dependencies: + bcrypt 3.2.2 py310h02f21da_0 conda-forge/osx-arm64 41kB + dill 0.3.5.1 pyhd8ed1ab_0 conda-forge/noarch 73kB + globus-sdk 3.10.1 pyhd8ed1ab_0 conda-forge/noarch 117kB + libsodium 1.0.18 h27ca646_1 conda-forge/osx-arm64 325kB + paramiko 2.11.0 pyhd8ed1ab_0 conda-forge/noarch 143kB + parsl 1.2.0 pyhd8ed1ab_0 conda-forge/noarch 274kB + pynacl 1.5.0 py310hf8d0d8f_1 conda-forge/osx-arm64 1MB + pyzmq 23.2.0 py310hede2015_0 conda-forge/osx-arm64 478kB + tblib 1.7.0 pyhd8ed1ab_0 conda-forge/noarch 15kB + typeguard 2.13.0 pyhd8ed1ab_0 conda-forge/noarch 19kB + zeromq 4.3.4 hbdafb3b_1 conda-forge/osx-arm64 318kB
            Hide
            jbosch Jim Bosch added a comment -

            from me. Last I heard, parsl was still in the running to be what DM developers use at USDF, and while I'm quite happy to have Paul Price, James Chiang, and other DESC people do the bulk of the maintenance and development now, I think having something like it that can target traditional HPC is sufficiently important that it'd be appropriate for DM to be willing to step into that role if others bow out.

            Show
            jbosch Jim Bosch added a comment - from me. Last I heard, parsl was still in the running to be what DM developers use at USDF, and while I'm quite happy to have Paul Price , James Chiang , and other DESC people do the bulk of the maintenance and development now, I think having something like it that can target traditional HPC is sufficiently important that it'd be appropriate for DM to be willing to step into that role if others bow out.
            Hide
            mgower Michelle Gower added a comment -

            I fully support this.  

             
            I would just like to clarify a couple things:

            1. ctrl_bps_parsl is currently the only plugin that supports typical HPC clusters with Slurm/PBS/LSF schedulers. Users of the Gen2 packages ctrl_pool and pipe_drivers will find ctrl_bps_parsl to be the appropriate tool under Gen3.

             
            HTCondor does support typical HPC clusters, but the user must submit HPC jobs to annex the nodes as part of a user condor pool.  (This introduces job walltimes that the native HTCondor pool does not have to have.)   So possible, but extra work compared to Parsl.

            2. ctrl_bps_parsl is currently the only plugin that supports operating BPS on the local machine. This allows developers to test BPS workflows without access to a cluster, and it opens the possibility of developing a simple continuous integration (CI) test suite for BPS.

             
            HTCondor does support setting up a condor instance on your local machine (including Macs) and CentOS even has rpms for it.  Again possible to do, but extra work compared to Parsl.

            Show
            mgower Michelle Gower added a comment - I fully support this.     I would just like to clarify a couple things: 1. ctrl_bps_parsl is currently the only plugin that supports typical HPC clusters with Slurm/PBS/LSF schedulers. Users of the Gen2 packages ctrl_pool and pipe_drivers will find ctrl_bps_parsl to be the appropriate tool under Gen3.   HTCondor does support typical HPC clusters, but the user must submit HPC jobs to annex the nodes as part of a user condor pool.  (This introduces job walltimes that the native HTCondor pool does not have to have.)   So possible, but extra work compared to Parsl. 2. ctrl_bps_parsl is currently the only plugin that supports operating BPS on the local machine. This allows developers to test BPS workflows without access to a cluster, and it opens the possibility of developing a simple continuous integration (CI) test suite for BPS.   HTCondor does support setting up a condor instance on your local machine (including Macs) and CentOS even has rpms for it.  Again possible to do, but extra work compared to Parsl.
            Hide
            tjenness Tim Jenness added a comment -

            DMCCB are okay with this RFC and we will also include parsl in the rubin-env. The timeline for adding it to rubin-env would be September but ctrl_bps_parsl can be added ahead of that.

            Show
            tjenness Tim Jenness added a comment - DMCCB are okay with this RFC and we will also include parsl in the rubin-env. The timeline for adding it to rubin-env would be September but ctrl_bps_parsl can be added ahead of that.

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Watchers:
              Eli Rykoff, James Chiang, Jim Bosch, Kian-Tat Lim, Michelle Gower, Mikolaj Kowalik, Paul Price, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.