# Include ctrl_bps_parsl in lsst_bps_plugins

XMLWordPrintable

#### Details

• Type: RFC
• Status: Implemented
• Resolution: Done
• Component/s:
• Labels:
None

#### Description

ctrl_bps_parsl is a BPS plugin implemented with Parsl. Originally written by James Chiang for DESC as gen3_workflow, it has been refactored for general use, and will be adopted by the Subaru Prime Focus Spectrograph (PFS) and the Merian survey. For more details, see the package README.

I propose that ctrl_bps_parsl be included in the lsst_bps_plugins metapackage, which will make it readily available to everyone who installs lsst_distrib. The advantages are:
1. ctrl_bps_parsl is currently the only plugin that supports typical HPC clusters with Slurm/PBS/LSF schedulers. Users of the Gen2 packages ctrl_pool and pipe_drivers will find ctrl_bps_parsl to be the appropriate tool under Gen3.
2. ctrl_bps_parsl is currently the only plugin that supports operating BPS on the local machine. This allows developers to test BPS workflows without access to a cluster, and it opens the possibility of developing a simple continuous integration (CI) test suite for BPS.

One concern has to do with support, since there are currently no LSST-DM developers with first-hand experience using it. I intend to support ctrl_bps_parsl since we need it for PFS. Beyond that, I believe the code is written to LSST-DM standards (including GHA checks with black and mypy, among others), and LSST-DM developers should have little difficulty finding their way around. DESC's Computing Infrastructure team includes a Parsl developer in case knowledge from or fixes to Parsl are required.

Adding ctrl_bps_parsl to lsst_bps_plugins could be done either with or without adding the parsl package to the rubin-env conda environment. parsl is available from conda-forge. Given the possibility of using ctrl_bps_parsl for CI, I think it would be best to go ahead and add it now. This would make ctrl_bps_parsl useful without any additional effort, and therefore allow use of BPS to the regular LSST-DM user without any additional setup.

#### Activity

Hide
Eli Rykoff added a comment -

This sounds like a good idea, I know that a lot of users have wanted first-class parsl support.

As for adding parsl to rubin-env I note that this would bring in the following dependencies:

  + bcrypt 3.2.2 py310h02f21da_0 conda-forge/osx-arm64 41kB  + dill 0.3.5.1 pyhd8ed1ab_0 conda-forge/noarch 73kB  + globus-sdk 3.10.1 pyhd8ed1ab_0 conda-forge/noarch 117kB  + libsodium 1.0.18 h27ca646_1 conda-forge/osx-arm64 325kB  + paramiko 2.11.0 pyhd8ed1ab_0 conda-forge/noarch 143kB  + parsl 1.2.0 pyhd8ed1ab_0 conda-forge/noarch 274kB  + pynacl 1.5.0 py310hf8d0d8f_1 conda-forge/osx-arm64 1MB  + pyzmq 23.2.0 py310hede2015_0 conda-forge/osx-arm64 478kB  + tblib 1.7.0 pyhd8ed1ab_0 conda-forge/noarch 15kB  + typeguard 2.13.0 pyhd8ed1ab_0 conda-forge/noarch 19kB  + zeromq 4.3.4 hbdafb3b_1 conda-forge/osx-arm64 318kB 

Show
Eli Rykoff added a comment - This sounds like a good idea, I know that a lot of users have wanted first-class parsl support. As for adding parsl to rubin-env I note that this would bring in the following dependencies: + bcrypt 3.2.2 py310h02f21da_0 conda-forge/osx-arm64 41kB + dill 0.3.5.1 pyhd8ed1ab_0 conda-forge/noarch 73kB + globus-sdk 3.10.1 pyhd8ed1ab_0 conda-forge/noarch 117kB + libsodium 1.0.18 h27ca646_1 conda-forge/osx-arm64 325kB + paramiko 2.11.0 pyhd8ed1ab_0 conda-forge/noarch 143kB + parsl 1.2.0 pyhd8ed1ab_0 conda-forge/noarch 274kB + pynacl 1.5.0 py310hf8d0d8f_1 conda-forge/osx-arm64 1MB + pyzmq 23.2.0 py310hede2015_0 conda-forge/osx-arm64 478kB + tblib 1.7.0 pyhd8ed1ab_0 conda-forge/noarch 15kB + typeguard 2.13.0 pyhd8ed1ab_0 conda-forge/noarch 19kB + zeromq 4.3.4 hbdafb3b_1 conda-forge/osx-arm64 318kB
Hide
Jim Bosch added a comment -

from me. Last I heard, parsl was still in the running to be what DM developers use at USDF, and while I'm quite happy to have Paul Price, James Chiang, and other DESC people do the bulk of the maintenance and development now, I think having something like it that can target traditional HPC is sufficiently important that it'd be appropriate for DM to be willing to step into that role if others bow out.

Show
Jim Bosch added a comment - from me. Last I heard, parsl was still in the running to be what DM developers use at USDF, and while I'm quite happy to have Paul Price , James Chiang , and other DESC people do the bulk of the maintenance and development now, I think having something like it that can target traditional HPC is sufficiently important that it'd be appropriate for DM to be willing to step into that role if others bow out.
Hide
Michelle Gower added a comment -

I fully support this.

I would just like to clarify a couple things:

1. ctrl_bps_parsl is currently the only plugin that supports typical HPC clusters with Slurm/PBS/LSF schedulers. Users of the Gen2 packages ctrl_pool and pipe_drivers will find ctrl_bps_parsl to be the appropriate tool under Gen3.

HTCondor does support typical HPC clusters, but the user must submit HPC jobs to annex the nodes as part of a user condor pool.  (This introduces job walltimes that the native HTCondor pool does not have to have.)   So possible, but extra work compared to Parsl.

2. ctrl_bps_parsl is currently the only plugin that supports operating BPS on the local machine. This allows developers to test BPS workflows without access to a cluster, and it opens the possibility of developing a simple continuous integration (CI) test suite for BPS.

HTCondor does support setting up a condor instance on your local machine (including Macs) and CentOS even has rpms for it.  Again possible to do, but extra work compared to Parsl.

Show
Michelle Gower added a comment - I fully support this.     I would just like to clarify a couple things: 1. ctrl_bps_parsl is currently the only plugin that supports typical HPC clusters with Slurm/PBS/LSF schedulers. Users of the Gen2 packages ctrl_pool and pipe_drivers will find ctrl_bps_parsl to be the appropriate tool under Gen3.   HTCondor does support typical HPC clusters, but the user must submit HPC jobs to annex the nodes as part of a user condor pool.  (This introduces job walltimes that the native HTCondor pool does not have to have.)   So possible, but extra work compared to Parsl. 2. ctrl_bps_parsl is currently the only plugin that supports operating BPS on the local machine. This allows developers to test BPS workflows without access to a cluster, and it opens the possibility of developing a simple continuous integration (CI) test suite for BPS.   HTCondor does support setting up a condor instance on your local machine (including Macs) and CentOS even has rpms for it.  Again possible to do, but extra work compared to Parsl.
Hide
Tim Jenness added a comment -

DMCCB are okay with this RFC and we will also include parsl in the rubin-env. The timeline for adding it to rubin-env would be September but ctrl_bps_parsl can be added ahead of that.

Show
Tim Jenness added a comment - DMCCB are okay with this RFC and we will also include parsl in the rubin-env. The timeline for adding it to rubin-env would be September but ctrl_bps_parsl can be added ahead of that.

#### People

Assignee:
Paul Price
Reporter:
Paul Price
Watchers:
Eli Rykoff, James Chiang, Jim Bosch, Kian-Tat Lim, Michelle Gower, Mikolaj Kowalik, Paul Price, Tim Jenness
0 Vote for this issue
Watchers:
8 Start watching this issue

#### Dates

Created:
Updated:
Resolved:
Planned End:

#### Jenkins

No builds found.