Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
ctrl_bps_parsl is a BPS plugin implemented with Parsl. Originally written by James Chiang for DESC as gen3_workflow, it has been refactored for general use, and will be adopted by the Subaru Prime Focus Spectrograph (PFS) and the Merian survey. For more details, see the package README.
I propose that ctrl_bps_parsl be included in the lsst_bps_plugins metapackage, which will make it readily available to everyone who installs lsst_distrib. The advantages are:
1. ctrl_bps_parsl is currently the only plugin that supports typical HPC clusters with Slurm/PBS/LSF schedulers. Users of the Gen2 packages ctrl_pool and pipe_drivers will find ctrl_bps_parsl to be the appropriate tool under Gen3.
2. ctrl_bps_parsl is currently the only plugin that supports operating BPS on the local machine. This allows developers to test BPS workflows without access to a cluster, and it opens the possibility of developing a simple continuous integration (CI) test suite for BPS.
One concern has to do with support, since there are currently no LSST-DM developers with first-hand experience using it. I intend to support ctrl_bps_parsl since we need it for PFS. Beyond that, I believe the code is written to LSST-DM standards (including GHA checks with black and mypy, among others), and LSST-DM developers should have little difficulty finding their way around. DESC's Computing Infrastructure team includes a Parsl developer in case knowledge from or fixes to Parsl are required.
Adding ctrl_bps_parsl to lsst_bps_plugins could be done either with or without adding the parsl package to the rubin-env conda environment. parsl is available from conda-forge. Given the possibility of using ctrl_bps_parsl for CI, I think it would be best to go ahead and add it now. This would make ctrl_bps_parsl useful without any additional effort, and therefore allow use of BPS to the regular LSST-DM user without any additional setup.
This sounds like a good idea, I know that a lot of users have wanted first-class parsl support.
As for adding parsl to rubin-env I note that this would bring in the following dependencies:
+ bcrypt 3.2.2 py310h02f21da_0 conda-forge/osx-arm64 41kB
+ dill 0.3.5.1 pyhd8ed1ab_0 conda-forge/noarch 73kB
+ globus-sdk 3.10.1 pyhd8ed1ab_0 conda-forge/noarch 117kB
+ libsodium 1.0.18 h27ca646_1 conda-forge/osx-arm64 325kB
+ paramiko 2.11.0 pyhd8ed1ab_0 conda-forge/noarch 143kB
+ parsl 1.2.0 pyhd8ed1ab_0 conda-forge/noarch 274kB
+ pynacl 1.5.0 py310hf8d0d8f_1 conda-forge/osx-arm64 1MB
+ pyzmq 23.2.0 py310hede2015_0 conda-forge/osx-arm64 478kB
+ tblib 1.7.0 pyhd8ed1ab_0 conda-forge/noarch 15kB
+ typeguard 2.13.0 pyhd8ed1ab_0 conda-forge/noarch 19kB
+ zeromq 4.3.4 hbdafb3b_1 conda-forge/osx-arm64 318kB