Not ideal and not following RFC-775, but as a reference this is the collection of bps submit yaml files for our interim/ongoing usage of DP0.2 processing on IDF: https://github.com/lsst-dm/dp02-processing This uses the v23 stack or it release candidates.

I believe I've got something like what this ticket needs: I'm adding repos with config files to the lsst-dm repo... one for rc2 config files: https://github.com/lsst-dm/gen3_bps_rc2/tree/tickets/DM-35022

...and one for dc2 config files: https://github.com/lsst-dm/gen3_bps_dc2/tree/tickets/DM-35022

Thanks, Brock Brendal [X]. I've also got some contributed BPS configs for clustering from Paul Price over on https://github.com/lsst/drp_pipe/pull/27/files, so I'm reassigning this ticket to me, and I'll just drop some configuration snippets in drp_pipe/bps directories that can be included in other BPS configs. That will definitely include your requestMemory values, so once this is all done I'm hoping you can remove them from your templates and instead do something like

includeConfigs:

 - ${DRP_PIPE_DIR}/bps/HSC/DRP-resources.yaml

...but I'll let you know the details once it's actually done.

I'm adding a commit with the updated requestMemory as used in USDF's w_2022_36 HSC-RC2 reprocessing. Clustering isn't useable at USDF at the moment so that part wasn't used in this run.

What do we need to do for merging this ticket? Once it's merged we can use it for routine reprocessing and update as needed.

Hsin-Fang Chiang, thanks for the updates.  I was just waiting to collect input others on this ticket before putting it out for review, because I didn't trust many of the numbers I was starting with, and then it fell off my radar.  But with your additions and confirmations from much of the DRP team that they don't have anything better in hand, I think it's time to just merge this as-is and update it (especially the still-suspect DC2 numbers) as we go.

Lee Kelvin, would you mind reviewing?

Thanks all, this looks great. I've been experimenting with this clustering setup on test DECam data, and it seems to work very nicely. I have a few minor comments on the PR relating to formatting issues, but other than that, I think this looks good to me. I agree that it's probably most important to get this merged as quickly as possible, and then update it down the line as needed.

Hsin-Fang Chiang, this has been merged, but note that a USDF-specific queue value in the commit you provided (looked like it was for a particularly memory-hungry task) was removed during review, so you'll want to be sure to restore that particular BPS config setting at some other layer when using these configs.