Step1 finished with only DB connection failures. All of which were resolved upon resubmission with the dag file.

Step2 completed with no issues

Step 3 finished with DB connection issues as well, resubmission should similarly cure these. 

Step 4 failed continuously due to DB connection issues (required ~8 re-submissions). There are 33 non-DB connection issues, all in ImageDifference and all of which look like:

79004_imageDifference_36214_71.4978176.err:lsst.pex.exceptions.wrappers.InvalidParameterError:

79004_imageDifference_36214_71.4978176.err:lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (14684.8, 25724.4); no input images at that point.'

The good news is it looks like there are no longer any "No coadd PhotoCalib found" errors this time around!!

The path to these jobs is: /scratch/brendal4/bps-gen3-rc2/submit/HSC/runs/RC2/w_2021_38/DM-31795/20210922T164123Z/jobs/imageDifference

Ticket already exists for those errors: DM-31777.

Great, thanks for pointing this out, Lauren MacArthur!

Continuing to step5, I added execution_butler to the submission script in order to avoid any more DB connection problems:

includeConfigs:

- ${CTRL_BPS_DIR}/doc/lsst.ctrl.bps/execution_butler.yaml

 There were two errors, one in each of forcedPhotDiffOnDiaObjects and forcedPhotCcdOnDiaObjects. A new job, mergeExecutionButler, seemingly failed (according to bps report). However, grepping through the logs doesn't immediately uncover any errors (that I can see at least), and the mergeExecutionButler.5197569.out file says that 174079 files were transferred successfully. So, I'm not sure why bps report is claiming that this job has failed.

Path the the submission: /scratch/brendal4/bps-gen3-rc2/submit/HSC/runs/RC2/w_2021_38/DM-31795/20210928T182458Z