As a validation of this fix, I ran two Gen2 processCcd  processing of a single HSC visit (1228 which is HSC-I and in the COSMOS 9813 tract) with the latest w_2021_09 master stack, and against this change.  I also ran the same visit with Gen3.  The latter Gen2 and Gen3 visits now compare bitwise identical for all image planes (image, variance, mask) in both the icExp and calexp outputs for all 103 CCDs.  The src catalogs also compare almost identically, but for the minor differences noted in DM-28858 (based on ci_hsc_gen2 vs. ci_hsc_gen3 comparisons) and currently under investigation (but blocked by this ticket) on DM-28936.

I also compared the two Gen2 runs. While there are very small differences in the data products due to the minor differences in the variance planes (resulting in slight thresholding differences --> slightly different numbers of CRs detected and source footprint sizes --> slight differences in centroids for deblended sources, etc...), these were discussed during Princeton's regular Monday Meeting on Mar 1, 2021, and it was agreed they were indeed at too small a level to cause any concern and that this change should be made. Direct comparison plots can be perused (for the time being!) here and I've attached a few examples to the ticket for posterity (make particular note of the scales/units on these comparison plots...differences are tiny!).

Would you mind giving this a look? I think you were present at the meeting this morning where this issue was discussed (at some length!), but do let me know if you would like any further detail.

Jenkins + ci_hsc is running.

This all looks fine to me (especially when I double checked scales).  Mostly for my own curiosity, do you know if the outlier detectors in the final two focal plane plots are always different, or if the set changes depending on visit?

I don't have any concerns, so feel free to merge once Jenkins is happy.

Thanks for the speedy turnaround, Chris!  No, I don't know if the outlier detectors are always as such as I have only run the one visit.  It's a good question though, so I'll run a few more visits (and in different bands) and have a look.  It could also vary for different CALIBs, so, to start, I'll try to pick visits that will all use the same master bias and dark frames to take that element out of the equation.

So...the answer is no.  Having checked that the same CALIBs would be in play:

rootDirGen2 = "/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654"

butlerGen2 = dafPersist.Butler(rootDirGen2)

ccd = 49

dataType = "bias"

for visit in [1228, 1220, 1194, 1882,]:

    print(butlerGen2.getUri(dataType, visit=visit, ccd=ccd))

for visit in [1228, 1220, 1194, 1882,]:

    print(butlerGen2.getUri("calexp", visit=visit, ccd=ccd))

/datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits

/datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits

/datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits

/datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits

/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-I/corr/CORR-0001228-049.fits

/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-R/corr/CORR-0001220-049.fits

/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-Z/corr/CORR-0001194-049.fits

/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00823/HSC-Y/corr/CORR-0001882-049.fits

I ran those visits on the ticket branch and made the same comparison plots against the w_20201_06 RC2 run.
The same general pattern of some outlier CCDs is present, but it seems to be a different set of CCDs each time. The good news is that all differences in the above new runs were all at the same very low levels.

Jenkins was indeed happy, so merged and done!