Note to self: Keep DM-28806 in mind while becoming more familiar with the structure and execution of these scripts.

I have made some detailed comparisons of the outputs from the two ci_hsc repos based on w_2020_08. This includes comparisons of the following for each visit/ccd combination for both the ic and calexp versions of the following:

• image arrays (image, variance, mask planes)
• photoCalib objects
• PSFs
• WCSs
• every column in the source tables

I think the main conclusion is that things look pretty consistent!

Here is a quick summary:

All images and (central CCD position) PSFs are bitwise identical for both ic and calexp.

Differences appear in some of the photoCalib objects associated with the calexp objects, e.g.

Comparing gen2 vs. gen3 calexps for HSC-R 903334 16:

photoCalib diff: calMean: 3.339e-08 calErr: 1.086e-11 instFluxAtMag0: -2.502e+04

Differences appear in the calexp WCSs, e.g.:

gen2 vs. gen3 calexp WCSs for HSC-R 903334  16: Pixel Scale Diff -0.0582 (mas)

Consequently, differences appear in the calexp src coords, e.g.

Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 16:

coord_ra   Absolute diff (mas): mean: 2.273  min: 0.001  max 25.897

coord_dec  Absolute diff (mas): mean: 1.596  min: 0.000  max 12.967

All deblend_peakId values in the calexp src catalogs that are not 0 are offset by a constant integer, e.g:

Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 16:

deblend_peakId id number offset: 284224

Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 22:

deblend_peakId id number offset: 196013

The actual sources "used" in astrom/calib numbers can differ by small(ish) numbers ("total" here is the total number of sources in the catalog, the number of actual calib "used" sources ranges from ~110 - 280), e.g.

Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 16:

calib_astrometry_used Number of differences: 12 of 1337 total

calib_photometry_used Number of differences:  4 of 1337 total

Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 22:

calib_astrometry_used Number of differences:  3 of 1256 total

The above differences likely explain the WCS differences.

I'm not sure if it's of interest, but the ordering of schema.getOrderedNames() in the icSrc catalogs differs from gen2 to gen3 (are the plugins not being executed in a deterministic order in ISR as they are for SFP?). The set s are the same, though.

For (much) more detail, I attach a rendering of the notebook I wrote to make these comparisons which shows the full output for all visit/ccd comparisons.

I'm throwing this one to you, Jim, but feel free to redirect if appropriate! I am interested in the thresholds/tolerances for the differences here and if they require follow-up. My initial instinct is that they are, for the most part, quite small and mostly explained by small differences in the sources that get used in the astrometric and photometric calibrations. I suspect we should be able to make these consistent in both cases(?). If that is desired, please let me know and I'd be happy to look into that (here or on another ticket).

Thanks for the detailed investigation, and I'm very glad to see that the results are already so close (and especially that the images are already identical).

Matching to the reference catalog is something I expect to produce slightly different results, because it's a very different code path to determine which initial reference catalog objects are provided to the Tasks. Since we're already so close to being exactly identical in other respects, I'm tempted to try to see if we can make that difference go away as well, and make our coadds identical, too. I suspect the order of the reference objects passed in is at least different, so deterministic sorting on both sides might help. And it's probable that the Gen3 reference objects being passed in are a superset of those Gen2 is getting; if that's the case, we ought to be able to make the Gen3 code path do the exact same down-selection Gen2 is doing, and then we'll really be in business. Would you mind opening a new ticket to look into that, and taking it on? Might involve getting in there with a debugger to see exactly how the reference catalog objects differ, but I think it'll be worth it if we can make the coadd images the same (or at least give Jointcal and FGCM the exact same inputs).

I'm not quite sure what's going on with the deblend_peakId values; that's a new field that I'm not super familiar with yet, and I know the logic that generates the visit+detector integer IDs that gets mangled into many of our source IDs hasn't settled down yet. I'm working on my own tickets to deal with the latter, so let's revisit this when those are done. In any case, a constant integer offset is an easy difference to deal with even if it stays around and propagates through.

The schema ordering difference is one I don't understand. The plugins are sorted by name using the same (subtask) code in Gen2 and Gen3, and it's the same code in CharacterizeImageTask and CalibrateTask. So I'm guessing the ordering is related to when fields from one or more other subtasks are added (propagated flags? stuff from astrometry or deblending?) relative to each other or the measurement plugins. Could you look at icSrcSchema.getOrderedNames() again to see if you can spot any blocks of names that have been swapped, reflecting a task-initialization reordering that we might be able to revert?

Thanks for the review Jim. I've created DM-28936 to try to sync up the calibration source selections and will get started on that soon. I've also looked more deeply into the schema ordering issue. It was stemming from the aperture correction "sets" getting populated in different orders, e.g.:
Gen2:

_ApCorrNameSet =  {'ext_convolved_ConvolvedFlux_2_kron', 'modelfit_CModel_initial', 'ext_convolved_ConvolvedFlux_1_6_0', 'ext_convolved_ConvolvedFlux_2_3_3', 'ext_convolved_ConvolvedFlux_1_4_5', 'ext_convolved_ConvolvedFlux_0_kron', 'ext_convolved_ConvolvedFlux_2_4_5', 'ext_convolved_ConvolvedFlux_0_6_0', 'ext_convolved_ConvolvedFlux_3_4_5', 'base_GaussianFlux', 'modelfit_CModel_exp', 'ext_photometryKron_KronFlux', 'ext_convolved_ConvolvedFlux_2_6_0', 'ext_convolved_ConvolvedFlux_1_3_3', 'ext_convolved_ConvolvedFlux_3_3_3', 'ext_convolved_ConvolvedFlux_3_kron', 'ext_convolved_ConvolvedFlux_3_6_0', 'ext_convolved_ConvolvedFlux_0_4_5', 'base_PsfFlux', 'modelfit_CModel_dev', 'ext_convolved_ConvolvedFlux_1_kron', 'modelfit_CModel', 'ext_convolved_ConvolvedFlux_0_3_3'}

Gen3:

_ApCorrNameSet =  {'ext_photometryKron_KronFlux', 'ext_convolved_ConvolvedFlux_0_6_0', 'ext_convolved_ConvolvedFlux_3_4_5', 'ext_convolved_ConvolvedFlux_2_4_5', 'ext_convolved_ConvolvedFlux_2_3_3', 'ext_convolved_ConvolvedFlux_0_kron', 'modelfit_CModel_dev', 'modelfit_CModel_exp', 'ext_convolved_ConvolvedFlux_0_4_5', 'ext_convolved_ConvolvedFlux_3_6_0', 'modelfit_CModel_initial', 'ext_convolved_ConvolvedFlux_1_6_0', 'ext_convolved_ConvolvedFlux_3_3_3', 'base_GaussianFlux', 'ext_convolved_ConvolvedFlux_0_3_3', 'ext_convolved_ConvolvedFlux_1_4_5', 'ext_convolved_ConvolvedFlux_2_6_0', 'ext_convolved_ConvolvedFlux_2_kron', 'ext_convolved_ConvolvedFlux_3_kron', 'modelfit_CModel', 'ext_convolved_ConvolvedFlux_1_3_3', 'base_PsfFlux', 'ext_convolved_ConvolvedFlux_1_kron'}

I was puzzled having seen your commit to make the apCorr ordering deterministic here.
However, I then found that the same sorting is not being done where the apcorr_{name}_used columns get set here.
This also explains why this only turned up in the icSrc catalogs as these columns do not get added to the src catalogs.

As a fix, I've pushed a meas_algorithms branch to add the sorting. I also tried to add a "best effort" to the unittest along the lines of what you did in meas_base. Could you give this PR a review as well? I reran the two ci_hsc s against this branch and can confirm that I am now seeing the same ordering of the gen2 and gen3 icSrc schemas.  I've also got a Jenkins + ci_hsc running, but it seems that is destined to fail on ci_hsc until an unrelated bug gets fixed (I'll run it again when that happens).

In addressing your review comments, I've added another PR just to keep things consistent (sorry!) I've also kicked off another Jenkins + ci_hsc which I think has a shot at passing this time.

Thanks again Jim. Jenkins passed, so both branches have been merged to master.