The speculations in the description indeed prove true.  The differences are down to levels so low (typically of order mean difference[stdev] ~0.001[0.02] mas for S/N>100 stars, and this is down by large factors of ~10-100 from the differences in the previous w_2021_46 runs) that I don't think they are of any scientific interest.  These remaining differences are almost certainly due to differences in the order of inputs and are not worth further investigation (noting that it will be up to the Gen3 version to provide the ability to order its inputs deterministically).  As such, I am ready to pull the plug on Gen2 support for jointcal.

For a few more details, the following is our worst case in terms of the mean Gen2/Gen3 difference in Delta(RA)*cos(Dec) for S/N>100 stars (and note I did also look at the individual Delta(RA) and Delta(Dec) but did not notice any distinctions worth any scrutiny):

This mean[stdev] difference is down from 2.80[1.15] mas in the previous w_2021_46 runs.

We also see some that look to have some stochasticity in them (but do note the 1e-7 on the y-axis scale!):

(note the stochasticity is not detector-specific, but rather they are spread throughout the focal plane).

And a handful of intra-detector gradients:

Overall, even our seemingly worst-cases are at levels so low as to be uninteresting.

As far as how the above visit-level differences propagate into coadd differences, the effects are again very small (differences typically of order mean[stdev] ~0.003[0.06] mas for S/N~>500 stars, and again down by large factors of ~10 from the differences in the previous w_2021_46 runs)

Here is our worst case scenario:

and a more typical case:

See DM-29820 if interested in the differences in the Stellar Locus plots.

Again, this looks overall, to be small enough as to be uninteresting.

With the above results, I am prepared to pull the plug on Gen2 support for jointcal, but
let me know what you think. If there are any particular cases you want looked at, let me know, or feel free to poke around yourself at:
Gen2:
https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/plots/
Gen3:
https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/vsGen3/plots/
(it is the latter that has the direct comparison plots).

Jointcal input ordering is something I wondered about from early on (see DM-9393). It's a small effect, and I could believe the difference you see here is entirely due to that. When the stars are associated across visits, the exact details of the cross matches can be slightly different, but that difference should come out by the end of the fit. Can you give me links to the logs for a couple of matched gen2/gen3 runs from this comparison? I'm curious whether the final number of stars in the fit and the final chi2 and ndof values are the same or slightly different? If the fitting process is perfect, I'd expect them to be identical (any ambiguous matches would have been removed), but HSC is now using astrometryOutlierRelativeTolerance, which means the fit ends before it's fully converged (to reduce the runtime). I'd bet that that accounts for most (all?) of the difference you see here. I don't think it's worth exploring what gen2/gen3 differences on HSC data look like with that config set to None, since it's so small, but a quick check of the final gen2/gen3 chi2 and ndof, values might be informative, as to how different jointcal thinks it is in those two cases.

This also reminds me that jointcal doesn't include a final count of the number of fittedStars/measuredStars. Those get incorporated into ndof, so it's probably not relevant most of the time.

Great, thanks John!

The logs are located on lsst-devl at:

 Gen2: /datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457/logs/jointcal/

 Gen3: /scratch/brendal4/bps-gen3-rc2/w_2022_04/submit/HSC/runs/RC2/w_2022_04/DM-33402/20220128T212035Z/jobs/jointcal/

The logs are not terribly easy to parse & compare, so as an example, I pulled out all the lines with chi2 &/or ndof in them and compare the Gen2 vs. Gen3 versions directly. So, for

/datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457/logs/jointcal/jointcal9813-HSC-I-59795.log

vs.

/scratch/brendal4/bps-gen3-rc2/w_2022_04/submit/HSC/runs/RC2/w_2022_04/DM-33402/20220128T212035Z/jobs/jointcal/9813/HSC-I/e6e6d88e-f483-4041-98ef-8390437230c7_jointcal_9813_.9754984.err 

the corresponding lines are:

Gen2: Initial chi2/ndof : 4.21833e+16/490498=8.60009e+10

Gen3: Initial chi2/ndof : 4.21833e+16/490498=8.60009e+10

Gen2: Initialize DistortionsVisit chi2/ndof : 1.87784e+08/488122=384.707

Gen3: Initialize DistortionsVisit chi2/ndof : 1.87614e+08/488122=384.358

Gen2: Initialize Distortions chi2/ndof : 9.44308e+06/487510=19.37

Gen3: Initialize Distortions chi2/ndof : 9.27373e+06/487510=19.0227

Gen2: Initialize Positions chi2/ndof : 1.96279e+06/454112=4.32226

Gen3: Initialize Positions chi2/ndof : 1.86372e+06/454112=4.10409

Gen2: Initialize DistortionsPositions chi2/ndof : 1.62282e+06/451124=3.59729

Gen3: Initialize DistortionsPositions chi2/ndof : 1.52377e+06/451124=3.37773

Gen2: Iterations stopped with chi2 cut at 81.596 and relative change of -0.0947015

Gen3: Iterations stopped with chi2 cut at 81.7825 and relative change of -0.109253

Gen2: Fit iteration 0 chi2/ndof : 803915/450176=1.78578

Gen3: Fit iteration 0 chi2/ndof : 799876/450090=1.77715

Gen2: Iterations stopped with chi2 cut at 23.7586 and relative change of -0.00395254

Gen3: Iterations stopped with chi2 cut at 23.9852 and relative change of -0.00407391

Gen2: Fit completed chi2/ndof : 611196/447240=1.36659

Gen3: Fit completed chi2/ndof : 612823/447312=1.37001

Are these differences within the range you would expect?

Differences in the initialization chi2/ndof are unsurprising (that's the ordering issue). Seeing that iterations stopped at a slightly different point during the fit I think is exactly what I'd expect (it's not following exactly the same gradient). If its stopped from fully converging, there will be some remaining difference. I don't really know what to "expect" here, but this doesn't seem unreasonable. Having of order 100 stars different (the final ndof value) is believable, I think.

Great, thanks again!  I'll update the confluence page to confirm we agree that Gen2 support for jointcal can now be considered a thing of the past!