To facilitate this comparison, I have "hacked-up" the compareVisitAnalysis.py script in pipe_analysis to be able to read in datasets from both gen2 and gen3 output repositories. As a first check on the fgcm photoCalib objects persisted by the two middlewares for a single visit (1228 COSMOS HSC-I), the following are plots directly comparing the gen2 vs. gen3 src catalogs having applied the associated fgcm results for each (having confirmed that the non-corrected ones compare identically and all non-photometric quantities compare identically):

As can be seen, there are differences (while at a very low level...too low to care? Answer may hinge on how different the coadds look...stay tuned for that!)

The attached file also includes direct and percent difference between the values returned by the getCalibrationMean(), getCalibrationErr(), and getInstFluxAtZeroMagnitude() functions on the respective photoCalib objects for each ccd in the visit.

Given the pre-fgcm issues discovered in DM-29818 (and resolved in DM-29881 & DM-30030), I have updated the subject of this comparison to be based on the more recent w_2021_22 RC2 runs (from those of  w_2021_14).

We have now essentially reached parity between gen2 & gen3 SFM, with the small caveat that the gen3 w_2021_22 run had 17 failed quanta (see DM-30647 for full details).  As such, it's time to move on to the post-SFM comparisons...this being one of them!  Unfortunately, despite effective SFM parity, there is still a difference between the gen2 & gen3 fgcm results.  The same visit above shows similar levels of difference, but with a different pattern in the w_2021_22 runs (a WIDE visit comparison shows similar levels, as do visits in other bands):

Seeing as I'm totally unfamiliar with the inner workings of fgcm, I fear I must reach out to the expert (singular!) on this one as to whether they have any insights into possible causes for (very small!) differences in the gen2 vs. gen3 fgcm solutions. So, here's looking at you Eli Rykoff! You've mentioned in the past that the fgcm is inherently non-deterministic, so parity in this case may not be achievable (although I thought there was a config setting turned on to make things deterministic for this parity testing). If such is the case (and it has been deemed an "acceptable" situation), then we may want to consider making coadds in both gen2 and gen3 but leaving out the fgcm calibrations so that we won't be second-guessing any downstream differences as "possibly" being rooted in residual fgcm effects.

Okay, a very quick comment here. To be clear, the (slightly) non-deterministic part of fgcm is that there is a randomized selection of reserved stars that are not used in the fits. We are forcing a random seed which will give repeatable results given identical inputs.

In this case, I just checked and due to either processing failures or missing raws (the common refrain...) there were 432 visits in the Gen2 calibration and 415 visits in the Gen3 calibration. Therefore, even with the same random seed it is not reasonable to expect that the two fgcmcal runs will arrive the same solution.

I do want to highlight that when we do get the processing/inputs squared away I am not 100% sure that the inputs will be given to fgcm in the same order, or that the input files will be sorted the same way (especially the parquet tables). Therefore, I'm not 100% sure that using the same random seed will result in the same selection of reserved stars.

Oh, this is HSC, not something to do with missing raws...

Nope, but it is all about the abandoned visits due to single ccd failures (the 17 failed quanta noted on DM-30365).  It looks promising that these were all due to the HSM shape bug that has since been fixed, so the w_2021_26 run (happening now) will be free from this issue.  As such, I propose we punt once again until I’ve had a look at those results.

I have looked at the results of the latest w_2022_04 runs (Gen2: DM-33457, Gen3: DM-33402). The TL;DR is that I think that, while we don't have parity, we have reached the point where the differences are small enough (and expected given known areas that just can't be synced), that we can safely pull the plug on Gen2 for FGCM.

For a few more details, having again confirmed parity at the visit-level without before applying the external calibrations, I made direct comparisons of the two runs with the calibrations applied (as above). Our worst-case in terms of a mean offset is this one:

The following was the worst in terms of std:

But many looked really close indeed:

And then there are some with slightly funkier patterns...

I think it is expected that the HSC-G filter seems to be the least well-behaved and some have order 10mmag offsets between Gen2/Gen3 (as in the first example above). If Eli Rykoff is not worried about this, neither am I!

As for how this propagates to differences in the coadds, this is our worst case:


and our best case is tract 9615 HSC-Z with mean=0.04, stdev=0.99!

Finally, the stellar loci look quite similar:
Gen2:

vs. Gen3:

All-in-all, I say we are good to pull the plug on Gen2.

Let me know what you think.  And if there are any particular cases you want looked at, let me know, or poke around yourself at:
Gen2:
https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/plots/

Gen3:
https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/vsGen3/plots/

(it is the latter that has the direct comparison plots).

Oh ugh, I do not like that large offset in 9796/34384 g-band that you posted.

Can you point me to the specific repos where these were processed?

Yup (tickets listed above).
Gen2: /datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457
Gen3 collection: HSC/runs/RC2/w_2022_04/DM-33402

Would you like a list of more the worst cases?

E.g.
'visit': 34422, 'filter': 'HSC-G', 'tract': 9697, Run Comparison: CircApRad12pix mag diff (mmag):

'visit': 34342, 'filter': 'HSC-G', 'tract': 9697, Run Comparison: CircApRad12pix mag diff (mmag):

'visit': 34400, 'filter': 'HSC-G', 'tract': 9697, Run Comparison: CircApRad12pix mag diff (mmag):

Oh, and FYI, the big offset is "new", as in it was much smaller (almost zero, in fact) in the previous run:
https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2021_46/vsGen3/plots/HSC-G/tract-9697/visit-34384/compareVisit-v34384-diff_base_CircularApertureFlux_12_0-psfMagHist.png

Bad news and good news.

I did a comparison between the fgcm outputs for the standard stars on gen2/gen3 on w_2022_04 and it looks like complete garbage:

But I also did the same comparison between the standard stars on gen3 w_2021_50 and gen3 w_2022_04 and they look just fine:

So whatever has gone terribly wrong seems to have gone wrong on the gen2 side. I do not think it is worth keeping gen2 around to investigate what has gone wrong on that end. (Hopefully Lauren MacArthur agrees.). However, it is vital that we monitor the statistics from run to run in gen3 in case there is some transient issue that just happened to hit in gen2. On the other hand, I think it's more likely that I inadvertently broke something on the gen2 side (not that there's any smoking gun, it's just that debugging gen2 does not seem like the best use of time.)

Lauren MacArthur agrees

I guess this is reviewed then?

I agree again!  Thanks, Eli.  I will update the confluence page to note our collective assessment.