I couldn't tell 100% from the description, but are you mixing and matching meas_mosaic and jointcal outputs and application? Other than being insanely slow, I didn't think that was supposed to work.

What do you mean when you say "that"? I'll attach my plots with some further descriptions in the next few minutes and we can regroup.

Here are plots of the various S/N distributions of the PSF flux:

Raw instrumental:

jointcal calibrated (calibrations get applied via the persisted photoCalib dataset):

meas_mosaic using OLD-STYLE calibration (i.e. using meas_mosaic's own applyMosaicResultsExposure() function – which uses the persisted fcr dataset to apply the calibration):

meas_mosaic using the photoCalib object persisted by meas_mosaic:

As far as I'm aware, all of the above should "work" (i.e. the last two plots should look the same as they are based on the same calibration...it's just the path of persistence and application that's different). Note that the flux distributions all look ok, so the pathology in this last plot is coming from the flux errors.

In case it's of interest, for both the jointcal and meas_mosaic persisted photoCalibs, I obtain the calibrated fluxes and errors via:

photoCalib = dataRef.get("jointcal_photoCalib")

calibratedFluxAndErrArray = photoCalib.instFluxToNanojansky(catalog, baseName)

catalog[fluxKey] = calibratedFluxAndErrArray[:, 0]

catalog[fluxErrKey] = calibratedFluxAndErrArray[:, 1]

To summarize the situation: something is amiss with the implementation and/or application of the photoCalib as persisted by meas_mosaic. In applying it to calibrate a catalog (shown in the above code snippet – I do realize there is now a calibrateCatalog() function and have ticket DM-19457 to make the conversion which is not done yet, but I did confirm that I get the same results from both the above snippet and the calibrateCatalog() function for the calibrated flux and fluxErrs), the fluxes seem to come out ok, but the S/N distributions (see above plots) are completely messed up, so it is the fluxErr's that are getting computed erroneously.

The pipe_analysis scripts may be the only place in existence that is actually using the photoCalib objects persisted from meas_mosaic, and that’s just for catalog calibration (I think Yusra AlSayyad has attempted to build a coadd using meas_mosaic's photoCalib, but a long runtime led to her having to give up...)  As such, it may be that there’s no impetus to fix this — I can easily be “unblocked” by reverting back to using the old internal meas_mosaic function & datasets in pipe_analysis — but the fact that it’s broken could be a red-herring for something being fundamentally awry with a certain use flavour/pattern for/with photoCalib persistence/application that should be addressed so that potential future users do not hit it. If no other action is taken, at the very least we should cease our persistence of the photoCalib objects from meas_mosaic since they cannot be used to correctly apply the calibrations to a catalog (and slow runtimes may prohibit their use in the calibration of images for coaddition as well, but this last statement has not been fully pursued).

As a slightly unconventional review, would you both mind weighing in on this and the appropriate action to be taken (from which followup tickets can be spawned)?

This must have something to do with the implementation of FluxFitBoundedField, which is specific to meas_mosaic. Unfortunately, I think Jim Bosch is the only one who can really answer questions about that. The math in the code code in photoCalib.instFluxToNanojansky() is correct as far as I can tell, and is pretty well tested.

John Parejko, can you confirm (or reject) my recollection that we decided to propagate flux uncertainties just by scaling the measurement errors, and thus ignoring the uncertainties in the PhotoCalib itself?  I know we had an extensive discussion about how best to handle the fact that errors in the calibration are potentially highly correlated with the uncertainties of the measurements used to calibrate, but I don't remember where we landed.

I think that discussion was DM-16598, which only applied to calibrating images. When calibrating catalogs and individual sources, we still apply the photoCalib uncertainty, and I think that's the correct behavior.

Could it then be that the calibration errors are getting double-counted somehow in the meas_mosaic case?

I doubt it; I suspect the meas_mosaic errors are just being packaged up into the PhotoCalib incorrectly - that was always the most likely problem, and hearing that we are trying to propagate those errors in this context makes it even more likely in my mind.  I'll try to look into that code later this week.

Jim Bosch, if I create a follow-up ticket for this along the lines of "Decide how to handle erroneous error propagation noted in DM-19828", are you ok with me closing this one?

Yes, that's fine (and sorry for the delay).  Since this is blocked on me as it is, it probably makes sense for the blocked thing to be a ticket actually assigned to me.

Thanks Jim!  Follow-up ticket is DM-20053.  Would you mind giving this a formal "reviewed" green light?