Ok, so we really are now down to "just" the ref vs. meas offset issue?

I think so.

NOTE: The following comments apply to the w_2018_48 RC2 reprocessing which includes coadds made using both the meas_mosaic and jointcal fits.

I have just confirmed that the colorterm issue is our main contributor to the offsets noted above. The way things are currently running are actually inconsistent. The meas_mosaic and jointcal fits use the same Single Frame Processing run as input. The sfm's apply the colorterm corrections to the reference catalog magnitudes to do the initial photometric calibration. meas_mosaic also applies the colorterms when using the reference stars in its fit. jointcal does not. This puts jointcal in the position of "erasing" those corrections from the initial sfm photoCal and also, effectively, is using a mix-match of effective filters for the fitting (the reference stars are still on their own system whereas the newly added "internal" reference stars are on the observed system).

My confirmation comes from remaking my plots but not applying the colorterm to the reference stars for the plots comparing ref to src matches (they look up the colorterms essentially how sfm would, so were being applied as they were unaware that jointcal was not applying them). This brings the offset down from 88[-59] mmag to 25[-4.5] mmag in HSC-I[HSC-G].

When directly comparing the meas_mosaic and jointcal cModel mags (based on a matched catalog), the offset between them is also of order 88[-64] mmag, so the magnitude of the effect is in the right ballpark. The remaining 25[-4.5] mmag offset between the reference catalog could very well result from the relative weight of the effective mix-matching between filter systems in the jointcal fit.

Taking into account the offset, there is still evidence for increased discrepancies around bright objects. This is perhaps the remaining effect that will be resolved with DM-16598. Yusra AlSayyad, let me know if you want me to point my scripts at any particular run that incorporates that change.

My bottom line for now: the colorterms must be applied to the reference catalog mags in jointcal. I also believe the work for this has been done (but is perhaps a bit rotted...heh, heh) by Dominique Boutigny on DM-13054 in these commits: https://github.com/lsst/jointcal/commit/e1d0292c71275315a6b17b67ecaf47acd2ffbee8
https://github.com/lsst/jointcal/commit/05f60a9a2c9b86a33487cccddd9b0640ca90faf1
Given the significant magnitude of the effect, I believe this should block any further QA efforts (but I would, of course, be happy to pick them up along with the functionality being implemented). Hopefully the recent decision to Removing from current sprint per discussion of 2018-12-19. can be amended?

Let me know if you think the work to date on this ticket covers its scope and can be closed out.  If so, I would expect we would add further similar tickets following decisions and outcomes based on my recommendations here.

The post DM-16598 rerun is in: https://lsst-web.ncsa.illinois.edu/~yusra/RC_QA/w_2018_48_jointcal_noScaleVar_redo/ --> /datasets/hsc/repo/rerun/private/yusra/RC2/w_2018_48_jointcal_noScaleVar_redo/

Yes, we can reassess after colorterms have been applied. On DM-16598, I just have to update pipe_tasks, run jenkins and that ticket should be good to merge.