boutigny, we discussed this a bit today and that raised some questions about whether known issues with DC2 reference catalogs might be in play here:
- I know there are some problems with galaxies and extinction; are those the kind that would make the reference catalog fluxes inconsistent with the images, or just the kind that would make both unphysical but self-consistent?
- I'd like to find a way to investigate the possibility of both the single-epoch and jointcal fits locking on to the reference catalog as the source of truth and disregarding (in the case of jointcal) the extra information from multiple observations, which could happen if there are many more sources with reference catalog matches and/or lower-than-expected uncertainties in the reference catalog. That would indicate that the only "problem" here is an unrealistically small dispersion for the single-epoch photometry, and that might be what we're seeing: 19mm is definite much larger than what we want to see for LSST, but I think it's pretty similar to what we get with meas_mosaic and the current state of calibrations for HSC (though there we know we are limited by chromatic effects in the filters that may not be in play here, and I'm not at all certain that's an apples-to-apples comparison in terms of SNR cuts). We could investigate this by re-running with a SourceSelector for both single-epoch and jointcal that filters the reference catalog down to more realistic magnitude limits and densities.
- Is the timeline for a new run that will fix some of those issues soon enough that we should just wait for that?
All that said, I'm not thrilled about the prospect of trying to debug this by just turning knobs and hoping the results improve; it'd be much more productive to make targeted diagnostic plots (RMS relative to reference catalog, filter galaxies in and out of various plots, etc.) to actually test some of these hypotheses. We're talking about how to get better at doing that, too (the notebook you attached is definitely a very helpful first step!).