I gave up on running validate_drp on your output, because the dataIds did not have the int detector field, just the string detectorName. If you could help fix that, I can do some more data exploration (validate_drp computes a number of different metrics).

Dominique Boutigny, we discussed this a bit today and that raised some questions about whether known issues with DC2 reference catalogs might be in play here:

• I know there are some problems with galaxies and extinction; are those the kind that would make the reference catalog fluxes inconsistent with the images, or just the kind that would make both unphysical but self-consistent?
• I'd like to find a way to investigate the possibility of both the single-epoch and jointcal fits locking on to the reference catalog as the source of truth and disregarding (in the case of jointcal) the extra information from multiple observations, which could happen if there are many more sources with reference catalog matches and/or lower-than-expected uncertainties in the reference catalog.  That would indicate that the only "problem" here is an unrealistically small dispersion for the single-epoch photometry, and that might be what we're seeing: 19mm is definite much larger than what we want to see for LSST, but I think it's pretty similar to what we get with meas_mosaic and the current state of calibrations for HSC (though there we know we are limited by chromatic effects in the filters that may not be in play here, and I'm not at all certain that's an apples-to-apples comparison in terms of SNR cuts).  We could investigate this by re-running with a SourceSelector for both single-epoch and jointcal that filters the reference catalog down to more realistic magnitude limits and densities.
• Is the timeline for a new run that will fix some of those issues soon enough that we should just wait for that?

All that said, I'm not thrilled about the prospect of trying to debug this by just turning knobs and hoping the results improve; it'd be much more productive to make targeted diagnostic plots (RMS relative to reference catalog, filter galaxies in and out of various plots, etc.) to actually test some of these hypotheses.  We're talking about how to get better at doing that, too (the notebook you attached is definitely a very helpful first step!).

Jim Bosch I came to the same conclusion about the reference catalog and phosim DC2 data. We can't draw any conclusions if the reference catalog is polluted by sources with fluxes that are not matching the data. Today I saw that I can easily modify loadIndexedReferenceObjects.py to exclude galaxies from the reference catalog. I will rerun jointcal with this modification as soon as I have some time to do it.

The other possibility is to run jointcal on imsim data as we know that the reference catalog is correctly matching the data.

In a general way, it would be useful to have selectors for the reference sources as we have for the detected sources. 

Today I reran joincal on imsim data (Run1.2i) and I get a very similar result as what I obtained initially on Run1.2p and the chi2/dof is = 67 for the photometry.
I really don't know how to handle this...

Lets look at this in the new year: one of the priorities for January is building some more analysis tools for jointcal's output.

Recording discussion from the DESC-DM-DC2 discussion of 2019-02-26:

• Dominique Boutigny has tried something he believes is equivalent to enabling the new use-only-stars-from-reference-catalog option added by Eli Rykoff on DM-17917, and saw no significant change.  He would like this to be checked more carefully.
• Someone from DM (I'm asking Yusra AlSayyad to coordinate, though it may not be her) should try running jointcal on the DC2 data on lsst-dev with the option from DM-17917 enabled, and then try to reproduce Dominique Boutigny's plots on the results of that processing.

Thinking about this a bit further now, it's not totally clear to me that the DM-17917 code can be used with jointcal to select only stars from the reference catalog without limiting jointcal to using only reference catalog objects entirely.  Eli Rykoff or John Parejko, can you comment on that?

I think we might have to implement DM-13053, before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket?

I think we might have to implement DM-13053, before we can easily do refcat sub-selection in jointcal. Should I up the priority on that ticket?

Yes, please.  (I'm just taking your word that it's what's needed to try what I've said want to try, but I'm happy to do that unless you think I need to understand the details).

This just came up on the DC2 phonecon.  I am not all that surprised that jointcal doesn't do better than processCcd given a noise-free reference catalogue to 23rd magnitude – all that jointcal can do is bootstrap a deeper catalogue, and there's not much need under the circumstances.   If I'm misinformed about the reference catalogue that we're using please forgive this note.

I am still very concerned about a 20mmag scatter, though.  Is this still present after all the work discussed on this ticket?

> I am still very concerned about a 20mmag scatter, though. Is this still present after all the work discussed on this ticket?

Using 7 overlapping visits of Run2.1i data, jointcal in w_2019_09, and a stars-only reference catalog, I get the same ~20mmag scatter:

So processCcd (photoCal precisely) can only do one zeropoint per ccd. In general, jointcal should be able to do higher order fits in the constrained (vs simple) models, though I'm not sure if James Chiang's recent run uses the higher order stuff (I know he was building up to getting that working).

Here's what I used:

config.photometryRejectBadFluxes=False

config.astrometryModel="constrained"

config.astrometryChipOrder=1

config.astrometryVisitOrder=3

config.photometryModel="simpleMagnitude" 

I guess I need to use "constrainedMagnitude" for the photometryModel and dial up photometryVisitOrder for which the default is 7?

Try running jointcal with the defaults: that will use the constrained model for both astrometry and photometry, with a 5th and 7th order polynomial, respectively. What version are you running?

The "simple" models are just extensions of the processCcd model (one model per ccd/visit: 2d polynomial for astrometry, one calibration factor for photometry). They are primarily useful for debugging or fitting non-mosaic camera data.

We should also keep in mind that the astrometry fit is not giving very good results either. Running `jointcal` with default configuration I get an astrometric dispersion = 11.4 mas (mag<21) to be compared to 7.9 mas with the processCcd astrometry (which is also not particularly good given an almost perfect reference catalog).
I think that we cannot exclude a problem in the reference catalog itself. I am excluding a problem in the simulation as I got similar results with phosim.  

Based on the discussion on slack, it appears to me that the reason jointcal doesn't do better than processCcd on this imsim data, is that there is nothing for it to fit beyond processCcd's model. Quoting myself from slack:

If there are no throughput variations due to flat fielding, optics or atmosphere transparency, then there's really no way for jointcal (single value per sensor+focal plane polynomial per visit) to do better than processCcd (single value per sensor/visit).

Can we close this as Invalid or Won't Fix, due to DC2 not having any photometric distortions for jointcal to model, and the known issues with the refcat?

Yes.