Running jointcal master on DM-13666 output now.

jointcal run completed over the weekend, all final chi2s were reasonable. I've modified my validate_drp execution scripts now, and have submitted those jobs. Will post here when finished.

Validate_drp output complete (had to rerun a few of them due to out of memory problems running 5 per node), and summarized in the attached tarball. I modified the plotting script to make three sets of plots: single->jointcal, single->mosaic, mosaic->jointcal, to help with the overall comparison, and I tried to keep the plot limits easily comparable between the two jointcal plots so you can easily flip between them.

According to validate_drp, only one jointcal astrometric result on these data exceeded an outlier metric (AF1 on HSC-R 9559), and none exceeded the repeatability metrics. For this run, I used the config defaults, which means the visit polynomial is 5th order for astrometry and 7th order for photometry: is it worth running with higher order for either of those to see if we can improve further?

I'll try to compute the final summary statistics described on the acceptance test page. John Swinbank: should I make a ticket to do that work on?

I've put this "in review": Jim Bosch, please look at the attached plots and tables. It's up to you whether you want to review the changes I made to the slurm and plotting scripts on the associated PR to jointcal_compare.

is it worth running with higher order for either of those to see if we can improve further?

I think I'd say "not at this time"; I don't think this needs to block jointcal acceptance, and I'd like to minimize those blockers.  Please create a ticket to do this in the future, but unless we have a reason to move it up, let's aim to do it after we try to include the Jacobian as a fixed term in the photometric fit, as I would definitely expect that to affect the desired polynomial order for photometry (and perhaps indirectly the order for astrometry).

've put this "in review": Jim Bosch, please look at the attached plots and tables. It's up to you whether you want to review the changes I made to the slurm and plotting scripts on the associated PR to jointcal_compare

Plots look good.  The high outlier fraction in all of the photometric metric measurements makes me worry a bit about the reliability of those metrics, but there are also many potential explanations for that that wouldn't be a source of worry.  Do you (or Michael Wood-Vasey?) have any guesses as to what kinds of objects we're throwing away in those?

I have not looked at the slurm or plotting scripts, and don't think I need to.

Looking at the validate_drp output (which lives at /project/parejkoj/DM-11783/DM-15617), it seems apparent that there are a lot more points included in the comparison for jointcal, vs. meas_mosaic. I don't know what this implies, although it might be related to the fact that jointcal saves all of its ccds, whereas I think meas_mosaic just saves the ccds that land on that tract. Also, the structure of some of the plots looks different, but I'm not sure how to interpret that.

HSC-R 9559 does show a small tail in the AM1/AF1 histogram for jointcal (hence the large AF1 there), but I don't yet know why. The other plots don't appear to be odd.

FYI, here's an rsync command to pull down all the validate_drp plots into your current directory, organized by calibration and tract:

rsync -arv --exclude="*.json" -e"ssh" lsst-dev:/project/parejkoj/DM-11783/DM-15617/validate-* .

Pasting the final summary statistics here from DM-15695, for easier reference (since they applied to this ticket's run):

jointcal tracts that exceed mosaic metric for AF1
HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64

HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34

HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5

HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46

HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15

HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85

HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83

HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4
HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4
HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4

jointcal tracts that exceed mosaic metric for AF2
HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23

HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76

HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0

HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83