Now running validate_drp on the higher order jointcal output.

Going to a higher order polynomial took care of the outlier problem on HSC-R 9559. I've attached the plots of this run, and pasted the only portion of of the summarize script output where jointcal is exceeds the mosaic value+N*sigma.

jointcal tracts that exceed mosaic metric for AF1
HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 3.37

HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 4.99

HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 8.12
HSC-Z 9558 : (1.55 + 2*2.914 = 7.378) < 8.12

HSC-I 9693 : (0.206 + 1*2.914 = 3.120) < 4.97

HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 4.93

HSC-Y 9559 : (0.903 + 1*2.914 = 3.817) < 4.99

In this case, only one filter+tract exceeds mosaic's value by 2 sigma, and none by 3 sigma, so I believe we've now met your acceptance criteria.

Whether we should up the default polynomial order to 7 (astrometry) and 9 (photometry) is a separate question.

Please look at the plots. I believe with these results we've now met the acceptance criteria laid out on the acceptance test page.

Looks good!  Given that the higher outlier fractions are accompanied by a consistent improvement in the repeatability, it's entirely possible that jointcal is actually doing better than meas_mosaic in these tracts (given that we don't have ground truth for which objects actually should be considered outliers).

As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

Ask and ye shall receive (an attachment)...

It looks to me like going from a 5->7th order visit polynomial clearly improves the astrometry, while going from a 7->9th order visit polynomial for photometry doesn't result in much improvement. Three tract+filter combinations show significant AF1 regression at 7th order: HSC-I 9693, HSC-Y 9559, HSC-Z 9558

Here's the full list of values where the lower order does "better" than the higher order. As you say though, we don't know truth here, so it may be that a higher outlier fraction is in fact "more correct".

7th order tracts that exceed the 5th order metric for AF1
HSC-Y 16010 1.33 < 1.47
HSC-G 8521 1.32 < 2.09
HSC-Z 9698 0.342 < 1.3
HSC-I 9693 2.45 < 4.97
HSC-I 9694 0.591 < 0.688
HSC-Z 9694 5.28 < 5.38
HSC-I 9695 1.38 < 1.53
HSC-I 9559 0.314 < 0.833
HSC-Y 9559 2.59 < 4.99
HSC-Z 9558 4.64 < 8.12

7th order tracts that exceed the 5th order metric for AF2
HSC-I 9693 0.743 < 0.811

7th order tracts that exceed the 5th order metric for PA1
HSC-R 8525 17.1 < 17.3
HSC-R 8524 16.6 < 16.7
HSC-G 9558 12.4 < 12.5

7th order tracts that exceed the 5th order metric for PF1
HSC-I 16010 42.7 < 43.0
HSC-R 9698 42.2 < 42.3
HSC-G 16009 30.1 < 30.3
HSC-R 16009 32.1 < 32.2
HSC-R 9693 33.2 < 33.4
HSC-R 9373 37.6 < 37.8
HSC-G 9374 32.6 < 32.7
HSC-R 9374 33.7 < 33.8
HSC-Y 9372 20.6 < 20.7
HSC-R 9697 39.8 < 40.1
HSC-I 15832 42.5 < 42.7
HSC-R 9371 39.7 < 40.0
HSC-R 8525 40.0 < 40.4
HSC-I 8523 42.6 < 42.7
HSC-G 8524 36.8 < 37.0
HSC-R 8524 38.3 < 38.7