# Run jointcal comparison with higher order polynomial

XMLWordPrintable

#### Details

• Story
• Status: Done
• Resolution: Done
• None
• 1
• AP F18-4

#### Description

To see whether it improves the outlier rejection fraction, I'm going to increase the visit polynomial order by 2 (from 5 to 7) and see if that helps. It's just CPU time: don't have to make any real modifications to the execution scripts.

#### Attachments

1. DM-15713-order-compare.tar.gz
578 kB
2. DM-15713-performance.tar.gz
2.13 MB

#### Activity

John Parejko added a comment -

Now running validate_drp on the higher order jointcal output.

John Parejko added a comment - Now running validate_drp on the higher order jointcal output.

Going to a higher order polynomial took care of the outlier problem on HSC-R 9559. I've attached the plots of this run, and pasted the only portion of of the summarize script output where jointcal is exceeds the mosaic value+N*sigma.

jointcal tracts that exceed mosaic metric for AF1
HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 3.37

HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 4.99

HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 8.12
HSC-Z 9558 : (1.55 + 2*2.914 = 7.378) < 8.12

HSC-I 9693 : (0.206 + 1*2.914 = 3.120) < 4.97

HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 4.93

HSC-Y 9559 : (0.903 + 1*2.914 = 3.817) < 4.99

In this case, only one filter+tract exceeds mosaic's value by 2 sigma, and none by 3 sigma, so I believe we've now met your acceptance criteria.

Whether we should up the default polynomial order to 7 (astrometry) and 9 (photometry) is a separate question.

John Parejko added a comment - Going to a higher order polynomial took care of the outlier problem on HSC-R 9559. I've attached the plots of this run, and pasted the only portion of of the summarize script output where jointcal is exceeds the mosaic value+N*sigma. jointcal tracts that exceed mosaic metric for AF1 HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 3.37 HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 4.99 HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 8.12 HSC-Z 9558 : (1.55 + 2*2.914 = 7.378) < 8.12 HSC-I 9693 : (0.206 + 1*2.914 = 3.120) < 4.97 HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 4.93 HSC-Y 9559 : (0.903 + 1*2.914 = 3.817) < 4.99 In this case, only one filter+tract exceeds mosaic's value by 2 sigma, and none by 3 sigma, so I believe we've now met your acceptance criteria. Whether we should up the default polynomial order to 7 (astrometry) and 9 (photometry) is a separate question.

Please look at the plots. I believe with these results we've now met the acceptance criteria laid out on the acceptance test page.

John Parejko added a comment - Please look at the plots. I believe with these results we've now met the acceptance criteria laid out on the acceptance test page .
Jim Bosch added a comment -

Looks good!  Given that the higher outlier fractions are accompanied by a consistent improvement in the repeatability, it's entirely possible that jointcal is actually doing better than meas_mosaic in these tracts (given that we don't have ground truth for which objects actually should be considered outliers).

As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

Jim Bosch added a comment - Looks good!  Given that the higher outlier fractions are accompanied by a consistent improvement in the repeatability, it's entirely possible that jointcal is actually doing better than meas_mosaic in these tracts (given that we don't have ground truth for which objects actually should be considered outliers). As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

It looks to me like going from a 5->7th order visit polynomial clearly improves the astrometry, while going from a 7->9th order visit polynomial for photometry doesn't result in much improvement. Three tract+filter combinations show significant AF1 regression at 7th order: HSC-I 9693, HSC-Y 9559, HSC-Z 9558

Here's the full list of values where the lower order does "better" than the higher order. As you say though, we don't know truth here, so it may be that a higher outlier fraction is in fact "more correct".

7th order tracts that exceed the 5th order metric for AF1
HSC-Y 16010 1.33 < 1.47
HSC-G 8521 1.32 < 2.09
HSC-Z 9698 0.342 < 1.3
HSC-I 9693 2.45 < 4.97
HSC-I 9694 0.591 < 0.688
HSC-Z 9694 5.28 < 5.38
HSC-I 9695 1.38 < 1.53
HSC-I 9559 0.314 < 0.833
HSC-Y 9559 2.59 < 4.99
HSC-Z 9558 4.64 < 8.12

7th order tracts that exceed the 5th order metric for AF2
HSC-I 9693 0.743 < 0.811

7th order tracts that exceed the 5th order metric for PA1
HSC-R 8525 17.1 < 17.3
HSC-R 8524 16.6 < 16.7
HSC-G 9558 12.4 < 12.5

7th order tracts that exceed the 5th order metric for PF1
HSC-I 16010 42.7 < 43.0
HSC-R 9698 42.2 < 42.3
HSC-G 16009 30.1 < 30.3
HSC-R 16009 32.1 < 32.2
HSC-R 9693 33.2 < 33.4
HSC-R 9373 37.6 < 37.8
HSC-G 9374 32.6 < 32.7
HSC-R 9374 33.7 < 33.8
HSC-Y 9372 20.6 < 20.7
HSC-R 9697 39.8 < 40.1
HSC-I 15832 42.5 < 42.7
HSC-R 9371 39.7 < 40.0
HSC-R 8525 40.0 < 40.4
HSC-I 8523 42.6 < 42.7
HSC-G 8524 36.8 < 37.0
HSC-R 8524 38.3 < 38.7

John Parejko added a comment - As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"? Ask and ye shall receive (an attachment)... It looks to me like going from a 5->7th order visit polynomial clearly improves the astrometry, while going from a 7->9th order visit polynomial for photometry doesn't result in much improvement. Three tract+filter combinations show significant AF1 regression at 7th order: HSC-I 9693, HSC-Y 9559, HSC-Z 9558 Here's the full list of values where the lower order does "better" than the higher order. As you say though, we don't know truth here, so it may be that a higher outlier fraction is in fact "more correct". 7th order tracts that exceed the 5th order metric for AF1 HSC-Y 16010 1.33 < 1.47 HSC-G 8521 1.32 < 2.09 HSC-Z 9698 0.342 < 1.3 HSC-I 9693 2.45 < 4.97 HSC-I 9694 0.591 < 0.688 HSC-Z 9694 5.28 < 5.38 HSC-I 9695 1.38 < 1.53 HSC-I 9559 0.314 < 0.833 HSC-Y 9559 2.59 < 4.99 HSC-Z 9558 4.64 < 8.12 7th order tracts that exceed the 5th order metric for AF2 HSC-I 9693 0.743 < 0.811 7th order tracts that exceed the 5th order metric for PA1 HSC-R 8525 17.1 < 17.3 HSC-R 8524 16.6 < 16.7 HSC-G 9558 12.4 < 12.5 7th order tracts that exceed the 5th order metric for PF1 HSC-I 16010 42.7 < 43.0 HSC-R 9698 42.2 < 42.3 HSC-G 16009 30.1 < 30.3 HSC-R 16009 32.1 < 32.2 HSC-R 9693 33.2 < 33.4 HSC-R 9373 37.6 < 37.8 HSC-G 9374 32.6 < 32.7 HSC-R 9374 33.7 < 33.8 HSC-Y 9372 20.6 < 20.7 HSC-R 9697 39.8 < 40.1 HSC-I 15832 42.5 < 42.7 HSC-R 9371 39.7 < 40.0 HSC-R 8525 40.0 < 40.4 HSC-I 8523 42.6 < 42.7 HSC-G 8524 36.8 < 37.0 HSC-R 8524 38.3 < 38.7

#### People

John Parejko
John Parejko
Jim Bosch
Jim Bosch, John Parejko, John Swinbank