Measure jointcal acceptance statistics

XMLWordPrintable

Details

• Story
• Status: Done
• Resolution: Done
• None
• 2
• AP F18-4

Description

Now that we have reliable output from jointcal+validate_drp, we need to compute the final summary statistics described on the acceptance test page. This is probably most easily added as a final step of jointcal_compare's summarizePerformanceRst.py.

Activity

John Parejko added a comment -

Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from wmwood-vasey welcome.

jbosch: Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended.

When the higher order run completes, I'll summarize it on that ticket (DM-15713).

jointcal tracts that exceed mosaic metric for AF1
HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64

HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34

HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5

HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46

HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15

HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85

HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83

HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4
HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4
HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4

jointcal tracts that exceed mosaic metric for AF2
HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23

HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76

HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0

HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83

John Parejko added a comment - Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from wmwood-vasey welcome. jbosch : Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended. When the higher order run completes, I'll summarize it on that ticket ( DM-15713 ). jointcal tracts that exceed mosaic metric for AF1 HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64 HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34 HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5 HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46 HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15 HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85 HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83 HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4 HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4 HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4 jointcal tracts that exceed mosaic metric for AF2 HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23 HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76 HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0 HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83
Jim Bosch added a comment -

Calculations look correct.

I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

Jim Bosch added a comment - Calculations look correct. I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

People

John Parejko
John Parejko
Jim Bosch
Jim Bosch, John Parejko, John Swinbank, Michael Wood-Vasey