 # Measure jointcal acceptance statistics

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
2
• Sprint:
AP F18-4
• Team:

#### Description

Now that we have reliable output from jointcal+validate_drp, we need to compute the final summary statistics described on the acceptance test page. This is probably most easily added as a final step of jointcal_compare's summarizePerformanceRst.py.

#### Activity

No builds found.
Field Original Value New Value
Link This issue is triggered by DM-15617 [ DM-15617 ]

Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from Michael Wood-Vasey welcome.

Jim Bosch: Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended.

When the higher order run completes, I'll summarize it on that ticket (DM-15713).

jointcal tracts that exceed mosaic metric for AF1
HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64

HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34

HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5

HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46

HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15

HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85

HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83

HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4
HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4
HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4

jointcal tracts that exceed mosaic metric for AF2
HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23

HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76

HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0

HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83

Show John Parejko added a comment - Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from Michael Wood-Vasey welcome. Jim Bosch : Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended. When the higher order run completes, I'll summarize it on that ticket ( DM-15713 ). jointcal tracts that exceed mosaic metric for AF1 HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64 HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34 HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5 HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46 HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15 HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85 HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83 HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4 HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4 HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4 jointcal tracts that exceed mosaic metric for AF2 HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23 HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76 HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0 HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83
 Reviewers Jim Bosch [ jbosch ] Status To Do [ 10001 ] In Review [ 10004 ]

Calculations look correct.

I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

Show Jim Bosch added a comment - Calculations look correct. I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.
 Status In Review [ 10004 ] Reviewed [ 10101 ]
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]

#### People

Assignee: John Parejko
Reporter: John Parejko
Reviewers:
Jim Bosch
Watchers:
Jim Bosch, John Parejko, John Swinbank, Michael Wood-Vasey