Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15695

Measure jointcal acceptance statistics

    Details

      Description

      Now that we have reliable output from jointcal+validate_drp, we need to compute the final summary statistics described on the acceptance test page. This is probably most easily added as a final step of jointcal_compare's summarizePerformanceRst.py.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from Michael Wood-Vasey welcome.

            Jim Bosch: Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended.

            When the higher order run completes, I'll summarize it on that ticket (DM-15713).

            jointcal tracts that exceed mosaic metric for AF1
            HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64

            HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34

            HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5

            HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46

            HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15

            HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85

            HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83

            HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4
            HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4
            HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4

            jointcal tracts that exceed mosaic metric for AF2
            HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23

            HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76

            HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0

            HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83

            Show
            Parejkoj John Parejko added a comment - Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from Michael Wood-Vasey welcome. Jim Bosch : Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended. When the higher order run completes, I'll summarize it on that ticket ( DM-15713 ). jointcal tracts that exceed mosaic metric for AF1 HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64 HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34 HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5 HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46 HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15 HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85 HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83 HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4 HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4 HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4 jointcal tracts that exceed mosaic metric for AF2 HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23 HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76 HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0 HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83
            Hide
            jbosch Jim Bosch added a comment -

            Calculations look correct.

            I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

            Show
            jbosch Jim Bosch added a comment - Calculations look correct. I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Michael Wood-Vasey
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel