Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15695

Measure jointcal acceptance statistics

    XMLWordPrintable

Details

    Description

      Now that we have reliable output from jointcal+validate_drp, we need to compute the final summary statistics described on the acceptance test page. This is probably most easily added as a final step of jointcal_compare's summarizePerformanceRst.py.

      Attachments

        Issue Links

          Activity

            Parejkoj John Parejko added a comment -

            Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from wmwood-vasey welcome.

            jbosch: Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended.

            When the higher order run completes, I'll summarize it on that ticket (DM-15713).

            jointcal tracts that exceed mosaic metric for AF1
            HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64

            HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34

            HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5

            HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46

            HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15

            HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85

            HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83

            HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4
            HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4
            HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4

            jointcal tracts that exceed mosaic metric for AF2
            HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23

            HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76

            HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0

            HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83

            Parejkoj John Parejko added a comment - Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from wmwood-vasey welcome. jbosch : Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended. When the higher order run completes, I'll summarize it on that ticket ( DM-15713 ). jointcal tracts that exceed mosaic metric for AF1 HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64 HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34 HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5 HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46 HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15 HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85 HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83 HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4 HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4 HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4 jointcal tracts that exceed mosaic metric for AF2 HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23 HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76 HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0 HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83
            jbosch Jim Bosch added a comment -

            Calculations look correct.

            I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

            jbosch Jim Bosch added a comment - Calculations look correct. I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

            People

              Parejkoj John Parejko
              Parejkoj John Parejko
              Jim Bosch
              Jim Bosch, John Parejko, John Swinbank, Michael Wood-Vasey
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.