Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15695

Measure jointcal acceptance statistics

    XMLWordPrintable

    Details

      Description

      Now that we have reliable output from jointcal+validate_drp, we need to compute the final summary statistics described on the acceptance test page. This is probably most easily added as a final step of jointcal_compare's summarizePerformanceRst.py.

        Attachments

          Issue Links

          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15617 Process HSC PDR1 data with Jointcal as of 2018-08-31.

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from Michael Wood-Vasey welcome.

            Jim Bosch: Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended.

            When the higher order run completes, I'll summarize it on that ticket (DM-15713).

            jointcal tracts that exceed mosaic metric for AF1
            HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64

            HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34

            HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5

            HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46

            HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15

            HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85

            HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83

            HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4
            HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4
            HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4

            jointcal tracts that exceed mosaic metric for AF2
            HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23

            HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76

            HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0

            HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83

            Show
            Parejkoj John Parejko added a comment - Here's the filter+tracts where the jointcal AF1 or AF2 measurement is greater by 1, 2, or 3 sigma (RMS) of meas_mosaic in the outlier metrics (no photometric or AM values are worse by even 1sigma in jointcal). Only HSC-R 9559 is worse by 2 or 3 sigma RMS but that does mean we are still failing the acceptance criteria (none may deviate by more than 3σ). Time to figure out why that filter+tract has such a high AF1 (but, notably, not AF2!). Ideas from Michael Wood-Vasey welcome. Jim Bosch : Please look at my modifications to the summarize script in the PR, specifically the final "compute final summary statistics" block, to make sure I'm doing the calculation you'd intended. When the higher order run completes, I'll summarize it on that ticket ( DM-15713 ). jointcal tracts that exceed mosaic metric for AF1 HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 4.64 HSC-I 8524 : (2.84 + 1*2.914 = 5.754) < 6.34 HSC-I 8521 : (0.495 + 1*2.914 = 3.409) < 4.5 HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 4.46 HSC-I 8523 : (1.14 + 1*2.914 = 4.054) < 5.15 HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 5.85 HSC-G 9559 : (0.779 + 1*2.914 = 3.693) < 3.83 HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 13.4 HSC-R 9559 : (0.954 + 2*2.914 = 6.782) < 13.4 HSC-R 9559 : (0.954 + 3*2.914 = 9.696) < 13.4 jointcal tracts that exceed mosaic metric for AF2 HSC-R 15831 : (0.106 + 1*2.646 = 2.752) < 3.23 HSC-I 8523 : (1.29 + 1*2.646 = 3.936) < 4.76 HSC-R 16009 : (0.19 + 1*2.646 = 2.836) < 3.0 HSC-Z 9693 : (1.13 + 1*2.646 = 3.776) < 3.83
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Calculations look correct.

            I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

            Show
            jbosch Jim Bosch added a comment - Calculations look correct. I'm not particularly bothered by this particular failure of the acceptance criteria, given how much else is going right.  But it sounds like DM-15713 may fix that anyway, so let's pick up that discussion there.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, John Parejko, John Swinbank, Michael Wood-Vasey
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.