Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15713

Run jointcal comparison with higher order polynomial

    Details

      Description

      To see whether it improves the outlier rejection fraction, I'm going to increase the visit polynomial order by 2 (from 5 to 7) and see if that helps. It's just CPU time: don't have to make any real modifications to the execution scripts.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Now running validate_drp on the higher order jointcal output.

            Show
            Parejkoj John Parejko added a comment - Now running validate_drp on the higher order jointcal output.
            Hide
            Parejkoj John Parejko added a comment -

            Going to a higher order polynomial took care of the outlier problem on HSC-R 9559. I've attached the plots of this run, and pasted the only portion of of the summarize script output where jointcal is exceeds the mosaic value+N*sigma.

            jointcal tracts that exceed mosaic metric for AF1
            HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 3.37

            HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 4.99

            HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 8.12
            HSC-Z 9558 : (1.55 + 2*2.914 = 7.378) < 8.12

            HSC-I 9693 : (0.206 + 1*2.914 = 3.120) < 4.97

            HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 4.93

            HSC-Y 9559 : (0.903 + 1*2.914 = 3.817) < 4.99

            In this case, only one filter+tract exceeds mosaic's value by 2 sigma, and none by 3 sigma, so I believe we've now met your acceptance criteria.

            Whether we should up the default polynomial order to 7 (astrometry) and 9 (photometry) is a separate question.

            Show
            Parejkoj John Parejko added a comment - Going to a higher order polynomial took care of the outlier problem on HSC-R 9559. I've attached the plots of this run, and pasted the only portion of of the summarize script output where jointcal is exceeds the mosaic value+N*sigma. jointcal tracts that exceed mosaic metric for AF1 HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 3.37 HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 4.99 HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 8.12 HSC-Z 9558 : (1.55 + 2*2.914 = 7.378) < 8.12 HSC-I 9693 : (0.206 + 1*2.914 = 3.120) < 4.97 HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 4.93 HSC-Y 9559 : (0.903 + 1*2.914 = 3.817) < 4.99 In this case, only one filter+tract exceeds mosaic's value by 2 sigma, and none by 3 sigma, so I believe we've now met your acceptance criteria. Whether we should up the default polynomial order to 7 (astrometry) and 9 (photometry) is a separate question.
            Hide
            Parejkoj John Parejko added a comment -

            Please look at the plots. I believe with these results we've now met the acceptance criteria laid out on the acceptance test page.

            Show
            Parejkoj John Parejko added a comment - Please look at the plots. I believe with these results we've now met the acceptance criteria laid out on the acceptance test page .
            Hide
            jbosch Jim Bosch added a comment -

            Looks good!  Given that the higher outlier fractions are accompanied by a consistent improvement in the repeatability, it's entirely possible that jointcal is actually doing better than meas_mosaic in these tracts (given that we don't have ground truth for which objects actually should be considered outliers).

            As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

            Show
            jbosch Jim Bosch added a comment - Looks good!  Given that the higher outlier fractions are accompanied by a consistent improvement in the repeatability, it's entirely possible that jointcal is actually doing better than meas_mosaic in these tracts (given that we don't have ground truth for which objects actually should be considered outliers). As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?
            Hide
            Parejkoj John Parejko added a comment -

            As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

            Ask and ye shall receive (an attachment)...

            It looks to me like going from a 5->7th order visit polynomial clearly improves the astrometry, while going from a 7->9th order visit polynomial for photometry doesn't result in much improvement. Three tract+filter combinations show significant AF1 regression at 7th order: HSC-I 9693, HSC-Y 9559, HSC-Z 9558

            Here's the full list of values where the lower order does "better" than the higher order. As you say though, we don't know truth here, so it may be that a higher outlier fraction is in fact "more correct".

            7th order tracts that exceed the 5th order metric for AF1
            HSC-Y 16010 1.33 < 1.47
            HSC-G 8521 1.32 < 2.09
            HSC-Z 9698 0.342 < 1.3
            HSC-I 9693 2.45 < 4.97
            HSC-I 9694 0.591 < 0.688
            HSC-Z 9694 5.28 < 5.38
            HSC-I 9695 1.38 < 1.53
            HSC-I 9559 0.314 < 0.833
            HSC-Y 9559 2.59 < 4.99
            HSC-Z 9558 4.64 < 8.12

            7th order tracts that exceed the 5th order metric for AF2
            HSC-I 9693 0.743 < 0.811

            7th order tracts that exceed the 5th order metric for PA1
            HSC-R 8525 17.1 < 17.3
            HSC-R 8524 16.6 < 16.7
            HSC-G 9558 12.4 < 12.5

            7th order tracts that exceed the 5th order metric for PF1
            HSC-I 16010 42.7 < 43.0
            HSC-R 9698 42.2 < 42.3
            HSC-G 16009 30.1 < 30.3
            HSC-R 16009 32.1 < 32.2
            HSC-R 9693 33.2 < 33.4
            HSC-R 9373 37.6 < 37.8
            HSC-G 9374 32.6 < 32.7
            HSC-R 9374 33.7 < 33.8
            HSC-Y 9372 20.6 < 20.7
            HSC-R 9697 39.8 < 40.1
            HSC-I 15832 42.5 < 42.7
            HSC-R 9371 39.7 < 40.0
            HSC-R 8525 40.0 < 40.4
            HSC-I 8523 42.6 < 42.7
            HSC-G 8524 36.8 < 37.0
            HSC-R 8524 38.3 < 38.7

            Show
            Parejkoj John Parejko added a comment - As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"? Ask and ye shall receive (an attachment)... It looks to me like going from a 5->7th order visit polynomial clearly improves the astrometry, while going from a 7->9th order visit polynomial for photometry doesn't result in much improvement. Three tract+filter combinations show significant AF1 regression at 7th order: HSC-I 9693, HSC-Y 9559, HSC-Z 9558 Here's the full list of values where the lower order does "better" than the higher order. As you say though, we don't know truth here, so it may be that a higher outlier fraction is in fact "more correct". 7th order tracts that exceed the 5th order metric for AF1 HSC-Y 16010 1.33 < 1.47 HSC-G 8521 1.32 < 2.09 HSC-Z 9698 0.342 < 1.3 HSC-I 9693 2.45 < 4.97 HSC-I 9694 0.591 < 0.688 HSC-Z 9694 5.28 < 5.38 HSC-I 9695 1.38 < 1.53 HSC-I 9559 0.314 < 0.833 HSC-Y 9559 2.59 < 4.99 HSC-Z 9558 4.64 < 8.12 7th order tracts that exceed the 5th order metric for AF2 HSC-I 9693 0.743 < 0.811 7th order tracts that exceed the 5th order metric for PA1 HSC-R 8525 17.1 < 17.3 HSC-R 8524 16.6 < 16.7 HSC-G 9558 12.4 < 12.5 7th order tracts that exceed the 5th order metric for PF1 HSC-I 16010 42.7 < 43.0 HSC-R 9698 42.2 < 42.3 HSC-G 16009 30.1 < 30.3 HSC-R 16009 32.1 < 32.2 HSC-R 9693 33.2 < 33.4 HSC-R 9373 37.6 < 37.8 HSC-G 9374 32.6 < 32.7 HSC-R 9374 33.7 < 33.8 HSC-Y 9372 20.6 < 20.7 HSC-R 9697 39.8 < 40.1 HSC-I 15832 42.5 < 42.7 HSC-R 9371 39.7 < 40.0 HSC-R 8525 40.0 < 40.4 HSC-I 8523 42.6 < 42.7 HSC-G 8524 36.8 < 37.0 HSC-R 8524 38.3 < 38.7

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, John Parejko, John Swinbank
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel