Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15713

Run jointcal comparison with higher order polynomial

    XMLWordPrintable

Details

    Description

      To see whether it improves the outlier rejection fraction, I'm going to increase the visit polynomial order by 2 (from 5 to 7) and see if that helps. It's just CPU time: don't have to make any real modifications to the execution scripts.

      Attachments

        Issue Links

          Activity

            Parejkoj John Parejko added a comment -

            Now running validate_drp on the higher order jointcal output.

            Parejkoj John Parejko added a comment - Now running validate_drp on the higher order jointcal output.

            Going to a higher order polynomial took care of the outlier problem on HSC-R 9559. I've attached the plots of this run, and pasted the only portion of of the summarize script output where jointcal is exceeds the mosaic value+N*sigma.

            jointcal tracts that exceed mosaic metric for AF1
            HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 3.37

            HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 4.99

            HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 8.12
            HSC-Z 9558 : (1.55 + 2*2.914 = 7.378) < 8.12

            HSC-I 9693 : (0.206 + 1*2.914 = 3.120) < 4.97

            HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 4.93

            HSC-Y 9559 : (0.903 + 1*2.914 = 3.817) < 4.99

            In this case, only one filter+tract exceeds mosaic's value by 2 sigma, and none by 3 sigma, so I believe we've now met your acceptance criteria.

            Whether we should up the default polynomial order to 7 (astrometry) and 9 (photometry) is a separate question.

            Parejkoj John Parejko added a comment - Going to a higher order polynomial took care of the outlier problem on HSC-R 9559. I've attached the plots of this run, and pasted the only portion of of the summarize script output where jointcal is exceeds the mosaic value+N*sigma. jointcal tracts that exceed mosaic metric for AF1 HSC-R 8521 : (0.329 + 1*2.914 = 3.243) < 3.37 HSC-Z 8523 : (1.77 + 1*2.914 = 4.684) < 4.99 HSC-Z 9558 : (1.55 + 1*2.914 = 4.464) < 8.12 HSC-Z 9558 : (1.55 + 2*2.914 = 7.378) < 8.12 HSC-I 9693 : (0.206 + 1*2.914 = 3.120) < 4.97 HSC-R 9559 : (0.954 + 1*2.914 = 3.868) < 4.93 HSC-Y 9559 : (0.903 + 1*2.914 = 3.817) < 4.99 In this case, only one filter+tract exceeds mosaic's value by 2 sigma, and none by 3 sigma, so I believe we've now met your acceptance criteria. Whether we should up the default polynomial order to 7 (astrometry) and 9 (photometry) is a separate question.

            Please look at the plots. I believe with these results we've now met the acceptance criteria laid out on the acceptance test page.

            Parejkoj John Parejko added a comment - Please look at the plots. I believe with these results we've now met the acceptance criteria laid out on the acceptance test page .
            jbosch Jim Bosch added a comment -

            Looks good!  Given that the higher outlier fractions are accompanied by a consistent improvement in the repeatability, it's entirely possible that jointcal is actually doing better than meas_mosaic in these tracts (given that we don't have ground truth for which objects actually should be considered outliers).

            As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

            jbosch Jim Bosch added a comment - Looks good!  Given that the higher outlier fractions are accompanied by a consistent improvement in the repeatability, it's entirely possible that jointcal is actually doing better than meas_mosaic in these tracts (given that we don't have ground truth for which objects actually should be considered outliers). As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

            As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"?

            Ask and ye shall receive (an attachment)...

            It looks to me like going from a 5->7th order visit polynomial clearly improves the astrometry, while going from a 7->9th order visit polynomial for photometry doesn't result in much improvement. Three tract+filter combinations show significant AF1 regression at 7th order: HSC-I 9693, HSC-Y 9559, HSC-Z 9558

            Here's the full list of values where the lower order does "better" than the higher order. As you say though, we don't know truth here, so it may be that a higher outlier fraction is in fact "more correct".

            7th order tracts that exceed the 5th order metric for AF1
            HSC-Y 16010 1.33 < 1.47
            HSC-G 8521 1.32 < 2.09
            HSC-Z 9698 0.342 < 1.3
            HSC-I 9693 2.45 < 4.97
            HSC-I 9694 0.591 < 0.688
            HSC-Z 9694 5.28 < 5.38
            HSC-I 9695 1.38 < 1.53
            HSC-I 9559 0.314 < 0.833
            HSC-Y 9559 2.59 < 4.99
            HSC-Z 9558 4.64 < 8.12

            7th order tracts that exceed the 5th order metric for AF2
            HSC-I 9693 0.743 < 0.811

            7th order tracts that exceed the 5th order metric for PA1
            HSC-R 8525 17.1 < 17.3
            HSC-R 8524 16.6 < 16.7
            HSC-G 9558 12.4 < 12.5

            7th order tracts that exceed the 5th order metric for PF1
            HSC-I 16010 42.7 < 43.0
            HSC-R 9698 42.2 < 42.3
            HSC-G 16009 30.1 < 30.3
            HSC-R 16009 32.1 < 32.2
            HSC-R 9693 33.2 < 33.4
            HSC-R 9373 37.6 < 37.8
            HSC-G 9374 32.6 < 32.7
            HSC-R 9374 33.7 < 33.8
            HSC-Y 9372 20.6 < 20.7
            HSC-R 9697 39.8 < 40.1
            HSC-I 15832 42.5 < 42.7
            HSC-R 9371 39.7 < 40.0
            HSC-R 8525 40.0 < 40.4
            HSC-I 8523 42.6 < 42.7
            HSC-G 8524 36.8 < 37.0
            HSC-R 8524 38.3 < 38.7

            Parejkoj John Parejko added a comment - As for whether we should make the parameters here the default, how hard would it be to make a set of these plots for "jointcal old parameters" vs "jointcal new parameters"? Ask and ye shall receive (an attachment)... It looks to me like going from a 5->7th order visit polynomial clearly improves the astrometry, while going from a 7->9th order visit polynomial for photometry doesn't result in much improvement. Three tract+filter combinations show significant AF1 regression at 7th order: HSC-I 9693, HSC-Y 9559, HSC-Z 9558 Here's the full list of values where the lower order does "better" than the higher order. As you say though, we don't know truth here, so it may be that a higher outlier fraction is in fact "more correct". 7th order tracts that exceed the 5th order metric for AF1 HSC-Y 16010 1.33 < 1.47 HSC-G 8521 1.32 < 2.09 HSC-Z 9698 0.342 < 1.3 HSC-I 9693 2.45 < 4.97 HSC-I 9694 0.591 < 0.688 HSC-Z 9694 5.28 < 5.38 HSC-I 9695 1.38 < 1.53 HSC-I 9559 0.314 < 0.833 HSC-Y 9559 2.59 < 4.99 HSC-Z 9558 4.64 < 8.12 7th order tracts that exceed the 5th order metric for AF2 HSC-I 9693 0.743 < 0.811 7th order tracts that exceed the 5th order metric for PA1 HSC-R 8525 17.1 < 17.3 HSC-R 8524 16.6 < 16.7 HSC-G 9558 12.4 < 12.5 7th order tracts that exceed the 5th order metric for PF1 HSC-I 16010 42.7 < 43.0 HSC-R 9698 42.2 < 42.3 HSC-G 16009 30.1 < 30.3 HSC-R 16009 32.1 < 32.2 HSC-R 9693 33.2 < 33.4 HSC-R 9373 37.6 < 37.8 HSC-G 9374 32.6 < 32.7 HSC-R 9374 33.7 < 33.8 HSC-Y 9372 20.6 < 20.7 HSC-R 9697 39.8 < 40.1 HSC-I 15832 42.5 < 42.7 HSC-R 9371 39.7 < 40.0 HSC-R 8525 40.0 < 40.4 HSC-I 8523 42.6 < 42.7 HSC-G 8524 36.8 < 37.0 HSC-R 8524 38.3 < 38.7

            People

              Parejkoj John Parejko
              Parejkoj John Parejko
              Jim Bosch
              Jim Bosch, John Parejko, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.