Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6125

Do robustness tests of ngmix PSF approx plugin

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_extensions_ngmix
    • Labels:
      None
    • Story Points:
      6
    • Epic Link:
    • Sprint:
      DRP X16-3
    • Team:
      Data Release Production

      Description

      Run tests on the ngmix PSF approx plugin similar to those which were run on ShapeletPsfApprox. We will test both for how long the plugin takes to run, and how often it fails.

      Note previous report on CModel and SPA was DM-4368

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment - - edited

            Per discussion with Perry: this is not really about "performance", in the sense of timing, and I've updated the description to say so. The issue really is that he believes the ngmix code is not robust and fails on some simple PSF shapes, and he wants to characterize that in more detail. In that sense, this is not analogous to DM-4368.

            Show
            swinbank John Swinbank added a comment - - edited Per discussion with Perry: this is not really about "performance", in the sense of timing, and I've updated the description to say so. The issue really is that he believes the ngmix code is not robust and fails on some simple PSF shapes, and he wants to characterize that in more detail. In that sense, this is not analogous to DM-4368 .
            Hide
            pgee Perry Gee added a comment -

            I am now finding the EmPsfApprox to be reasonably fast and reliable. See the notes under DM-6124.

            I did run a series of "robustness" test, which seem to indicate that the algorithm no longer fails at all with the PhoSim Psf simulations we have been testing with. I ran 15000 Psfs without failure over 15000 exposures with different Psfs. The Pixel variation from the original input PSF found by the model is consistent with what we found for the ShapeletPsfApprox results found in DM-6124

            SUMMARY using PsfApprox nGauss1 with 15000 exposures
            PsfApprox runtime: 0.0015 sec. +- 0.000049 sec.
            CModel runtime: 0.0113 sec. +- 0.008044 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00016254
            PSF std pixel dff stdev: 0.00231341

            Show
            pgee Perry Gee added a comment - I am now finding the EmPsfApprox to be reasonably fast and reliable. See the notes under DM-6124 . I did run a series of "robustness" test, which seem to indicate that the algorithm no longer fails at all with the PhoSim Psf simulations we have been testing with. I ran 15000 Psfs without failure over 15000 exposures with different Psfs. The Pixel variation from the original input PSF found by the model is consistent with what we found for the ShapeletPsfApprox results found in DM-6124 SUMMARY using PsfApprox nGauss1 with 15000 exposures PsfApprox runtime: 0.0015 sec. +- 0.000049 sec. CModel runtime: 0.0113 sec. +- 0.008044 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00016254 PSF std pixel dff stdev: 0.00231341
            Hide
            pgee Perry Gee added a comment -

            Here is set of runs to show that the EmPsfApprox algorithm is stable with 3 different seeing values of the
            PhoSim simulated Psfs:

            1. With 200 exposure with seeing 0.5
            SUMMARY using PsfApprox nGauss1 with 200 exposures
            PsfApprox runtime: 0.0014 sec. +- 0.000100 sec.
            CModel runtime: 0.0084 sec. +- 0.006040 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00016376
            PSF std pixel dff stdev: 0.00292722
            SUMMARY using PsfApprox nGauss2 with 200 exposures
            PsfApprox runtime: 0.0140 sec. +- 0.000661 sec.
            CModel runtime: 0.0113 sec. +- 0.002151 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00015757
            PSF std pixel dff stdev: 0.00301078
            SUMMARY using PsfApprox nGauss3 with 200 exposures
            PsfApprox runtime: 0.2525 sec. +- 0.029623 sec.
            CModel runtime: 0.0159 sec. +- 0.003145 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00015643
            PSF std pixel dff stdev: 0.00302485

            2. With 200 exposure with seeing 0.7
            SUMMARY using PsfApprox nGauss1 with 200 exposures
            PsfApprox runtime: 0.0015 sec. +- 0.000051 sec.
            CModel runtime: 0.0178 sec. +- 0.017543 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00016231
            PSF std pixel dff stdev: 0.00231461
            SUMMARY using PsfApprox nGauss2 with 200 exposures
            PsfApprox runtime: 0.0193 sec. +- 0.000415 sec.
            CModel runtime: 0.0120 sec. +- 0.002745 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00015718
            PSF std pixel dff stdev: 0.00238253
            SUMMARY using PsfApprox nGauss3 with 200 exposures
            PsfApprox runtime: 0.2405 sec. +- 0.027247 sec.
            CModel runtime: 0.0172 sec. +- 0.003851 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00015615
            PSF std pixel dff stdev: 0.00239068

            3. With 200 exposure with seeing 0.9
            SUMMARY using PsfApprox nGauss1 with 200 exposures
            PsfApprox runtime: 0.0015 sec. +- 0.000056 sec.
            CModel runtime: 0.0225 sec. +- 0.022049 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00015982
            PSF std pixel dff stdev: 0.00193976
            SUMMARY using PsfApprox nGauss2 with 200 exposures
            PsfApprox runtime: 0.0270 sec. +- 0.000659 sec.
            CModel runtime: 0.0136 sec. +- 0.004662 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00015543
            PSF std pixel dff stdev: 0.00199526
            SUMMARY using PsfApprox nGauss3 with 200 exposures
            PsfApprox runtime: 0.2525 sec. +- 0.073085 sec.
            CModel runtime: 0.0191 sec. +- 0.005762 sec
            Falure count for PsfApprox: 0
            PSF avg pixel diff: -0.00015451
            PSF std pixel dff stdev: 0.00200097

            Show
            pgee Perry Gee added a comment - Here is set of runs to show that the EmPsfApprox algorithm is stable with 3 different seeing values of the PhoSim simulated Psfs: 1. With 200 exposure with seeing 0.5 SUMMARY using PsfApprox nGauss1 with 200 exposures PsfApprox runtime: 0.0014 sec. +- 0.000100 sec. CModel runtime: 0.0084 sec. +- 0.006040 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00016376 PSF std pixel dff stdev: 0.00292722 SUMMARY using PsfApprox nGauss2 with 200 exposures PsfApprox runtime: 0.0140 sec. +- 0.000661 sec. CModel runtime: 0.0113 sec. +- 0.002151 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00015757 PSF std pixel dff stdev: 0.00301078 SUMMARY using PsfApprox nGauss3 with 200 exposures PsfApprox runtime: 0.2525 sec. +- 0.029623 sec. CModel runtime: 0.0159 sec. +- 0.003145 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00015643 PSF std pixel dff stdev: 0.00302485 2. With 200 exposure with seeing 0.7 SUMMARY using PsfApprox nGauss1 with 200 exposures PsfApprox runtime: 0.0015 sec. +- 0.000051 sec. CModel runtime: 0.0178 sec. +- 0.017543 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00016231 PSF std pixel dff stdev: 0.00231461 SUMMARY using PsfApprox nGauss2 with 200 exposures PsfApprox runtime: 0.0193 sec. +- 0.000415 sec. CModel runtime: 0.0120 sec. +- 0.002745 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00015718 PSF std pixel dff stdev: 0.00238253 SUMMARY using PsfApprox nGauss3 with 200 exposures PsfApprox runtime: 0.2405 sec. +- 0.027247 sec. CModel runtime: 0.0172 sec. +- 0.003851 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00015615 PSF std pixel dff stdev: 0.00239068 3. With 200 exposure with seeing 0.9 SUMMARY using PsfApprox nGauss1 with 200 exposures PsfApprox runtime: 0.0015 sec. +- 0.000056 sec. CModel runtime: 0.0225 sec. +- 0.022049 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00015982 PSF std pixel dff stdev: 0.00193976 SUMMARY using PsfApprox nGauss2 with 200 exposures PsfApprox runtime: 0.0270 sec. +- 0.000659 sec. CModel runtime: 0.0136 sec. +- 0.004662 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00015543 PSF std pixel dff stdev: 0.00199526 SUMMARY using PsfApprox nGauss3 with 200 exposures PsfApprox runtime: 0.2525 sec. +- 0.073085 sec. CModel runtime: 0.0191 sec. +- 0.005762 sec Falure count for PsfApprox: 0 PSF avg pixel diff: -0.00015451 PSF std pixel dff stdev: 0.00200097
            Hide
            pgee Perry Gee added a comment -

            The EmPsfApprox now seems to be robust over both a large number of Psf samples, all three fits (nGauss1, 2, and 3) and over the set of seeing values we have in our PhoSim simulation (raw seeing 0.5, 0.7, and 0.9 arcsec)

            Show
            pgee Perry Gee added a comment - The EmPsfApprox now seems to be robust over both a large number of Psf samples, all three fits (nGauss1, 2, and 3) and over the set of seeing values we have in our PhoSim simulation (raw seeing 0.5, 0.7, and 0.9 arcsec)
            Hide
            jbosch Jim Bosch added a comment -

            Looks great. I'm especially encouraged to see that the runtime is now consistently much less than the CModel runtime, which what I'd hoped.

            Show
            jbosch Jim Bosch added a comment - Looks great. I'm especially encouraged to see that the runtime is now consistently much less than the CModel runtime, which what I'd hoped.

              People

              • Assignee:
                pgee Perry Gee
                Reporter:
                pgee Perry Gee
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, John Swinbank, Perry Gee
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel