Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3057

KPM Measurement: Relative Astrometry (AM1), FY15

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Epic Name:
      KPM: AM1, FY15
    • WBS:
      02C.04
    • Team:
      Data Release Production
    • Cycle:
      Summer 2015

      Description

      This will be measured on precursor data (almost certainly from HSC, via DM-2380) using the following procedure:

      • Run visit-level processing (i.e. ProcessCcdTask, probably via the new HSC driver ported on DM-3368).
      • Run relative astrometric calibration (i.e. meas_mosaic, DM-2674).
      • Select bright stars and match across visits (new scripts, mostly delegating to existing code). Just selecting the stars used for PSF determination would probably work.
      • Generate pairs of objects with separation near the target (5' for this issue, 20' or 200' for DM-3064 and DM-3071). Bin widths will have to be determined, as that's not included in the requirement specification. Will require new code.
      • Plot separation deltas (difference from per-pair mean separation) vs. magnitude; measure RMS and outliers in magnitude bins.

      This requirement is specified only for r and i band.

      The same procedure will be used for the AM1 (DLP-310), AM2 (DLP-311, DM-3064), and AM3 (DLP-312, DM-3071) measurements for this cycle.

        Attachments

        1. D_5_ARCMIN_16.0-16.5.jpg
          D_5_ARCMIN_16.0-16.5.jpg
          305 kB
        2. D_5_ARCMIN_16.5-17.0.jpg
          D_5_ARCMIN_16.5-17.0.jpg
          323 kB
        3. D_5_ARCMIN_17.0-17.5.jpg
          D_5_ARCMIN_17.0-17.5.jpg
          324 kB
        4. D_5_ARCMIN_17.5-18.0.jpg
          D_5_ARCMIN_17.5-18.0.jpg
          319 kB
        5. D_5_ARCMIN_18.0-18.5.jpg
          D_5_ARCMIN_18.0-18.5.jpg
          328 kB
        6. D_5_ARCMIN_18.5-19.0.jpg
          D_5_ARCMIN_18.5-19.0.jpg
          328 kB
        7. D_5_ARCMIN_19.0-19.5.jpg
          D_5_ARCMIN_19.0-19.5.jpg
          326 kB
        8. D_5_ARCMIN_scatter.jpg
          D_5_ARCMIN_scatter.jpg
          898 kB
        9. Deq5ARCMIN.jpg
          Deq5ARCMIN.jpg
          311 kB
        10. generateMatches.py
          4 kB
        11. jimBoschScript.py
          4 kB
        12. KPMScript_hist.py
          5 kB
        13. KPMScript_scatter.py
          5 kB
        14. KPMScript1.py
          4 kB
        15. ListOfLists.pkl
          6.47 MB
        16. Utilities.py
          8 kB

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Vishal Kasliwal [X] has produced this figure using the scripts attached.

            The brief summary is that we achieve 12.49 mas here, substantially better than the 60 mas requirement. Once Vishal Kasliwal [X] has his JIRA account up and running, he will update this issue with details of the data used and processing applied.

            Show
            swinbank John Swinbank added a comment - Vishal Kasliwal [X] has produced this figure using the scripts attached. The brief summary is that we achieve 12.49 mas here, substantially better than the 60 mas requirement. Once Vishal Kasliwal [X] has his JIRA account up and running, he will update this issue with details of the data used and processing applied.
            Hide
            swinbank John Swinbank added a comment -

            I note the description says we should do this in magnitude bins, which we're not doing (it looks from the script as though we use a single bin spanning mags 18--19.5). Should either add the other bins or explain why we've changed the procedure.

            Show
            swinbank John Swinbank added a comment - I note the description says we should do this in magnitude bins, which we're not doing (it looks from the script as though we use a single bin spanning mags 18--19.5). Should either add the other bins or explain why we've changed the procedure.
            Hide
            vpk24 Vishal Kasliwal [X] (Inactive) added a comment - - edited

            Here are a few plots to illustrate the current status of the Relative Astrometry KPM. First we've constructed histograms of the RMS relative separation deltas for pairs of stars with ~ 5 arcmin of each other that fall within a 0.5-magnitude wide bin. To do so, we used Jim Bosch's pairwise matching script (jimBoschScript.py) to select a list of 'safe' matches. Flags related to de-blending were applied based on a recommendation by Lauren MacArthur. The relevant quantities (id, coord_ra, coord_dec, object, visit, base_PsfFlux_mag) were read out of the GroupView object safeMatches into lists which were then pickled into a single file ListOfLists.pkl to avoid having to re-run the Jim Bosch script.
            To create the histograms using KPMScript_hist.py, we first read the pickle file in and then find all matches within an Annulus of central radius D using simple squared Cartesian distance. For each pair, we check to see if a. Both members lie within a given magnitude bin, and b. Both members were observed on the same visit (not always true to the dithering). The spherical distance between the members of the pair is computed for each common visit and the RMS spread in separation measured for a given pair.
            To create the scatter plot using KPMScript_scatter.py, a similar routine is followed. However, no rejection is performed based on bins. Instead, all pairs of objects (regardless of magnitude) are used to compute the RMS RS scatter. The file Utilities.py is used to email the user at the end of the script.

            Descriptions of individual plots follow.


            Plotted above is the histogram of point sources with PSF magnitudes from 16 (inclusive) to 16.5 (exclusive). The RMS of the relative separation from 36 pairs (across visits) is 10.05 mas (miliarcsec) which is well below the S15/S16 targets of 60 and 50 MAS respectively. In fact, the current value is almost within the final spec. Note also the percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 2.78%, well under the final AF1 spec (10%).


            Plotted above is the histogram of point sources with PSF magnitudes from 16.5 (inclusive) to 17.0 (exclusive). The RMS of the relative separation from 124 pairs (across visits) is 9.89 mas which is within the final spec (10 mas). Note also the percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 5.65%, well under the final AF1 spec (10%).


            Plotted above is the histogram of point sources with PSF magnitudes from 17.0 (inclusive) to 17.5 (exclusive). The RMS of the relative separation from 358 pairs (across visits) is 12.58 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. However, the percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 4.19% which remains under the final AF1 spec (10%).


            Plotted above is the histogram of point sources with PSF magnitudes from 17.5 (inclusive) to 18.0 (exclusive). The RMS of the relative separation from 871 pairs (across visits) is 11.77 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. The percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 4.13%, which continues to remain under the final AF1 spec (10%).


            Plotted above is the histogram of point sources with PSF magnitudes from 18.0 (inclusive) to 18.5 (exclusive). The RMS of the relative separation from 1664 pairs (across visits) is 12.42 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. The percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 3.85%, which continues to remain under the final AF1 spec (10%).


            Plotted above is the histogram of point sources with PSF magnitudes from 18.5 (inclusive) to 19.0 (exclusive). The RMS of the relative separation from 2905 pairs (across visits) is 11.95 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. The percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 3.99%, which continues to remain under the final AF1 spec (10%).


            Plotted above is the histogram of point sources with PSF magnitudes from 18.0 (inclusive) to 18.5 (exclusive). The RMS of the relative separation from 4963 pairs (across visits) is 12.43 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. The percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 3.12%, which continues to remain under the final AF1 spec (10%).

            Next we show a scatter plot of the PSF magnitudes of all pairs of targets. The individual plot points are colored by the RMS RS and are also sized by the RMS RS (for additional clarity) i.e. bigger, darker points represent pairs with poorer RMS RS repeatability.

            Notice that there is an indication of some sort of 'binning' in magnitudes. This behavior is much more evident in the D = 20 arcmin scatter plot (https://jira.lsstcorp.org/browse/DLP-311).

            Show
            vpk24 Vishal Kasliwal [X] (Inactive) added a comment - - edited Here are a few plots to illustrate the current status of the Relative Astrometry KPM. First we've constructed histograms of the RMS relative separation deltas for pairs of stars with ~ 5 arcmin of each other that fall within a 0.5-magnitude wide bin. To do so, we used Jim Bosch's pairwise matching script ( jimBoschScript.py ) to select a list of 'safe' matches. Flags related to de-blending were applied based on a recommendation by Lauren MacArthur. The relevant quantities (id, coord_ra, coord_dec, object, visit, base_PsfFlux_mag) were read out of the GroupView object safeMatches into lists which were then pickled into a single file ListOfLists.pkl to avoid having to re-run the Jim Bosch script. To create the histograms using KPMScript_hist.py , we first read the pickle file in and then find all matches within an Annulus of central radius D using simple squared Cartesian distance. For each pair, we check to see if a. Both members lie within a given magnitude bin, and b. Both members were observed on the same visit (not always true to the dithering). The spherical distance between the members of the pair is computed for each common visit and the RMS spread in separation measured for a given pair. To create the scatter plot using KPMScript_scatter.py , a similar routine is followed. However, no rejection is performed based on bins. Instead, all pairs of objects (regardless of magnitude) are used to compute the RMS RS scatter. The file Utilities.py is used to email the user at the end of the script. Descriptions of individual plots follow. Plotted above is the histogram of point sources with PSF magnitudes from 16 (inclusive) to 16.5 (exclusive). The RMS of the relative separation from 36 pairs (across visits) is 10.05 mas (miliarcsec) which is well below the S15/S16 targets of 60 and 50 MAS respectively. In fact, the current value is almost within the final spec. Note also the percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 2.78%, well under the final AF1 spec (10%). Plotted above is the histogram of point sources with PSF magnitudes from 16.5 (inclusive) to 17.0 (exclusive). The RMS of the relative separation from 124 pairs (across visits) is 9.89 mas which is within the final spec (10 mas). Note also the percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 5.65%, well under the final AF1 spec (10%). Plotted above is the histogram of point sources with PSF magnitudes from 17.0 (inclusive) to 17.5 (exclusive). The RMS of the relative separation from 358 pairs (across visits) is 12.58 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. However, the percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 4.19% which remains under the final AF1 spec (10%). Plotted above is the histogram of point sources with PSF magnitudes from 17.5 (inclusive) to 18.0 (exclusive). The RMS of the relative separation from 871 pairs (across visits) is 11.77 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. The percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 4.13%, which continues to remain under the final AF1 spec (10%). Plotted above is the histogram of point sources with PSF magnitudes from 18.0 (inclusive) to 18.5 (exclusive). The RMS of the relative separation from 1664 pairs (across visits) is 12.42 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. The percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 3.85%, which continues to remain under the final AF1 spec (10%). Plotted above is the histogram of point sources with PSF magnitudes from 18.5 (inclusive) to 19.0 (exclusive). The RMS of the relative separation from 2905 pairs (across visits) is 11.95 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. The percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 3.99%, which continues to remain under the final AF1 spec (10%). Plotted above is the histogram of point sources with PSF magnitudes from 18.0 (inclusive) to 18.5 (exclusive). The RMS of the relative separation from 4963 pairs (across visits) is 12.43 mas which is within the S20 AM1 spec (15 mas), but larger than the final target. The percentage of targets with RMS relative separation (RMS RS) under AM1+AD1 is 3.12%, which continues to remain under the final AF1 spec (10%). Next we show a scatter plot of the PSF magnitudes of all pairs of targets. The individual plot points are colored by the RMS RS and are also sized by the RMS RS (for additional clarity) i.e. bigger, darker points represent pairs with poorer RMS RS repeatability. Notice that there is an indication of some sort of 'binning' in magnitudes. This behavior is much more evident in the D = 20 arcmin scatter plot ( https://jira.lsstcorp.org/browse/DLP-311 ).
            Hide
            vpk24 Vishal Kasliwal [X] (Inactive) added a comment - - edited

            NB (John Swinbank Mario Juric Robert Lupton): Mike Jarvis at UPenn (mjarvis@physics.upenn.edu) has pointed out that the final AM1 should be atleast as good as 3 MAS rather than 10 for weak lensing science. We should probably get him to give us a tentative AM1 number based on a detailed calculation and update the final science requirement and stretch goal accordingly.

            Show
            vpk24 Vishal Kasliwal [X] (Inactive) added a comment - - edited NB ( John Swinbank Mario Juric Robert Lupton ): Mike Jarvis at UPenn (mjarvis@physics.upenn.edu) has pointed out that the final AM1 should be atleast as good as 3 MAS rather than 10 for weak lensing science. We should probably get him to give us a tentative AM1 number based on a detailed calculation and update the final science requirement and stretch goal accordingly.
            Hide
            swinbank John Swinbank added a comment -

            We've now shipped the S15 release including the KPM measurement required by this epic. Is any further work required before we close it? Further, do you have any thoughts on Vishal Kasliwal [X]/Mike Jarvis's suggestion that the requirement on AM1 should be tightened?

            Show
            swinbank John Swinbank added a comment - We've now shipped the S15 release including the KPM measurement required by this epic. Is any further work required before we close it? Further, do you have any thoughts on Vishal Kasliwal [X] /Mike Jarvis's suggestion that the requirement on AM1 should be tightened?
            Hide
            mjuric Mario Juric added a comment -

            John Swinbank et al., the methodology looks sound, but before we close:

            • Are the various scripts that are being used somewhere in a git repository, together with sufficient instructions on how to reproduce the measurements?
            • I think both I and David Nidever [X] should do a more careful code review on them to a) double-check for consistency with the letter and spirit of the SRD requirements and, b) check for any potential mistakes.

            In general, I think David should audit proposed ways to measure KPMs (sorry David ) – it's always good to have another pair of eyes there, as these tests ultimately determine whether we've passed or failed. We'll probably have to have an independent review of them in sometime near the end of construction.

            PS: If it's better from the scheduling point of view, it's fine with me to make the above tasks for FY16 KPM measurements.

            Show
            mjuric Mario Juric added a comment - John Swinbank et al., the methodology looks sound, but before we close: Are the various scripts that are being used somewhere in a git repository, together with sufficient instructions on how to reproduce the measurements? I think both I and David Nidever [X] should do a more careful code review on them to a) double-check for consistency with the letter and spirit of the SRD requirements and, b) check for any potential mistakes. In general, I think David should audit proposed ways to measure KPMs (sorry David ) – it's always good to have another pair of eyes there, as these tests ultimately determine whether we've passed or failed. We'll probably have to have an independent review of them in sometime near the end of construction. PS: If it's better from the scheduling point of view, it's fine with me to make the above tasks for FY16 KPM measurements.
            Hide
            mjuric Mario Juric added a comment -

            Vishal Kasliwal [X], thanks for reminding me! Regarding Mike's comment – we've been aware of that for a while but didn't have the time to act on it yet.

            If he's right (and I think he is), it will necessitate a tightening of the SRD. That makes it first my problem (to prove this needs to be done), and then a PST problem (to further review it and assess whether it can be done, given what we've built so far). I'll add it on my TODO list (I'd expect to be able to take it up sometime early next year).

            Zeljko Ivezic, take a look some ~three comments up, this is about having to tighten the astrometry requirements to ~3mas (just FYI at this point).

            Show
            mjuric Mario Juric added a comment - Vishal Kasliwal [X] , thanks for reminding me! Regarding Mike's comment – we've been aware of that for a while but didn't have the time to act on it yet. If he's right (and I think he is), it will necessitate a tightening of the SRD. That makes it first my problem (to prove this needs to be done), and then a PST problem (to further review it and assess whether it can be done, given what we've built so far). I'll add it on my TODO list (I'd expect to be able to take it up sometime early next year). Zeljko Ivezic , take a look some ~three comments up, this is about having to tighten the astrometry requirements to ~3mas (just FYI at this point).
            Hide
            zivezic Zeljko Ivezic added a comment -

            Yes, Robert already mentioned this new 3mas goal to me some time ago. Before we can
            propose to change the SRD, we need to understand quantitatively how whatever metric
            was used depends on the actual value of that spec. I am puzzled by the following result.
            To get to 3mas, and using error ~ FWHM/SNR, we need SNR ~ 200 or larger. The faintest
            galaxies in our gold sample (i<25) will have SNR~20. So we need to go 2.5 mag brighter
            to even theoretically be able to achieve 3mas. But if you go 2.5 brighter, and the cumulative
            counts go down as logN = 0.31*(i-25), only about 15% of galaxies in the gold sample could
            theoretically have relative astrometric errors as small as 3 mas. Hence, based on this simple
            initial analysis, I conclude that 3 mas is an overkill. To convince me that I am wrong, we will
            need a more quantitative description of the origin of this 3 mas requirement.

            Show
            zivezic Zeljko Ivezic added a comment - Yes, Robert already mentioned this new 3mas goal to me some time ago. Before we can propose to change the SRD, we need to understand quantitatively how whatever metric was used depends on the actual value of that spec. I am puzzled by the following result. To get to 3mas, and using error ~ FWHM/SNR, we need SNR ~ 200 or larger. The faintest galaxies in our gold sample (i<25) will have SNR~20. So we need to go 2.5 mag brighter to even theoretically be able to achieve 3mas. But if you go 2.5 brighter, and the cumulative counts go down as logN = 0.31*(i-25), only about 15% of galaxies in the gold sample could theoretically have relative astrometric errors as small as 3 mas. Hence, based on this simple initial analysis, I conclude that 3 mas is an overkill. To convince me that I am wrong, we will need a more quantitative description of the origin of this 3 mas requirement.
            Hide
            nidever David Nidever [X] (Inactive) added a comment -

            I'll try to look at this. I hadn't realized there was progress on this front since Bremerton. It would be useful to have the code that is being used to derive these metrics.

            Show
            nidever David Nidever [X] (Inactive) added a comment - I'll try to look at this. I hadn't realized there was progress on this front since Bremerton. It would be useful to have the code that is being used to derive these metrics.
            Hide
            vpk24 Vishal Kasliwal [X] (Inactive) added a comment -

            David Nidever [X] The code used to generate the plots is attached to this ticket.

            jimBoschScript.py does the matching and generates ListOfLists.pkl which is a python pickled object consisting of the required data for all the matches.

            KPMScript_hist.py & KPMScript_scatter.py produce the required scatter plots and histograms from ListOfLists.pkl i.e. you can avoid re-running the jimBoschScript.py and just directly use the .pkl object. This may be your best bet because the jimBoschScript.py needs HSC data to run anyway. You can change the matching parameters in the two KPMScripts to make the plots for the required annulus etc...

            Hope this helps

            Show
            vpk24 Vishal Kasliwal [X] (Inactive) added a comment - David Nidever [X] The code used to generate the plots is attached to this ticket. jimBoschScript.py does the matching and generates ListOfLists.pkl which is a python pickled object consisting of the required data for all the matches. KPMScript_hist.py & KPMScript_scatter.py produce the required scatter plots and histograms from ListOfLists.pkl i.e. you can avoid re-running the jimBoschScript.py and just directly use the .pkl object. This may be your best bet because the jimBoschScript.py needs HSC data to run anyway. You can change the matching parameters in the two KPMScripts to make the plots for the required annulus etc... Hope this helps

              People

              • Assignee:
                vpk24 Vishal Kasliwal [X] (Inactive)
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Mario Juric
                Watchers:
                David Nidever [X] (Inactive), John Swinbank, Kian-Tat Lim, Mario Juric, Vishal Kasliwal [X] (Inactive), Zeljko Ivezic
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel