Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5252

Base "bright star" cut on S/N instead of magnitudes

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Validation
    • Labels:
      None

      Description

      The astrometry histogram generated by validateDrp.py conflates astrometric and photometric calibration because it uses magnitude for brightness, and this relies on the accuracy of the photometric calibration. Colin Slater suggests (and I agree) that brightness should be based on signal to noise ratio, thus making the astrometry histogram independent of photometric calibration.

        Attachments

          Activity

          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Agreed!

          Thanks for filing this ticket. To help prioritize, may I ask if there was a specific failure mode that prompted this ticket?

          Defining the "bright star" cutoff in terms of S/N is consistent with expectations, utility, and the SRD. The same cut will be used for both the astrometric and photometric metrics and plots.

          They should also be isolated from other sources.

          Implementation Question:
          ----------------------------------
          To preserve flexibility and clarity about when cuts are done, and to prevent preferential rejection of less-good measurements of a object, this cutoff should be applied after sources have been matched. How would you suggest defining the S/N of an object? The most obvious possibility that occurs to me is cutting for S/N > X for a specified percentage of the observations of a source. Should that quantile be 50% (median) or something else?

          The actual observed RMS (separate from reported S/N) is what we are measuring, so that's out.

          Pedantic Note:
          --------------------
          validateDrp.py will always conflate astrometric and photometric calibration. At least the photometric metrics will always be sensitive to the astrometric calibration as astrometric calibration is implicitly part of successfully matching sources across visits. Any CCDs or visits that are outliers will have a significant number of sources rejected from the matching.

          This ticket will help loosen the association in one direction, and is in keeping with the phrasing of the SRD.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Agreed! Thanks for filing this ticket. To help prioritize, may I ask if there was a specific failure mode that prompted this ticket? Defining the "bright star" cutoff in terms of S/N is consistent with expectations, utility, and the SRD. The same cut will be used for both the astrometric and photometric metrics and plots. They should also be isolated from other sources. Implementation Question: ---------------------------------- To preserve flexibility and clarity about when cuts are done, and to prevent preferential rejection of less-good measurements of a object, this cutoff should be applied after sources have been matched. How would you suggest defining the S/N of an object? The most obvious possibility that occurs to me is cutting for S/N > X for a specified percentage of the observations of a source. Should that quantile be 50% (median) or something else? The actual observed RMS (separate from reported S/N) is what we are measuring, so that's out. Pedantic Note: -------------------- validateDrp.py will always conflate astrometric and photometric calibration. At least the photometric metrics will always be sensitive to the astrometric calibration as astrometric calibration is implicitly part of successfully matching sources across visits. Any CCDs or visits that are outliers will have a significant number of sources rejected from the matching. This ticket will help loosen the association in one direction, and is in keeping with the phrasing of the SRD.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          I changed the title back to "bright star" because the SRD statements for both astrometric and photometric quantities apply to the "bright end" of isolated point sources.

          LSST SRD, Section 3.3
          """For quantities with a strong dependence on the signal-to-noise ratio (SNR), requirements are specified at the bright end, defined here as the magnitude range between 1 mag and 4 mag fainter than the saturation limit (full well) in a given bandpass. Assuming that the faint end of this range corresponds to r = 20, and that 5σ depth is achieved at r = 24.5, the photon statistics limits on photometric and astrometric accuracy (SNR ∼ 200) are 5 millimag and 4 milliarcsec for a fiducial delivered seeing of 0.7 arcsec."""

          So the SRD doesn't actually phrase it directly in terms of S/N, but S/N should be a good proxy for the intent. However, a S/N cut of >200 would be too limiting for the small datasets we're currently looking at. I suggest that the default be something like S/N >50, and that this will of course be configurable.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - I changed the title back to "bright star" because the SRD statements for both astrometric and photometric quantities apply to the "bright end" of isolated point sources. LSST SRD, Section 3.3 """For quantities with a strong dependence on the signal-to-noise ratio (SNR), requirements are specified at the bright end, defined here as the magnitude range between 1 mag and 4 mag fainter than the saturation limit (full well) in a given bandpass. Assuming that the faint end of this range corresponds to r = 20, and that 5σ depth is achieved at r = 24.5, the photon statistics limits on photometric and astrometric accuracy (SNR ∼ 200) are 5 millimag and 4 milliarcsec for a fiducial delivered seeing of 0.7 arcsec.""" So the SRD doesn't actually phrase it directly in terms of S/N, but S/N should be a good proxy for the intent. However, a S/N cut of >200 would be too limiting for the small datasets we're currently looking at. I suggest that the default be something like S/N >50, and that this will of course be configurable.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment - - edited

          Russell Owen Did a full draft start up on using a SNR cut instead of

          {good_mag_limit}

          cut.

          Need to:
          1. update the YAML configuration files.
          2. Think through goodMatches vs. safeMatches.
          3. Look at sample data sets (CFHT, DECam, HSC) to verify reasonableness.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - - edited Russell Owen Did a full draft start up on using a SNR cut instead of {good_mag_limit} cut. Need to: 1. update the YAML configuration files. 2. Think through goodMatches vs. safeMatches. 3. Look at sample data sets (CFHT, DECam, HSC) to verify reasonableness.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment - - edited

          Quick review. Make sure it runs for you. Check output plots to see if you like the SNR plotting and cuts.

          Uses S/N threshold instead of mag threshold to define the "bright" stars to use for the SRD and related calculations.

          Plots astrometric RMS vs. SNR and mag.

          Reworked the plotAstrometric and plotPhotometric functions to depend only on the variables they actually use. Updated the calls to these functions in validate.py.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - - edited Quick review. Make sure it runs for you. Check output plots to see if you like the SNR plotting and cuts. Uses S/N threshold instead of mag threshold to define the "bright" stars to use for the SRD and related calculations. Plots astrometric RMS vs. SNR and mag. Reworked the plotAstrometric and plotPhotometric functions to depend only on the variables they actually use. Updated the calls to these functions in validate.py.
          Hide
          rowen Russell Owen added a comment -

          This looks good. The astrometry plots are a bit narrow due to adding both an SNR and a magnitude plot.

          Would it suffice to print the estimated mag at the SNR cutoff? I realize picking a zero point becomes an issue.

          (The mag plot does provide a way to show the existence of insane zero points, but it's a fairly crude tool for that.)

          Show
          rowen Russell Owen added a comment - This looks good. The astrometry plots are a bit narrow due to adding both an SNR and a magnitude plot. Would it suffice to print the estimated mag at the SNR cutoff? I realize picking a zero point becomes an issue. (The mag plot does provide a way to show the existence of insane zero points, but it's a fairly crude tool for that.)
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Thanks for the feedback. I removed the astrom. vs. mag plot and instead added a SNR vs. mag plot on the photometry plot.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Thanks for the feedback. I removed the astrom. vs. mag plot and instead added a SNR vs. mag plot on the photometry plot.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          In the process of tying up this current ticket, I insert a quick fix on a new tickets/DM-4683 branch that removed the key=None keyword from a butler.queryMetadata call. I took the liberty of self-merging this 10-character change.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - In the process of tying up this current ticket, I insert a quick fix on a new tickets/ DM-4683 branch that removed the key=None keyword from a butler.queryMetadata call. I took the liberty of self-merging this 10-character change.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Merged to master.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Merged to master.

            People

            • Assignee:
              wmwood-vasey Michael Wood-Vasey
              Reporter:
              rowen Russell Owen
              Reviewers:
              Russell Owen
              Watchers:
              Colin Slater, Michael Wood-Vasey, Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: