Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14142

Simplify definition of validate_drp ADx and AFx metrics in lsst.verify_metrics

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: squash
    • Labels:
      None

      Description

      Currently lsst.verify_metrics defines AF1,AD1; AF2,AD2; and AF3, AD3 metrics for the three specification thresholds: minimum, design and stretch, see:

      https://github.com/lsst/verify_metrics/blob/master/metrics/validate_drp.yaml#L149

      The pair AFx, ADx characterize the distribution of the astrometric distance for stellar pairs with separation of D=5, 20 and 200 arcmin. Thus, strictly, there is a dependency on AFx when you compute ADx. For instance if you compute AD1 assuming the minimum specification for AF1 you have AF1_minimum and AD1_minimum, and so on.

      As a result, and adding AM1, AM2 and AM3 to the list we end up with 21 metrics to monitor the astrometric precision, which seems unnecessary.

      Talking with Keith Bechtol he suggested that we should look at AFx, ADx as a pair since they characterize the same distribution for a fixed separation D.

      LPM-17 says "No more than AFx % of the sample will deviate by more than ADx marcsec from the median."

      In that sense, AFx or ADx could be primary or secondary, depending on whether we want to quote a percentage or a value.

      Keith also suggested a nice visualization. If you think that AF1 and AD1 form a plane with axes of value and percent, the actual data makes a curve in that plane (that represents the distribution, basically a CDF) and the design, minimum, and stretch specifications represent points in that plane. See attachement.

      In summary, we could reduce table Table 18 of LPM-17 to 6 metrics: AMx and "AFx+ADx" for x=1,2,3.

      Michael Wood-Vasey Simon Krughoff do you agree with this interpretation?

        Attachments

        1. image.png
          image.png
          130 kB
        2. image (1).png
          image (1).png
          76 kB
        3. image (2).png
          image (2).png
          146 kB

          Issue Links

            Activity

            Hide
            wmwood-vasey Michael Wood-Vasey added a comment - - edited
            • Yes, I agree that Table 18 of LPM-17 is only 6 metrics.
            • We only need to formally monitor and track "design".
              • We should store "stretch" and "minimum".
            Show
            wmwood-vasey Michael Wood-Vasey added a comment - - edited Yes, I agree that Table 18 of LPM-17 is only 6 metrics. We only need to formally monitor and track "design". We should store "stretch" and "minimum".
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Editorial: Having a verify metric that is semantically aware of levels would help the implementation. It would be nicer to have an overall design that used dicts to store values for "design", "minimum", "stretch" instead of constructing variable names to include the level. I understand the appeal of having a flat, no-dependency, but it makes it harder to add levels, such as "FY18", "FY19". See, e.g.,

            https://github.com/lsst/validate_drp/blob/master/etc/release_metrics.yaml#L18

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Editorial: Having a verify metric that is semantically aware of levels would help the implementation. It would be nicer to have an overall design that used dicts to store values for "design", "minimum", "stretch" instead of constructing variable names to include the level. I understand the appeal of having a flat, no-dependency, but it makes it harder to add levels, such as "FY18", "FY19". See, e.g., https://github.com/lsst/validate_drp/blob/master/etc/release_metrics.yaml#L18

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Watchers:
                Angelo Fausti, Keith Bechtol, Michael Wood-Vasey, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel