Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14142

Simplify definition of validate_drp ADx and AFx metrics in lsst.verify_metrics

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: squash
    • Labels:
      None

      Description

      Currently lsst.verify_metrics defines AF1,AD1; AF2,AD2; and AF3, AD3 metrics for the three specification thresholds: minimum, design and stretch, see:

      https://github.com/lsst/verify_metrics/blob/master/metrics/validate_drp.yaml#L149

      The pair AFx, ADx characterize the distribution of the astrometric distance for stellar pairs with separation of D=5, 20 and 200 arcmin. Thus, strictly, there is a dependency on AFx when you compute ADx. For instance if you compute AD1 assuming the minimum specification for AF1 you have AF1_minimum and AD1_minimum, and so on.

      As a result, and adding AM1, AM2 and AM3 to the list we end up with 21 metrics to monitor the astrometric precision, which seems unnecessary.

      Talking with Keith Bechtol he suggested that we should look at AFx, ADx as a pair since they characterize the same distribution for a fixed separation D.

      LPM-17 says "No more than AFx % of the sample will deviate by more than ADx marcsec from the median."

      In that sense, AFx or ADx could be primary or secondary, depending on whether we want to quote a percentage or a value.

      Keith also suggested a nice visualization. If you think that AF1 and AD1 form a plane with axes of value and percent, the actual data makes a curve in that plane (that represents the distribution, basically a CDF) and the design, minimum, and stretch specifications represent points in that plane. See attachement.

      In summary, we could reduce table Table 18 of LPM-17 to 6 metrics: AMx and "AFx+ADx" for x=1,2,3.

      Michael Wood-Vasey Simon Krughoff do you agree with this interpretation?

        Attachments

        1. image.png
          image.png
          130 kB
        2. image (1).png
          image (1).png
          76 kB
        3. image (2).png
          image (2).png
          146 kB

          Issue Links

            Activity

            Hide
            wmwood-vasey Michael Wood-Vasey added a comment - - edited
            • Yes, I agree that Table 18 of LPM-17 is only 6 metrics.
            • We only need to formally monitor and track "design".
              • We should store "stretch" and "minimum".
            Show
            wmwood-vasey Michael Wood-Vasey added a comment - - edited Yes, I agree that Table 18 of LPM-17 is only 6 metrics. We only need to formally monitor and track "design". We should store "stretch" and "minimum".
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Editorial: Having a verify metric that is semantically aware of levels would help the implementation. It would be nicer to have an overall design that used dicts to store values for "design", "minimum", "stretch" instead of constructing variable names to include the level. I understand the appeal of having a flat, no-dependency, but it makes it harder to add levels, such as "FY18", "FY19". See, e.g.,

            https://github.com/lsst/validate_drp/blob/master/etc/release_metrics.yaml#L18

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Editorial: Having a verify metric that is semantically aware of levels would help the implementation. It would be nicer to have an overall design that used dicts to store values for "design", "minimum", "stretch" instead of constructing variable names to include the level. I understand the appeal of having a flat, no-dependency, but it makes it harder to add levels, such as "FY18", "FY19". See, e.g., https://github.com/lsst/validate_drp/blob/master/etc/release_metrics.yaml#L18
            Hide
            lguy Leanne Guy added a comment -

            validate_drp is now deprecated and replaced by faro. We reviewed the definitions of all SRD metrics at a recent DM Science team meeting. Outcomes are recorded at: https://confluence.lsstcorp.org/display/DM/Performance+Metrics+-+Plan+of+Action

            Comments are welcome.

            Show
            lguy Leanne Guy added a comment - validate_drp is now deprecated and replaced by faro. We reviewed the definitions of all SRD metrics at a recent DM Science team meeting. Outcomes are recorded at: https://confluence.lsstcorp.org/display/DM/Performance+Metrics+-+Plan+of+Action Comments are welcome.

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti, Keith Bechtol, Leanne Guy, Michael Wood-Vasey, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.