Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5219

Add configured requirements parameters. Pass/Fail test.

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Validation
    • Labels:
      None

      Description

      1. Add pass/fail routine to report success/fail against metrics. Do this for

      • SRD
      • Configured metrics

      2. Add pass/fail reporting to running of `validate.drp.run`

        Attachments

          Activity

          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Adds steps to bin.src/validateDrp and routines to the libraries to score performance against SRD or custom-specified requirements.

          Outputs statistics, thresholds, and pass/fail scores to STDOUT by default.

          The scoring requires only the input of the summary statistic JSON files and a configuration file that specified requestements to check. Scores pass/fail are calculated against both custom requirements and the LSST SRD requirements.

          The output of scoreMetrics could be easily connected to some other output structure for a test harness.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Adds steps to bin.src/validateDrp and routines to the libraries to score performance against SRD or custom-specified requirements. Outputs statistics, thresholds, and pass/fail scores to STDOUT by default. The scoring requires only the input of the summary statistic JSON files and a configuration file that specified requestements to check. Scores pass/fail are calculated against both custom requirements and the LSST SRD requirements. The output of scoreMetrics could be easily connected to some other output structure for a test harness.
          Hide
          jsick Jonathan Sick added a comment -

          I haven’t been following validate_drp so I’ll first take a bit of time to understand the code base and then dive into your changes.

          Show
          jsick Jonathan Sick added a comment - I haven’t been following validate_drp so I’ll first take a bit of time to understand the code base and then dive into your changes.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Thanks. I sent you this review in part to provide an opportunity for you to get a sense of what I've been working on.

          To preempt some thoughts and suggestions you may have going through the code, here are a couple outstanding tickets on the ToDo list:
          1. DM-5096 Make validate_drp a Task
          2. DM-5098 Provide tests to verify the calculation of metrics.

          Thanks for starting the GitHub Pull Request. I meant to do that first, but did thing in the other order as I was traveling today.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Thanks. I sent you this review in part to provide an opportunity for you to get a sense of what I've been working on. To preempt some thoughts and suggestions you may have going through the code, here are a couple outstanding tickets on the ToDo list: 1. DM-5096 Make validate_drp a Task 2. DM-5098 Provide tests to verify the calculation of metrics. Thanks for starting the GitHub Pull Request. I meant to do that first, but did thing in the other order as I was traveling today.
          Hide
          jsick Jonathan Sick added a comment -

          Comments on the GitHub PR. I have a lot of comments that are probably beyond scope, and only a few issues/questions for clarification on this PR. Otherwise looks great.

          Show
          jsick Jonathan Sick added a comment - Comments on the GitHub PR. I have a lot of comments that are probably beyond scope, and only a few issues/questions for clarification on this PR. Otherwise looks great.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Thanks very much for the review, Jonathan Sick.

          Merged to master.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Thanks very much for the review, Jonathan Sick . Merged to master.

            People

            • Assignee:
              wmwood-vasey Michael Wood-Vasey
              Reporter:
              wmwood-vasey Michael Wood-Vasey
              Reviewers:
              Jonathan Sick
              Watchers:
              Jonathan Sick, Michael Wood-Vasey
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: