Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6629

validate_drp: design and implement an API for metric measurements and serializations

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: QA
    • Labels:

      Description

      validate_drp computes metrics and generates JSON that, through the post-qa tool, is submitted to the SQuaSH REST API for persistence and display in a web app.

      A previous ticket, DM-6086, we bolted on a JSON serialization scheme compatible with SQuaSH. However, this approach was not well integrated with validate_drp. We want a framework/API where serialization is handled consistently and integrally with metric computations. This includes the semantic serialization of computational parameters and reduced datasets.

      This API can be applied beyond validate_drp as a means for metrics and integration tests to be submitted to SQuaSH as well.

        Attachments

          Issue Links

            Activity

            Hide
            jsick Jonathan Sick added a comment -

            See PR for sample plots, print output, and JSON.

            Docs are being prototyped at https://validate-drp.lsst.io/v/DM-6917/

            Show
            jsick Jonathan Sick added a comment - See PR for sample plots, print output, and JSON. Docs are being prototyped at https://validate-drp.lsst.io/v/DM-6917/
            Hide
            jsick Jonathan Sick added a comment -

            I’m putting this ticket into review, finally, and I’d like everyone involved in SQUASH and validate_drp to take a look since I think think this will be the API that everyone uses to implement metrics and measurements that get presented to the SQUASH dashboard and DB, in validate_drp or elsewhere in the stack.

            Take a look at the measurement classes (e.g., AMxMeasurement) in the calcsrd module to get a feel for what implementing measurements is like from a developer perspective. I’ve used a tiny bit of metaprogramming to making the data model easy to use for developers.

            I’ve also put together prototype user docs in a separate ticket, but they’re available at https://validate-drp.lsst.io/v/DM-6917/

            The PR also includes sample plots re-implemented with the new data model.

            Show
            jsick Jonathan Sick added a comment - I’m putting this ticket into review, finally, and I’d like everyone involved in SQUASH and validate_drp to take a look since I think think this will be the API that everyone uses to implement metrics and measurements that get presented to the SQUASH dashboard and DB, in validate_drp or elsewhere in the stack. Take a look at the measurement classes (e.g., AMxMeasurement) in the calcsrd module to get a feel for what implementing measurements is like from a developer perspective. I’ve used a tiny bit of metaprogramming to making the data model easy to use for developers. I’ve also put together prototype user docs in a separate ticket, but they’re available at https://validate-drp.lsst.io/v/DM-6917/ The PR also includes sample plots re-implemented with the new data model.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Jonathan Sick This is going to take me another of couple of days to get to. This review caught me at the LSST DESC collaboration meeting, and I'm still working my way out of the backlog from that.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Jonathan Sick This is going to take me another of couple of days to get to. This review caught me at the LSST DESC collaboration meeting, and I'm still working my way out of the backlog from that.
            Hide
            jsick Jonathan Sick added a comment -

            No worries Michael Wood-Vasey; no one else has commented on it yet either.

            Show
            jsick Jonathan Sick added a comment - No worries Michael Wood-Vasey ; no one else has commented on it yet either.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            To be honest, that makes me worry more. My goal is for you to get feedback, not for me to not be later than others

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - To be honest, that makes me worry more. My goal is for you to get feedback, not for me to not be later than others
            Hide
            jsick Jonathan Sick added a comment -

            Because of the difficult nature of such a large code review (and the pressing needs of EVM), let’s pivot this review:

            • I will close this ticket, along with the associated doc ticket DM-6917
            • I will merge this PR, along with DM-6917, onto a u/sqre/measurement-api integration branch in validate_drp.
            • I will convene a meeting next week (week of Aug 1, 2016). The purpose of this meeting will be to treat DM-6629 and DM-6917 as a design prototype prototype for a measurement API in validate_drp. In this meeting I’ll present the motivation, features, and other salient points about this prototype. Then we will discuss whether this is a design we want to use in validate_drp and other packages making metric measurements that might be report to SQUASH. We’ll also discuss how this proposed data model impacts SQuaSH. Everyone already named as a reviewer on this ticket is explicitly invited to this design review. I’d also like to invite Tim Jenness to represent the interests of DM Architecture.

            Any objections before I go ahead and mark this ticket as done?

            Show
            jsick Jonathan Sick added a comment - Because of the difficult nature of such a large code review (and the pressing needs of EVM), let’s pivot this review: I will close this ticket, along with the associated doc ticket DM-6917 I will merge this PR, along with DM-6917 , onto a u/sqre/measurement-api integration branch in validate_drp . I will convene a meeting next week (week of Aug 1, 2016). The purpose of this meeting will be to treat DM-6629 and DM-6917 as a design prototype prototype for a measurement API in validate_drp . In this meeting I’ll present the motivation, features, and other salient points about this prototype. Then we will discuss whether this is a design we want to use in validate_drp and other packages making metric measurements that might be report to SQUASH. We’ll also discuss how this proposed data model impacts SQuaSH. Everyone already named as a reviewer on this ticket is explicitly invited to this design review. I’d also like to invite Tim Jenness to represent the interests of DM Architecture. Any objections before I go ahead and mark this ticket as done?
            Hide
            jsick Jonathan Sick added a comment -

            Merged to u/sqre/measurement-api. We’ll review the new API proposal presented here on that branch.

            Show
            jsick Jonathan Sick added a comment - Merged to u/sqre/measurement-api . We’ll review the new API proposal presented here on that branch.

              People

              • Assignee:
                jsick Jonathan Sick
                Reporter:
                jsick Jonathan Sick
                Reviewers:
                Angelo Fausti, Frossie Economou, Joshua Hoblitt, Michael Wood-Vasey
                Watchers:
                Angelo Fausti, Frossie Economou, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel