Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7043

Update SQuaSH database model and JSON API with concepts from validate_drp measurement API

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: QA
    • Labels:

      Description

      In DM-6629, a new measurement API was introduced into validate_drp that established a JSON data model for metrics, specifications of metrics, measurements, and general blob datasets. The intent of that work is to enable rich plots in the SQUASH dashboard, with access to data behind measurements. The new data model also clarifies the subtleties of metric specifications (filter dependence, and dependence on other specifications). This ticket will incorporate validate_drp’s new data model into the SQUASH database and API.

      Also related to DM-7041, which will update the post-qa tool that submits validate_drp json to the SQUASH API.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            Here is how the API endpoints look like after changing the model and serializers.

            Notice that measurements metadata is not present in the Metrics App endpoint. We kept just the fields required for this app minimizing the amount of data returned by this endpoint.

            Show
            afausti Angelo Fausti added a comment - - edited Here is how the API endpoints look like after changing the model and serializers. Notice that measurements metadata is not present in the Metrics App endpoint. We kept just the fields required for this app minimizing the amount of data returned by this endpoint.
            Show
            afausti Angelo Fausti added a comment - See also PR https://github.com/lsst-sqre/qa-dashboard/pull/31
            Hide
            jsick Jonathan Sick added a comment -

            This looks fine. Some comments are in the PR. The main need I see is documenting the schema of these JSON fields (maybe in SQR-008) so that it's clear what data is in them (there's a slight difference between the json schema made by validate_base and what finally ends up in these JSON fields and it's good to make it clear).

            Show
            jsick Jonathan Sick added a comment - This looks fine. Some comments are in the PR. The main need I see is documenting the schema of these JSON fields (maybe in SQR-008) so that it's clear what data is in them (there's a slight difference between the json schema made by validate_base and what finally ends up in these JSON fields and it's good to make it clear).
            Hide
            afausti Angelo Fausti added a comment -

            Thanks, yes I will document that properly in the SQuaSH technote, basically the mapping of validade_base Job to SQUASH API is the following

            1) the content of blobs is ingested into the Job.data field (make sense to change the field name to blobs)
            2) the content of measurements is ingested in the Measurement.metadata field, with the exception of the metric specification which is ingested in the Metric table for normalization
            3) the measurement value is kept in a separate field in the Measurement table for convenience
            4) the metrics specification has unit, description, operator, specs and reference which are all individual fields in the Metric table.

            Show
            afausti Angelo Fausti added a comment - Thanks, yes I will document that properly in the SQuaSH technote, basically the mapping of validade_base Job to SQUASH API is the following 1) the content of blobs is ingested into the Job.data field (make sense to change the field name to blobs ) 2) the content of measurements is ingested in the Measurement.metadata field, with the exception of the metric specification which is ingested in the Metric table for normalization 3) the measurement value is kept in a separate field in the Measurement table for convenience 4) the metrics specification has unit , description , operator , specs and reference which are all individual fields in the Metric table.
            Hide
            afausti Angelo Fausti added a comment -

            Applied PR comments

            Show
            afausti Angelo Fausti added a comment - Applied PR comments

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                jsick Jonathan Sick
                Reviewers:
                Jonathan Sick
                Watchers:
                Angelo Fausti, Jonathan Sick
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel