Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7041

Update post-qa to submit new JSON from validate_drp measurement API

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: QA
    • Labels:

      Description

      DM-6629 introduced a new measurement API. The post-qa app needs to be updated to shim and upload this JSON to the SQUASH API.

      Note that this ticket only updates the shims for the new measurement api's JSON to the current SQUASH API. A separate ticket will be needed to then adapt this the output schema for a more fully-featured SQUASH API.

      This ticket should use JSON Schema to validate that the shims are successfully producing the JSON that is expected by the SQUASH API.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment -

            The changes discussed above will be implemented in a separate ticket DM-7098.

            Show
            afausti Angelo Fausti added a comment - The changes discussed above will be implemented in a separate ticket DM-7098 .
            Hide
            afausti Angelo Fausti added a comment -

            oh, this link provides all the enviroment variables that should be available for post_qa

            https://ci.lsst.codes/job/validate_drp/177/dataset=cfht,label=centos-7,python=py2/injectedEnvVars/

            there's already PRODUCT, dataset and label (but these two should be uppercase)

            Show
            afausti Angelo Fausti added a comment - oh, this link provides all the enviroment variables that should be available for post_qa https://ci.lsst.codes/job/validate_drp/177/dataset=cfht,label=centos-7,python=py2/injectedEnvVars/ there's already PRODUCT , dataset and label (but these two should be uppercase)
            Hide
            jsick Jonathan Sick added a comment -

            Can I call the API from inside the Jenkins job though? post-qa runs on Jenkins itself.

            Show
            jsick Jonathan Sick added a comment - Can I call the API from inside the Jenkins job though? post-qa runs on Jenkins itself.
            Hide
            afausti Angelo Fausti added a comment - - edited

            I think so, we could have:

            "ci_name" : "validate_drp",
            "ci_dataset": "cfht"
            "ci_label": "centos-7"

            I would say you could use the Jenkins API given the ci_id

            https://ci.lsst.codes/job/validate_drp/177/api/json?pretty=true

            but "dataset" and "label" only appear enconded in the url...

            Show
            afausti Angelo Fausti added a comment - - edited I think so, we could have: "ci_name" : "validate_drp", "ci_dataset": "cfht" "ci_label": "centos-7" I would say you could use the Jenkins API given the ci_id https://ci.lsst.codes/job/validate_drp/177/api/json?pretty=true but "dataset" and "label" only appear enconded in the url...
            Hide
            jsick Jonathan Sick added a comment - - edited

            Here's the line that sets ci_name:

            https://github.com/lsst-sqre/post-qa/blob/master/postqa/jenkinsenv.py#L26

            In other words, we just need a $DATASET environment variable in the Jenkins environment that indicates the current dataset. I think "ci_name" and your "ci_dataset" are pretty much the same thing?

            "ci_label" seems like a good idea. I'll have to find out how I can get that from the Jenkins environment.

            Update: Ah, it might make sense for "ci_name" to be the name of the Jenkins project. Again, I'll have to find out how to get that from the environment.

            Show
            jsick Jonathan Sick added a comment - - edited Here's the line that sets ci_name : https://github.com/lsst-sqre/post-qa/blob/master/postqa/jenkinsenv.py#L26 In other words, we just need a $DATASET environment variable in the Jenkins environment that indicates the current dataset. I think "ci_name" and your "ci_dataset" are pretty much the same thing? "ci_label" seems like a good idea. I'll have to find out how I can get that from the Jenkins environment. Update : Ah, it might make sense for "ci_name" to be the name of the Jenkins project. Again, I'll have to find out how to get that from the environment.
            Hide
            afausti Angelo Fausti added a comment - - edited

            Jonathan Sick, Joshua Hoblitt

            Related to the JSON generated by post_qa, right now it looks like the example below

            {
            "ci_id": "177",
            "ci_name": "demo",
            "ci_url": "https://ci.lsst.codes/job/validate_drp/dataset=cfht,label=centos-7,python=py2/177/",
            "date": "2016-07-27T16:32:08.300213+00:00",
            "measurements": [

            { "metric": "AM1", "value": 7.137208394754365 }

            ,

            ci_name should assume the value validate_drp right?

            and I am planning to add

            "ci_dataset": "cfht"

            and

            "ci_label": "centos-7"

            to implement the ability of filtering the data by dataset and platform in the dashboard (related to ticket DM-6992).

            do you agree with this change?

            Show
            afausti Angelo Fausti added a comment - - edited Jonathan Sick , Joshua Hoblitt Related to the JSON generated by post_qa, right now it looks like the example below { "ci_id": "177", "ci_name": "demo", "ci_url": "https://ci.lsst.codes/job/validate_drp/dataset=cfht,label=centos-7,python=py2/177/", "date": "2016-07-27T16:32:08.300213+00:00", "measurements": [ { "metric": "AM1", "value": 7.137208394754365 } , ci_name should assume the value validate_drp right? and I am planning to add "ci_dataset": "cfht" and "ci_label": "centos-7" to implement the ability of filtering the data by dataset and platform in the dashboard (related to ticket DM-6992 ). do you agree with this change?

              People

              • Assignee:
                jsick Jonathan Sick
                Reporter:
                jsick Jonathan Sick
                Reviewers:
                J Matt Peterson [X] (Inactive)
                Watchers:
                Angelo Fausti, J Matt Peterson [X] (Inactive), Jonathan Sick
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel