Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13122

Separate the execution environment metadata from the other metadata in the JSON document sent to SQuaSH (dispatch_verify.py)

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The verification framework adds metadata associated with measurements in the Job object created by lsst.verify. Dispatch verify enrich that with provenance information obtained from the execution enviroment. While the former is arbitrary the later is mainly fixed and required by SQuaSH, thus we propose to separate them.

      This will allow SQuaSH to support multiple execution environments as described in https://sqr-009.lsst.io/

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12193 Redesign SQuaSH QC Tier 0 database to allow measurements from different lsst.verify packages

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12194 Redesign SQuaSH REST API to make it complaint with lsst.verify

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12603 Deployment of the squash-restful-api service

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            afausti Angelo Fausti added a comment - - edited

            Follow https://pipelines.lsst.io/install/lsstsw.html to install the stack using the lsstsw method. Then follow https://pipelines.lsst.io/install/package-development.html to test the changes in the verify package, specifically:

            git clone https://github.com/lsst/verify
            		 
            cd verify
            		 
            git checkout tickets/DM-13122 
             
            		 
            setup -r . -t $USER
            		 
            scons -Q -j 6 opt=3

            You might get an output of verify from a CI run:

            https://ci.lsst.codes/job/sqre/job/validate_drp/1164/artifact/cfht-verify_port/archive/validation_data_cfht/Cfht_output_r.json.xz

            and execute dispatch_verify from the lsstsw folder to generate the JSON document that will be sent to SQuaSH:

             dispatch_verify.py --test --env jenkins --lsstsw $(pwd) Cfht_output_r.json --write verify_job.json

            You can use this example notebook https://github.com/lsst-sqre/squash-rest-api/blob/master/tests/test_api.ipynb to inspect what changed in the JSON document and exercise the SQuaSH RESTful API.

            Note that now we have the environment metadata associated to the verification job data stored separately from the other metadata:

            data['meta'].keys()
            		 
            dict_keys(['instrument', 'packages', 'dataset_repo_url', 'env', 'filter_name'])

            and similarly in the SQuaSH database. See https://sqr-009.lsst.io/#the-qc-tier-0-database for more details.

            Show
            afausti Angelo Fausti added a comment - - edited Follow https://pipelines.lsst.io/install/lsstsw.html to install the stack using the lsstsw method. Then follow https://pipelines.lsst.io/install/package-development.html to test the changes in the verify package, specifically: git clone https: //github .com /lsst/verify cd verify git checkout tickets /DM-13122   setup -r . -t $USER scons -Q -j 6 opt=3 You might get an output of verify from a CI run: https://ci.lsst.codes/job/sqre/job/validate_drp/1164/artifact/cfht-verify_port/archive/validation_data_cfht/Cfht_output_r.json.xz and execute dispatch_verify from the lsstsw folder to generate the JSON document that will be sent to SQuaSH: dispatch_verify.py -- test -- env jenkins --lsstsw $( pwd ) Cfht_output_r.json --write verify_job.json You can use this example notebook https://github.com/lsst-sqre/squash-rest-api/blob/master/tests/test_api.ipynb to inspect what changed in the JSON document and exercise the SQuaSH RESTful API. Note that now we have the environment metadata associated to the verification job data stored separately from the other metadata: data[ 'meta' ].keys() dict_keys([ 'instrument' , 'packages' , 'dataset_repo_url' , 'env' , 'filter_name' ]) and similarly in the SQuaSH database. See https://sqr-009.lsst.io/#the-qc-tier-0-database for more details.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            See PR https://github.com/lsst/verify/pull/21

            Show
            afausti Angelo Fausti added a comment - See PR https://github.com/lsst/verify/pull/21
            Hide
            Permalink
            jsick Jonathan Sick added a comment -

            Looks good. Just one style comment.

            Another thought, should the jenkins environment perhaps be renamed to the domain ci.lsst.codes? Using domains/URLs would have the benefit of being more specific and future proof should we start having other CI servers push to SQuaSH.

            Show
            jsick Jonathan Sick added a comment - Looks good. Just one style comment. Another thought, should the jenkins environment perhaps be renamed to the domain ci.lsst.codes ? Using domains/URLs would have the benefit of being more specific and future proof should we start having other CI servers push to SQuaSH.
            Hide
            Permalink
            afausti Angelo Fausti added a comment - - edited

            Jonathan Sick I think the URL could be an atribute of the environment, but I like the idea of having a name for the environment. It is used, for instance, to create an endpoint like /jenkins/<ci_id> in the SQuaSH RESTful API so that you can refer to a resource which represents the execution environment and look up for the corresponding verify job.

            Another thought regarding the user "local" execution environment, the natural choice would be the LSP or currently our JupyterLab deployment and in this case we have an associated URL too.

            Show
            afausti Angelo Fausti added a comment - - edited Jonathan Sick I think the URL could be an atribute of the environment, but I like the idea of having a name for the environment. It is used, for instance, to create an endpoint like /jenkins/<ci_id> in the SQuaSH RESTful API so that you can refer to a resource which represents the execution environment and look up for the corresponding verify job. Another thought regarding the user "local" execution environment, the natural choice would be the LSP or currently our JupyterLab deployment and in this case we have an associated URL too.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            Improved style and merged PR

            Show
            afausti Angelo Fausti added a comment - Improved style and merged PR

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Reviewers:
              Jonathan Sick
              Watchers:
              Angelo Fausti, Jonathan Sick
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.