# Separate the execution environment metadata from the other metadata in the JSON document sent to SQuaSH (dispatch_verify.py)

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
1.4
• Team:
SQuaRE

#### Description

The verification framework adds metadata associated with measurements in the Job object created by lsst.verify. Dispatch verify enrich that with provenance information obtained from the execution enviroment. While the former is arbitrary the later is mainly fixed and required by SQuaSH, thus we propose to separate them.

This will allow SQuaSH to support multiple execution environments as described in https://sqr-009.lsst.io/

#### Activity

Hide
Angelo Fausti added a comment - - edited

Follow https://pipelines.lsst.io/install/lsstsw.html to install the stack using the lsstsw method. Then follow https://pipelines.lsst.io/install/package-development.html to test the changes in the verify package, specifically:

 git clone https://github.com/lsst/verify cd verify git checkout tickets/DM-13122    setup -r . -t $USER scons -Q -j 6 opt=3  You might get an output of verify from a CI run: https://ci.lsst.codes/job/sqre/job/validate_drp/1164/artifact/cfht-verify_port/archive/validation_data_cfht/Cfht_output_r.json.xz and execute dispatch_verify from the lsstsw folder to generate the JSON document that will be sent to SQuaSH:   dispatch_verify.py --test --env jenkins --lsstsw$(pwd) Cfht_output_r.json --write verify_job.json 

You can use this example notebook https://github.com/lsst-sqre/squash-rest-api/blob/master/tests/test_api.ipynb to inspect what changed in the JSON document and exercise the SQuaSH RESTful API.

Note that now we have the environment metadata associated to the verification job data stored separately from the other metadata:

 data['meta'].keys() dict_keys(['instrument', 'packages', 'dataset_repo_url', 'env', 'filter_name']) 

and similarly in the SQuaSH database. See https://sqr-009.lsst.io/#the-qc-tier-0-database for more details.

Show
Angelo Fausti added a comment - - edited Follow https://pipelines.lsst.io/install/lsstsw.html to install the stack using the lsstsw method. Then follow https://pipelines.lsst.io/install/package-development.html to test the changes in the verify package, specifically: git clone https: //github .com /lsst/verify cd verify git checkout tickets /DM-13122   setup -r . -t $USER scons -Q -j 6 opt=3 You might get an output of verify from a CI run: https://ci.lsst.codes/job/sqre/job/validate_drp/1164/artifact/cfht-verify_port/archive/validation_data_cfht/Cfht_output_r.json.xz and execute dispatch_verify from the lsstsw folder to generate the JSON document that will be sent to SQuaSH: dispatch_verify.py -- test -- env jenkins --lsstsw$( pwd ) Cfht_output_r.json --write verify_job.json You can use this example notebook https://github.com/lsst-sqre/squash-rest-api/blob/master/tests/test_api.ipynb to inspect what changed in the JSON document and exercise the SQuaSH RESTful API. Note that now we have the environment metadata associated to the verification job data stored separately from the other metadata: data[ 'meta' ].keys() dict_keys([ 'instrument' , 'packages' , 'dataset_repo_url' , 'env' , 'filter_name' ]) and similarly in the SQuaSH database. See https://sqr-009.lsst.io/#the-qc-tier-0-database for more details.
Hide
Angelo Fausti added a comment -
Show
Angelo Fausti added a comment - See PR https://github.com/lsst/verify/pull/21
Hide
Jonathan Sick added a comment -

Looks good. Just one style comment.

Another thought, should the jenkins environment perhaps be renamed to the domain ci.lsst.codes? Using domains/URLs would have the benefit of being more specific and future proof should we start having other CI servers push to SQuaSH.

Show
Jonathan Sick added a comment - Looks good. Just one style comment. Another thought, should the jenkins environment perhaps be renamed to the domain ci.lsst.codes ? Using domains/URLs would have the benefit of being more specific and future proof should we start having other CI servers push to SQuaSH.
Hide
Angelo Fausti added a comment - - edited

Jonathan Sick I think the URL could be an atribute of the environment, but I like the idea of having a name for the environment. It is used, for instance, to create an endpoint like /jenkins/<ci_id> in the SQuaSH RESTful API so that you can refer to a resource which represents the execution environment and look up for the corresponding verify job.

Another thought regarding the user "local" execution environment, the natural choice would be the LSP or currently our JupyterLab deployment and in this case we have an associated URL too.

Show
Angelo Fausti added a comment - - edited Jonathan Sick I think the URL could be an atribute of the environment, but I like the idea of having a name for the environment. It is used, for instance, to create an endpoint like /jenkins/<ci_id> in the SQuaSH RESTful API so that you can refer to a resource which represents the execution environment and look up for the corresponding verify job. Another thought regarding the user "local" execution environment, the natural choice would be the LSP or currently our JupyterLab deployment and in this case we have an associated URL too.
Hide
Angelo Fausti added a comment -

Improved style and merged PR

Show
Angelo Fausti added a comment - Improved style and merged PR

#### People

Assignee:
Angelo Fausti
Reporter:
Angelo Fausti
Reviewers:
Jonathan Sick
Watchers:
Angelo Fausti, Jonathan Sick
0 Vote for this issue
Watchers:
2 Start watching this issue

#### Dates

Created:
Updated:
Resolved:

#### Jenkins

No builds found.