Fix Version/s: None
Once we have the new service deployed we need to migrate the data from the current production database squash-db to the new QC Tier 0 database schema implemented in
The task consists in grabbing the jobs from the current production API https://squash-api.lsst.codes/jobs/<id>, convert them to the new verification job format and use the new API https://squash-restful-api-demo.lsst.codes/job to restore these jobs.
The result should be an script that helps us to automate that task, so that we can repeat it easily and restore the current production database.
Assuming that the current production API provides all the data we need, i.e if no change in the API is required, the real work is to write the piece of code that converts the jobs to the new verification job format.
This notebook provides and example on how to use the new API and POST verification jobs.
Transformation from the old job format to the verification job format.
You can check the new format of the verification jobs in the above notebook. It has:
you don't have to worry about 'metrics' and 'specs' this info will not be restored. It is completely different in the new format, the only thing we need is to the metric name right which is the key for the metrics definition and specifications.
1.1) Now metrics are prefixed by the verification package, all metrics in production are for the validade_drp package so you have to rename the metric name to validate_drp.<metric>
1.2) Include unit field, its value can be discovered from the https://squash-api-dev.lsst.codes/metrics/<metric name>
1.3) I dont use the identifier field, so you can include and left its value null
1.4) include blob_refs field, the list of blob identifiers can be obtained from
or let it null if there's no blob associated to the job. Note that blob_refs is just a list in the new format.
Blobs format has not changed, they always have the following keys:
However, the extras and parameters fields in the old format:
became a single blob in the new format. So you have to merge extras and parameters into a single object that goes into the data key in
you have to generate a new blob UUID-4 based identifier using
Finally the blob name is the name of the associated metric (see 1.1) above.
The new blob must be added to the data['blobs'] list and its identifier added to the blob_refs for this measurement (see 1.4 above).
Example of data['blobs'] in the new format is available in the notebook mentioned above.
These fields in the OLD FORMAT:
now go to:
You have to add a new field, though
with the fixed value jenkins for all jobs.
packages are now part of metadata in the NEW FORMAT:
but the name field also became a key:
we should preserve the value of the 'date' field which corresponds to the timestamp when the job was originally created.