Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: squash
-
Labels:None
-
Story Points:1.4
-
Epic Link:
-
Team:SQuaRE
-
Urgent?:No
Description
This ticket revisits the current SQuaSH InfluxDB data model as implemented in DM-16300, summarizes ongoing changes and converge to a final proposal.
The mapping between lsst.verify concepts to InfluxDB concepts:
- lsst.verify package -> InfluxDB measurement
- lsst.verify metadata -> InfluxDB tag
- lsst.verify metric value - > InfluxDB field
- Jenkins pipeline runtime -> InfluxDB timestamp
Comments:
- It is not possible to combine metric values (i.e. do math operations with metric values) that are in different measurements.
- It seems more appropriate to add metadata like ci_id and squash_id as InfluxDB fields and not tags
- It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See this InfluxDB community post.
- Define an appropriate time precision for the SQuaSH database in InfluxDB
- The InlfuxDB schema is flat. It means that job metadata and environment metadata will be equally stored as InfluxDB tags.
- Different execution environments (CI, LDF) can store metadata in SQuaSH preserving the original keys. At InfluxDB we can have a common data model so that we present an unified view of the different execution environment to the users (e.g ci_id and {run_id}} mapped to run_id).
Attachments
Issue Links
- relates to
-
DM-18058 QAWG-REC-38: SQuaSH should be able to store and display appropriate metric values per DataId
- Done
-
DM-18060 Recreate squash-prod database in InfluxDB
- Done
-
DM-18343 Enable Markdown content in Chronograf table cells
- Done
-
DM-18399 Document SQuaSH data model in SQR-009
- Done
-
DM-16775 Implement notebook to exercise SQuaSH InfluxDB data model
- Done
-
DM-17049 Add verify timestamp as a new field in InfluxDB measurements
- Done
-
DM-17767 Mapping lsst.verify run time to InfluxDB
- Done
-
DM-18061 Add execution environment metadata as InfluxDB tags
- Done
-
DM-18342 Add ci_id, ci_url and squash_id as InfluxDB fields instead of tags
- Done
-
DM-18599 ap_verify jenkins pipeline does not set PRODUCT env variable
- To Do
-
DM-18706 Update SQuaSH job_to_influxdb celery task with the new data model
- Done
Consider DM-13122 that separates arbitrary job metadata from the execution environment metadata required by SQuaSH.