Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: squash
-
Labels:None
-
Story Points:1.4
-
Epic Link:
-
Team:SQuaRE
-
Urgent?:No
Description
This ticket revisits the current SQuaSH InfluxDB data model as implemented in DM-16300, summarizes ongoing changes and converge to a final proposal.
The mapping between lsst.verify concepts to InfluxDB concepts:
- lsst.verify package -> InfluxDB measurement
- lsst.verify metadata -> InfluxDB tag
- lsst.verify metric value - > InfluxDB field
- Jenkins pipeline runtime -> InfluxDB timestamp
Comments:
- It is not possible to combine metric values (i.e. do math operations with metric values) that are in different measurements.
- It seems more appropriate to add metadata like ci_id and squash_id as InfluxDB fields and not tags
- It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See this InfluxDB community post.
- Define an appropriate time precision for the SQuaSH database in InfluxDB
- The InlfuxDB schema is flat. It means that job metadata and environment metadata will be equally stored as InfluxDB tags.
- Different execution environments (CI, LDF) can store metadata in SQuaSH preserving the original keys. At InfluxDB we can have a common data model so that we present an unified view of the different execution environment to the users (e.g ci_id and {run_id}} mapped to run_id).
Attachments
Issue Links
- relates to
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
DM-18599 ap_verify jenkins pipeline does not set PRODUCT env variable
- To Do
-
- Done
(6 relates to)
Activity
| Field | Original Value | New Value |
|---|---|---|
| Epic Link |
|
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Watchers | Angelo Fausti, Krzysztof Findeisen, Simon Krughoff [ Angelo Fausti, Krzysztof Findeisen, Simon Krughoff ] | Angelo Fausti, Krzysztof Findeisen, Simon Krughoff, Yusra AlSayyad [ Angelo Fausti, Krzysztof Findeisen, Simon Krughoff, Yusra AlSayyad ] |
| Description |
This ticket captures the current SQuaSH InfluxDB data model, summarize the ongoing changes and converge to a final proposal.
The mapping between {{lsst.verify}} concepts to InfluxDB concepts: - {{lsst.verify}} package -> InfluxDB measurement - {{lsst.verify}} metadata -> InfluxDB tag - {{lsst.verify}} metric value - > InfluxDB field - Jenkins pipeline runtime -> InfluxDB timestamp Comments: - It is not possible to combine metric values (i.e. do math operations) that are in different measurements. - Some metadata should be added as InfluxDB field and not tag, for example {{ci_id}} and {{squash_id}} - It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See [this InfluxDB community post|https://community.influxdata.com/t/math-operations-on-field-value-and-time/6323/4] and [this other one|https://community.influxdata.com/t/storing-additional-timestamp-type-information-as-a-field/6082/2]. - Define the appropriate time precision for the SQuaSH database |
This ticket captures the current SQuaSH InfluxDB data model, summarize the ongoing changes and converge to a final proposal.
The mapping between {{lsst.verify}} concepts to InfluxDB concepts: - {{lsst.verify}} package -> InfluxDB measurement - {{lsst.verify}} metadata -> InfluxDB tag - {{lsst.verify}} metric value - > InfluxDB field - Jenkins pipeline runtime -> InfluxDB timestamp Comments: - It is not possible to combine metric values (i.e. do math operations with metric values) that are in different measurements. - It seems more appropriate to add metadata like {{ci_id}} and {{squash_id}} as InfluxDB fields and not tags - It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See [this InfluxDB community post|https://community.influxdata.com/t/math-operations-on-field-value-and-time/6323/4] and [this other one|https://community.influxdata.com/t/storing-additional-timestamp-type-information-as-a-field/6082/2]. - Define the appropriate time precision for the SQuaSH database - The InlfuxDB schema is flat. It means that job metadata and environment metadata will be equally stored as InfluxDB tags. |
| Summary | Review SQuaSH InfluxDB data model | Revisit SQuaSH InfluxDB data model |
| Description |
This ticket captures the current SQuaSH InfluxDB data model, summarize the ongoing changes and converge to a final proposal.
The mapping between {{lsst.verify}} concepts to InfluxDB concepts: - {{lsst.verify}} package -> InfluxDB measurement - {{lsst.verify}} metadata -> InfluxDB tag - {{lsst.verify}} metric value - > InfluxDB field - Jenkins pipeline runtime -> InfluxDB timestamp Comments: - It is not possible to combine metric values (i.e. do math operations with metric values) that are in different measurements. - It seems more appropriate to add metadata like {{ci_id}} and {{squash_id}} as InfluxDB fields and not tags - It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See [this InfluxDB community post|https://community.influxdata.com/t/math-operations-on-field-value-and-time/6323/4] and [this other one|https://community.influxdata.com/t/storing-additional-timestamp-type-information-as-a-field/6082/2]. - Define the appropriate time precision for the SQuaSH database - The InlfuxDB schema is flat. It means that job metadata and environment metadata will be equally stored as InfluxDB tags. |
This ticket revisits the current SQuaSH InfluxDB data model as implemented in [ The mapping between {{lsst.verify}} concepts to InfluxDB concepts: - {{lsst.verify}} package -> InfluxDB measurement - {{lsst.verify}} metadata -> InfluxDB tag - {{lsst.verify}} metric value - > InfluxDB field - Jenkins pipeline runtime -> InfluxDB timestamp Comments: - It is not possible to combine metric values (i.e. do math operations with metric values) that are in different measurements. - It seems more appropriate to add metadata like {{ci_id}} and {{squash_id}} as InfluxDB fields and not tags - It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See [this InfluxDB community post|https://community.influxdata.com/t/math-operations-on-field-value-and-time/6323/4] and [this other one|https://community.influxdata.com/t/storing-additional-timestamp-type-information-as-a-field/6082/2]. - Define the appropriate time precision for the SQuaSH database - The InlfuxDB schema is flat. It means that job metadata and environment metadata will be equally stored as InfluxDB tags. |
| Description |
This ticket revisits the current SQuaSH InfluxDB data model as implemented in [ The mapping between {{lsst.verify}} concepts to InfluxDB concepts: - {{lsst.verify}} package -> InfluxDB measurement - {{lsst.verify}} metadata -> InfluxDB tag - {{lsst.verify}} metric value - > InfluxDB field - Jenkins pipeline runtime -> InfluxDB timestamp Comments: - It is not possible to combine metric values (i.e. do math operations with metric values) that are in different measurements. - It seems more appropriate to add metadata like {{ci_id}} and {{squash_id}} as InfluxDB fields and not tags - It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See [this InfluxDB community post|https://community.influxdata.com/t/math-operations-on-field-value-and-time/6323/4] and [this other one|https://community.influxdata.com/t/storing-additional-timestamp-type-information-as-a-field/6082/2]. - Define the appropriate time precision for the SQuaSH database - The InlfuxDB schema is flat. It means that job metadata and environment metadata will be equally stored as InfluxDB tags. |
This ticket revisits the current SQuaSH InfluxDB data model as implemented in [ The mapping between {{lsst.verify}} concepts to InfluxDB concepts: - {{lsst.verify}} package -> InfluxDB measurement - {{lsst.verify}} metadata -> InfluxDB tag - {{lsst.verify}} metric value - > InfluxDB field - Jenkins pipeline runtime -> InfluxDB timestamp Comments: - It is not possible to combine metric values (i.e. do math operations with metric values) that are in different measurements. - It seems more appropriate to add metadata like {{ci_id}} and {{squash_id}} as InfluxDB fields and not tags - It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See [this InfluxDB community post|https://community.influxdata.com/t/math-operations-on-field-value-and-time/6323/4]. - Define an appropriate time precision for the SQuaSH database in InfluxDB - The InlfuxDB schema is flat. It means that job metadata and environment metadata will be equally stored as InfluxDB tags. - Different execution environments (CI, LDF) can store metadata in SQuaSH preserving the original keys. At InfluxDB we can have a common data model so that we present an unified view of the different execution environment to the users (e.g {{ci_id}} and {run_id}} mapped to {{run_id}}). |
| Component/s | squash [ 14169 ] |
| Status | In Progress [ 3 ] | To Do [ 10001 ] |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Story Points | 1.4 | |
| Urgent? | off |
SQR-009 text was reviewed. See PR #3
| Resolution | Done [ 10000 ] | |
| Status | In Progress [ 3 ] | Done [ 10002 ] |
Consider DM-13122 that separates arbitrary job metadata from the execution environment metadata required by SQuaSH.