Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18103

Revisit SQuaSH InfluxDB data model

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: squash
    • Labels:
      None

      Description

      This ticket revisits the current SQuaSH InfluxDB data model as implemented in DM-16300, summarizes ongoing changes and converge to a final proposal.

      The mapping between lsst.verify concepts to InfluxDB concepts:

      • lsst.verify package -> InfluxDB measurement
      • lsst.verify metadata -> InfluxDB tag
      • lsst.verify metric value - > InfluxDB field
      • Jenkins pipeline runtime -> InfluxDB timestamp

      Comments:

      • It is not possible to combine metric values (i.e. do math operations with metric values) that are in different measurements.
      • It seems more appropriate to add metadata like ci_id and squash_id as InfluxDB fields and not tags
      • It seems useful to have the timestamp as an InfluxDB field as well because it is not possible to do math with timestamps in InfluxDB. See this InfluxDB community post.
      • Define an appropriate time precision for the SQuaSH database in InfluxDB
      • The InlfuxDB schema is flat. It means that job metadata and environment metadata will be equally stored as InfluxDB tags.
      • Different execution environments (CI, LDF) can store metadata in SQuaSH preserving the original keys. At InfluxDB we can have a common data model so that we present an unified view of the different execution environment to the users (e.g ci_id and {run_id}} mapped to run_id).

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment -

            Consider DM-13122 that separates arbitrary job metadata from the execution environment metadata required by SQuaSH.

            Show
            afausti Angelo Fausti added a comment - Consider DM-13122 that separates arbitrary job metadata from the execution environment metadata required by SQuaSH.
            Hide
            afausti Angelo Fausti added a comment - - edited

            The SQuaSH InfluxDB data model was re-implemented in DM-18706. DM-27074 updates the notebook that demonstrates how the data model is constructed. We still need to update SQR-009 to reflect the recent changes.

            Show
            afausti Angelo Fausti added a comment - - edited The SQuaSH InfluxDB data model was re-implemented in DM-18706 . DM-27074 updates the notebook that demonstrates how the data model is constructed. We still need to update SQR-009 to reflect the recent changes.
            Hide
            afausti Angelo Fausti added a comment -

            SQR-009 text was reviewed. See PR #3

            Show
            afausti Angelo Fausti added a comment - SQR-009 text was reviewed. See PR #3

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti, Krzysztof Findeisen, Simon Krughoff, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: