Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27052

SQuaSH API and Telemetry infrastructure

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      sqre-f20-efd-2
    • Story Points:
      55
    • WBS:
      1.02C.10.02
    • Team:
      SQuaRE
    • Cycle:
      Fall 2020

      Description

      This epic captures work in two of our core metrics and telemetry systems, SQuaSH and EFD, as well as related ArgoCD and Kafka work.

      It includes:

      • Continuation of a sprint resolving technical debt on the SQuaSH APIa
      • Move to CMAK or other open source replacement for the Confluent control center in the EFD

        Attachments

          Issue Links

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          DM-27802

          Redeploy Summit EFD after k3s certificate issue

          1.4 Angelo Fausti Done
           
          DM-27705

          Assist Russel in deploying the exposurelog app

          4.2 Angelo Fausti Done
           
          DM-27838

          Mount NFS path in the exposurelog Helm Chart to access a butler registry at NTS

          1.4 Angelo Fausti Done
           
          DM-27846

          Deploy squash-sandbox.lsst.codes using the lsp-deploy install script

          1.4 Angelo Fausti Done
           
          DM-26979

          Modernize SQuaSH API tests

          7 Angelo Fausti Done
           
          DM-27074

          Update the SQuaSH InfluxDB data model notebook

          1.4 Angelo Fausti Done
           
          DM-27943

          Ensure Argo CD chart versions are consistent in lsp-deploy

          0.7 Angelo Fausti Done
           
          DM-27924

          Fix redis deployment in the SQuaSH API

          0.7 Angelo Fausti Done
           
          DM-27918

          Fix SQuaSH docker build

          0.7 Angelo Fausti Done
           
          DM-27538

          SQR-034 updates

          1.4 Angelo Fausti Done
           
          DM-27534

          Add lsst-ts GitHub org to the list of orgs for Chronograf authentication

          0.7 Angelo Fausti Done
           
          DM-27526

          InfluxDays talk preparation and presentation

          4.2 Angelo Fausti Done
           
          DM-27521

          Update TLS certs for SQuaSH and EFD services

          1.4 Angelo Fausti Done
           
          DM-23586

          Use the official Helm Chart repository for InfluxData applications

          1.4 Angelo Fausti Done
           
          DM-27211

          Kafka-connect-manager 0.8.0 does not read InfluxDB password from the environment

          0.7 Angelo Fausti Done
           
          DM-27210

          Upgrade version of Confluent Platform in all EFD envs

          1.4 Angelo Fausti Done
           
          DM-27187

          Upgrade versions of InfluxDB, Chronograf and Kapacitor in all EFD envs

          1.4 Angelo Fausti Done
           
          DM-27186

          Fix the cache-max-memory-size exceeded error

          1.4 Angelo Fausti Done
           
          DM-27376

          Kafka monitoring sprint presentation and kafdrop demo

          1.4 Angelo Fausti Done
           
          DM-27352

          Increase liveness probe initial delay in the InfluxDB deployment

          0.7 Angelo Fausti Done
           
          DM-27399

          Send Kafka JMX metrics to InfluxDB using the Telegraf Prometheus input plugin

          4.2 Angelo Fausti Done
           
          DM-27274

          Add Kafdrop to argocd-efd deployment

          4.2 Angelo Fausti Done
           
          DM-27257

          EFD support during summit power up

          1.4 Angelo Fausti Done
           
          DM-27345

          Fix schema incompatibility in the NTS EFD

          1.4 Angelo Fausti Done
           
          DM-18706

          Update SQuaSH job_to_influxdb celery task with the new data model

          7 Angelo Fausti Done
           
          DM-18103

          Revisit SQuaSH InfluxDB data model

          1.4 Angelo Fausti Done
           
          DM-18058

          QAWG-REC-38: SQuaSH should be able to store and display appropriate metric values per DataId

          Angelo Fausti Done

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            Accomplishments in this epic:

            • Support to the EFD enviroments, including Summit power up
            • Replaced Confluent Control Center by Kafdrop + sending Kafka metrics to InfluxDB using the Telegraf Prometheus input plugin.
            • Improved the SQuaSH API code base, refactored the job_to_influxdb celery task, and modernized the SQuaSH API package using all the best practices adopted in the Safir framework but typing annotations (future cycle).
            • Assisted Russel with the exposurelog deployment
            • SQuaSH can be deployed now using the lsp-deploy installer script. I have rehearsed that with https://squash-sandbox.lsst.codes, the remaining open stories are to finish the production (re)deployment which implies in recreating the squash database in InfluxDB.
            Show
            afausti Angelo Fausti added a comment - - edited Accomplishments in this epic: Support to the EFD enviroments, including Summit power up Replaced Confluent Control Center by Kafdrop + sending Kafka metrics to InfluxDB using the Telegraf Prometheus input plugin. Improved the SQuaSH API code base, refactored the job_to_influxdb celery task, and modernized the SQuaSH API package using all the best practices adopted in the Safir framework but typing annotations (future cycle). Assisted Russel with the exposurelog deployment SQuaSH can be deployed now using the lsp-deploy installer script. I have rehearsed that with https://squash-sandbox.lsst.codes , the remaining open stories are to finish the production (re)deployment which implies in recreating the squash database in InfluxDB.

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              frossie Frossie Economou
              Reviewers:
              Frossie Economou
              Watchers:
              Angelo Fausti, Frossie Economou
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.