[DM-27052] SQuaSH API and Telemetry infrastructure - Jira

XML

Word

Printable

Details

Type: Epic
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: None
Labels:
None

Epic Name:
sqre-f20-efd-2
Story Points:
55
WBS:
1.02C.10.02
Team:
SQuaRE
Cycle:
Fall 2020

Description

This epic captures work in two of our core metrics and telemetry systems, SQuaSH and EFD, as well as related ArgoCD and Kafka work.

It includes:

Continuation of a sprint resolving technical debt on the SQuaSH APIa
Move to CMAK or other open source replacement for the Confluent control center in the EFD

Attachments

Issue Links

clones

DM-25252 Metrics and Telemetry infrastructure

Done

is cloned by

DM-27988 EFD data at the LDF

Done

Stories in Epic (Custom Issue Matrix)

Key	Summary	Story Points	Assignee	Status
~~DM-27802~~	Redeploy Summit EFD after k3s certificate issue	1.4	Angelo Fausti	Done
~~DM-27705~~	Assist Russel in deploying the exposurelog app	4.2	Angelo Fausti	Done
~~DM-27846~~	Deploy squash-sandbox.lsst.codes using the lsp-deploy install script	1.4	Angelo Fausti	Done
~~DM-27838~~	Mount NFS path in the exposurelog Helm Chart to access a butler registry at NTS	1.4	Angelo Fausti	Done
~~DM-27943~~	Ensure Argo CD chart versions are consistent in lsp-deploy	0.7	Angelo Fausti	Done
~~DM-27924~~	Fix redis deployment in the SQuaSH API	0.7	Angelo Fausti	Done
~~DM-27918~~	Fix SQuaSH docker build	0.7	Angelo Fausti	Done
~~DM-27376~~	Kafka monitoring sprint presentation and kafdrop demo	1.4	Angelo Fausti	Done
~~DM-27352~~	Increase liveness probe initial delay in the InfluxDB deployment	0.7	Angelo Fausti	Done
~~DM-27345~~	Fix schema incompatibility in the NTS EFD	1.4	Angelo Fausti	Done
~~DM-27274~~	Add Kafdrop to argocd-efd deployment	4.2	Angelo Fausti	Done
~~DM-27257~~	EFD support during summit power up	1.4	Angelo Fausti	Done
~~DM-27211~~	Kafka-connect-manager 0.8.0 does not read InfluxDB password from the environment	0.7	Angelo Fausti	Done
~~DM-27210~~	Upgrade version of Confluent Platform in all EFD envs	1.4	Angelo Fausti	Done
~~DM-27187~~	Upgrade versions of InfluxDB, Chronograf and Kapacitor in all EFD envs	1.4	Angelo Fausti	Done
~~DM-27186~~	Fix the cache-max-memory-size exceeded error	1.4	Angelo Fausti	Done
~~DM-27534~~	Add lsst-ts GitHub org to the list of orgs for Chronograf authentication	0.7	Angelo Fausti	Done
~~DM-27526~~	InfluxDays talk preparation and presentation	4.2	Angelo Fausti	Done
~~DM-27521~~	Update TLS certs for SQuaSH and EFD services	1.4	Angelo Fausti	Done
~~DM-27399~~	Send Kafka JMX metrics to InfluxDB using the Telegraf Prometheus input plugin	4.2	Angelo Fausti	Done
~~DM-27538~~	SQR-034 updates	1.4	Angelo Fausti	Done
~~DM-27074~~	Update the SQuaSH InfluxDB data model notebook	1.4	Angelo Fausti	Done
~~DM-26979~~	Modernize SQuaSH API tests	7	Angelo Fausti	Done
~~DM-23586~~	Use the official Helm Chart repository for InfluxData applications	1.4	Angelo Fausti	Done
~~DM-18706~~	Update SQuaSH job_to_influxdb celery task with the new data model	7	Angelo Fausti	Done
~~DM-18103~~	Revisit SQuaSH InfluxDB data model	1.4	Angelo Fausti	Done
~~DM-18058~~	QAWG-REC-38: SQuaSH should be able to store and display appropriate metric values per DataId		Angelo Fausti	Done

Activity

Ascending order - Click to sort in descending order

No builds found.

Frossie Economou created issue - 03/Oct/20 7:31 AM

Frossie Economou made changes - 03/Oct/20 7:31 AM

Field	Original Value	New Value
Link		This issue clones ~~DM-25252~~ [ ~~DM-25252~~ ]

Frossie Economou made changes - 03/Oct/20 7:32 AM

Epic Name

sqre-f20-efd-1

sqre-f20-efd-2

Frossie Economou made changes - 03/Oct/20 7:33 AM

Story Points

Frossie Economou made changes - 03/Oct/20 7:36 AM

Description

This epic captures work in two of our core metrics and telemetry systems, SQuaSH and EFD, as well as related ArgoCD and Kafka work.

It includes:

* A sprint resolving technical debt on the SQuaSH APIs, including the ability to ship metrics from different environments (this is currently dealt with manually)
* A continuation of the aggregation work
* Input to the Observatory Logging project
* Support for Commissioning and Integration activities as requested, including training

This epic captures work in two of our core metrics and telemetry systems, SQuaSH and EFD, as well as related ArgoCD and Kafka work.

It includes:

* Continuation of a sprint resolving technical debt on the SQuaSH APIs, including the ability to ship metrics from different environments (this is currently dealt with manually)
* Input to the Observatory Logging project
* Support for Commissioning and Integration activities as requested, including training

Frossie Economou made changes - 03/Oct/20 7:40 AM

Epic Child

~~DM-21126~~ [ 402128 ]

Frossie Economou made changes - 03/Oct/20 7:40 AM

Epic Child

~~DM-18060~~ [ 240658 ]

Frossie Economou made changes - 03/Oct/20 7:40 AM

Epic Child

DM-26986 [ 440175 ]

Frossie Economou made changes - 03/Oct/20 7:40 AM

Epic Child

DM-26981 [ 440170 ]

Frossie Economou made changes - 03/Oct/20 7:40 AM

Epic Child

~~DM-18706~~ [ 249456 ]

Frossie Economou made changes - 03/Oct/20 7:40 AM

Epic Child

~~DM-26979~~ [ 440167 ]

Frossie Economou made changes - 03/Oct/20 7:41 AM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Frossie Economou made changes - 03/Oct/20 7:55 AM

Description

Frossie Economou made changes - 03/Oct/20 7:56 AM

Story Points

Angelo Fausti made changes - 05/Oct/20 9:00 PM

Epic Child

~~DM-18103~~ [ 240710 ]

Angelo Fausti made changes - 05/Oct/20 9:01 PM

Epic Child

~~DM-18058~~ [ 240654 ]

Angelo Fausti made changes - 05/Oct/20 10:57 PM

Epic Child

~~DM-27074~~ [ 440342 ]

Angelo Fausti made changes - 05/Oct/20 11:10 PM

Epic Child

DM-24812 [ 434534 ]

Angelo Fausti made changes - 05/Oct/20 11:11 PM

Epic Child

~~DM-24901~~ [ 434657 ]

Angelo Fausti made changes - 05/Oct/20 11:11 PM

Epic Child

DM-24659 [ 434039 ]

Angelo Fausti made changes - 05/Oct/20 11:12 PM

Epic Child

~~DM-21895~~ [ 425169 ]

Angelo Fausti made changes - 05/Oct/20 11:13 PM

Epic Child

DM-25384 [ 435951 ]

Angelo Fausti made changes - 05/Oct/20 11:14 PM

Epic Child

DM-26076 [ 437811 ]

Angelo Fausti made changes - 05/Oct/20 11:15 PM

Epic Child

~~DM-26450~~ [ 438936 ]

Angelo Fausti made changes - 05/Oct/20 11:16 PM

Epic Child

DM-16485 [ 234601 ]

Angelo Fausti made changes - 05/Oct/20 11:17 PM

Epic Child

~~DM-25964~~ [ 437456 ]

Angelo Fausti made changes - 05/Oct/20 11:18 PM

Epic Child

DM-15562 [ 184517 ]

Angelo Fausti made changes - 05/Oct/20 11:22 PM

Epic Child

DM-15388 [ 167528 ]

Angelo Fausti made changes - 05/Oct/20 11:26 PM

Epic Child

~~DM-23586~~ [ 430816 ]

Angelo Fausti made changes - 05/Oct/20 11:28 PM

Epic Child

DM-18583 [ 244768 ]

Angelo Fausti made changes - 15/Oct/20 5:15 PM

Epic Child

~~DM-27186~~ [ 440598 ]

Angelo Fausti made changes - 15/Oct/20 5:25 PM

Epic Child

~~DM-27187~~ [ 440599 ]

Angelo Fausti made changes - 16/Oct/20 11:34 PM

Epic Child

~~DM-27210~~ [ 440645 ]

Angelo Fausti made changes - 16/Oct/20 11:47 PM

Epic Child

~~DM-27211~~ [ 440646 ]

Angelo Fausti made changes - 18/Oct/20 8:38 PM

Epic Child

~~DM-27213~~ [ 440664 ]

Angelo Fausti made changes - 21/Oct/20 4:56 PM

Epic Child

~~DM-27257~~ [ 440759 ]

Angelo Fausti made changes - 22/Oct/20 6:24 PM

Epic Child

~~DM-27274~~ [ 440809 ]

Angelo Fausti made changes - 28/Oct/20 12:22 AM

Epic Child

~~DM-27345~~ [ 440959 ]

Angelo Fausti made changes - 28/Oct/20 5:57 PM

Epic Child

~~DM-27352~~ [ 441028 ]

Angelo Fausti made changes - 29/Oct/20 10:27 PM

Epic Child

~~DM-27376~~ [ 441086 ]

Angelo Fausti made changes - 02/Nov/20 6:59 PM

Epic Child

~~DM-27399~~ [ 441219 ]

Angelo Fausti made changes - 06/Nov/20 9:33 PM

Epic Child

~~DM-27454~~ [ 441373 ]

Angelo Fausti made changes - 09/Nov/20 9:32 PM

Epic Child

DM-27481 [ 441531 ]

Angelo Fausti made changes - 12/Nov/20 5:44 PM

Epic Child

~~DM-27521~~ [ 441641 ]

Angelo Fausti made changes - 12/Nov/20 6:12 PM

Epic Child

~~DM-27526~~ [ 441646 ]

Angelo Fausti made changes - 12/Nov/20 9:54 PM

Epic Child

~~DM-26450~~ [ 438936 ]

Angelo Fausti made changes - 12/Nov/20 9:55 PM

Epic Child

DM-25384 [ 435951 ]

Angelo Fausti made changes - 12/Nov/20 9:55 PM

Epic Child

~~DM-24901~~ [ 434657 ]

Angelo Fausti made changes - 12/Nov/20 9:57 PM

Epic Child

DM-24659 [ 434039 ]

Angelo Fausti made changes - 12/Nov/20 9:59 PM

Epic Child

~~DM-21895~~ [ 425169 ]

Angelo Fausti made changes - 12/Nov/20 10:00 PM

Epic Child

~~DM-25964~~ [ 437456 ]

Angelo Fausti made changes - 12/Nov/20 10:01 PM

Epic Child

~~DM-27213~~ [ 440664 ]

Angelo Fausti made changes - 12/Nov/20 10:02 PM

Epic Child

DM-24812 [ 434534 ]

Angelo Fausti made changes - 12/Nov/20 10:04 PM

Epic Child

DM-18583 [ 244768 ]

Angelo Fausti made changes - 12/Nov/20 11:08 PM

Epic Child

~~DM-27534~~ [ 441662 ]

Angelo Fausti made changes - 13/Nov/20 11:48 AM

Epic Child

~~DM-27538~~ [ 441667 ]

Angelo Fausti made changes - 13/Nov/20 5:44 PM

Epic Child

DM-27550 [ 441689 ]

Angelo Fausti made changes - 24/Nov/20 5:32 PM

Epic Child

~~DM-27705~~ [ 442107 ]

Angelo Fausti made changes - 01/Dec/20 4:55 PM

Epic Child

DM-27550 [ 441689 ]

Angelo Fausti made changes - 01/Dec/20 4:55 PM

Epic Child

DM-26986 [ 440175 ]

Angelo Fausti made changes - 01/Dec/20 4:56 PM

Epic Child

DM-26981 [ 440170 ]

Angelo Fausti made changes - 01/Dec/20 4:57 PM

Epic Child

DM-26076 [ 437811 ]

Angelo Fausti made changes - 01/Dec/20 4:58 PM

Epic Child

DM-16485 [ 234601 ]

Angelo Fausti made changes - 01/Dec/20 4:59 PM

Epic Child

DM-15388 [ 167528 ]

Angelo Fausti made changes - 01/Dec/20 5:02 PM

Epic Child

~~DM-21126~~ [ 402128 ]

Angelo Fausti made changes - 01/Dec/20 5:24 PM

Epic Child

DM-15562 [ 184517 ]

Angelo Fausti made changes - 01/Dec/20 5:50 PM

Epic Child

~~DM-27802~~ [ 442342 ]

Angelo Fausti made changes - 01/Dec/20 8:05 PM

Epic Child

DM-27808 [ 442357 ]

Angelo Fausti made changes - 01/Dec/20 8:29 PM

Epic Child

~~DM-27809~~ [ 442366 ]

Angelo Fausti made changes - 02/Dec/20 10:16 PM

Epic Child

~~DM-27838~~ [ 442405 ]

Angelo Fausti made changes - 03/Dec/20 4:24 PM

Epic Child

~~DM-27846~~ [ 442422 ]

Angelo Fausti made changes - 08/Dec/20 5:38 PM

Epic Child

~~DM-27918~~ [ 442617 ]

Angelo Fausti made changes - 08/Dec/20 7:27 PM

Epic Child

~~DM-27920~~ [ 442629 ]

Angelo Fausti made changes - 09/Dec/20 12:33 AM

Epic Child

~~DM-27924~~ [ 442645 ]

Angelo Fausti made changes - 09/Dec/20 10:00 PM

Epic Child

~~DM-27943~~ [ 442711 ]

Angelo Fausti made changes - 10/Dec/20 9:34 PM

Epic Child

~~DM-27969~~ [ 442752 ]

Angelo Fausti made changes - 10/Dec/20 11:28 PM

Epic Child

DM-27481 [ 441531 ]

Angelo Fausti made changes - 10/Dec/20 11:28 PM

Epic Child

~~DM-27454~~ [ 441373 ]

Hide

Permalink

Angelo Fausti added a comment - 11/Dec/20 12:21 AM - edited

Accomplishments in this epic:

Support to the EFD enviroments, including Summit power up
Replaced Confluent Control Center by Kafdrop + sending Kafka metrics to InfluxDB using the Telegraf Prometheus input plugin.
Improved the SQuaSH API code base, refactored the job_to_influxdb celery task, and modernized the SQuaSH API package using all the best practices adopted in the Safir framework but typing annotations (future cycle).
Assisted Russel with the exposurelog deployment
SQuaSH can be deployed now using the lsp-deploy installer script. I have rehearsed that with https://squash-sandbox.lsst.codes, the remaining open stories are to finish the production (re)deployment which implies in recreating the squash database in InfluxDB.

Show

Angelo Fausti added a comment - 11/Dec/20 12:21 AM - edited Accomplishments in this epic: Support to the EFD enviroments, including Summit power up Replaced Confluent Control Center by Kafdrop + sending Kafka metrics to InfluxDB using the Telegraf Prometheus input plugin. Improved the SQuaSH API code base, refactored the job_to_influxdb celery task, and modernized the SQuaSH API package using all the best practices adopted in the Safir framework but typing annotations (future cycle). Assisted Russel with the exposurelog deployment SQuaSH can be deployed now using the lsp-deploy installer script. I have rehearsed that with https://squash-sandbox.lsst.codes , the remaining open stories are to finish the production (re)deployment which implies in recreating the squash database in InfluxDB.

Frossie Economou made changes - 11/Dec/20 8:54 AM

Link

This issue is cloned by ~~DM-27988~~ [ ~~DM-27988~~ ]