Mapping metric names to InfluxDB fields

XMLWordPrintable

Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1.4
• Team:
SQuaRE

Description

Currently we map the full qualified metric name <verification_package>.<metric_name> to InfluxDB fields and the <verification_package> name to InfluxDB measurements.

We noticed that Chronograf always prepends the measurement name to the field names when displaying them in the interface (e.g. graph labels, table headers), and queries also look a bit redundant:

 SELECT "validate_drp.AM1" AS "AM1" FROM "squash-prod"."autogen"."validate_drp" 

We can improve this by mapping only the <metric_name> to InfluxDB field names, without the <verification_package> prefix.

That's possible because metrics belonging to a given verification package are grouped by InlfuxDB measurements anyways.

With this change we expect to avoid labels like:

validate_drp.validate_drp.AM1

and we can write queries like:

 SELECT "AM1" FROM "squash-prod"."autogen"."validate_drp" 

 SELECT "validate_drp.AM1" AS "AM1" FROM "squash-prod"."autogen"."validate_drp" 

Activity

Hide
Angelo Fausti added a comment - - edited

The plan to validate this implementation is to update the notebook that implements the mapping from lsst.verify concepts to InfluxDB and recreate the squash-demo database.

Once the squash-demo database is created we can clone the existing dashboards and rewrite the queries.

Show
Angelo Fausti added a comment - - edited The plan to validate this implementation is to update the notebook that implements the mapping from lsst.verify concepts to InfluxDB and recreate the squash-demo database. Once the squash-demo database is created we can clone the existing dashboards and rewrite the queries.
Hide
Angelo Fausti added a comment -

squash-demo database was recreated and this change can be validated looking at the following dashboards:

the queries now use the squash-demo database and were changed accordingly.

Show
Angelo Fausti added a comment - squash-demo database was recreated and this change can be validated looking at the following dashboards: AP catalog metrics from CI ( DM-19605 ) DRP metrics from CI ( DM-19605 ) the queries now use the squash-demo database and were changed accordingly.
Hide
Angelo Fausti added a comment -

Krzysztof Findeisen would like to review this ticket?

You will notice the new dashboards at https://chronograf-demo.lsst.codes. I think queries and field names look less redundant now.

I also included the "RUN CI ID" column in the tables which is a link to the CI runs and there's more metadata associated with the metric values as well.

Look at the squash-demo database for those changes. I plan to recreate the squash-prod database after that.

Show
Angelo Fausti added a comment - Krzysztof Findeisen would like to review this ticket? You will notice the new dashboards at https://chronograf-demo.lsst.codes . I think queries and field names look less redundant now. I also included the "RUN CI ID" column in the tables which is a link to the CI runs and there's more metadata associated with the metric values as well. Look at the squash-demo database for those changes. I plan to recreate the squash-prod database after that.
Hide
Krzysztof Findeisen added a comment -

The code changes have only minor issues. I've also reviewed the new dashboards and they look very nice; the only confusing bit is that the HSC reprocessing dashboard links to a Jira issue instead of a Jenkins build page.

Show
Krzysztof Findeisen added a comment - The code changes have only minor issues. I've also reviewed the new dashboards and they look very nice; the only confusing bit is that the HSC reprocessing dashboard links to a Jira issue instead of a Jenkins build page.
Hide
Angelo Fausti added a comment -

Thank you Krzysztof Findeisen for the review and useful comments on the notebook that exercises the mapping. Keep in mind that's prototype code and the actual implementation of the job_to_inluxdb task is going to be done in DM-18706

As for the HSC reprocessing dashboard, those are "manual" runs at LDF and we don't have a RUN ID, the ticket number is the closest information and we (Hsin-Fang Chiang and I) decided to use that.

Show
Angelo Fausti added a comment - Thank you  Krzysztof Findeisen for the review and useful comments on the notebook that exercises the mapping. Keep in mind that's prototype code and the actual implementation of the job_to_inluxdb task is going to be done in DM-18706 As for the HSC reprocessing dashboard, those are "manual" runs at LDF and we don't have a RUN ID, the ticket number is the closest information and we ( Hsin-Fang Chiang and I) decided to use that.

People

Assignee:
Angelo Fausti
Reporter:
Angelo Fausti
Reviewers:
Krzysztof Findeisen
Watchers:
Angelo Fausti, Krzysztof Findeisen