Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19605

Mapping metric names to InfluxDB fields

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: squash
    • Labels:
      None

      Description

      Currently we map the full qualified metric name <verification_package>.<metric_name> to InfluxDB fields and the <verification_package> name to InfluxDB measurements.

      We noticed that Chronograf always prepends the measurement name to the field names when displaying them in the interface (e.g. graph labels, table headers), and queries also look a bit redundant:

      SELECT "validate_drp.AM1" AS "AM1" FROM "squash-prod"."autogen"."validate_drp"

      We can improve this by mapping only the <metric_name> to InfluxDB field names, without the <verification_package> prefix.

      That's possible because metrics belonging to a given verification package are grouped by InlfuxDB measurements anyways.

      With this change we expect to avoid labels like:

      validate_drp.validate_drp.AM1

      and we can write queries like:

      SELECT "AM1" FROM "squash-prod"."autogen"."validate_drp"

      instead of

      SELECT "validate_drp.AM1" AS "AM1" FROM "squash-prod"."autogen"."validate_drp"

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18060 Recreate squash-prod database in InfluxDB

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-18706 Update SQuaSH job_to_influxdb celery task with the new data model

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            afausti Angelo Fausti added a comment - - edited

            The plan to validate this implementation is to update the notebook that implements the mapping from lsst.verify concepts to InfluxDB and recreate the squash-demo database.

            Once the squash-demo database is created we can clone the existing dashboards and rewrite the queries.

            Show
            afausti Angelo Fausti added a comment - - edited The plan to validate this implementation is to update the notebook that implements the mapping from lsst.verify concepts to InfluxDB and recreate the squash-demo database. Once the squash-demo database is created we can clone the existing dashboards and rewrite the queries.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            squash-demo database was recreated and this change can be validated looking at the following dashboards:

            the queries now use the squash-demo database and were changed accordingly.

            Show
            afausti Angelo Fausti added a comment - squash-demo database was recreated and this change can be validated looking at the following dashboards: AP catalog metrics from CI ( DM-19605 ) DRP metrics from CI ( DM-19605 ) the queries now use the squash-demo database and were changed accordingly.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            Krzysztof Findeisen would like to review this ticket?

            You will notice the new dashboards at https://chronograf-demo.lsst.codes. I think queries and field names look less redundant now.

            I also included the "RUN CI ID" column in the tables which is a link to the CI runs and there's more metadata associated with the metric values as well.

            Look at the squash-demo database for those changes. I plan to recreate the squash-prod database after that.

            Show
            afausti Angelo Fausti added a comment - Krzysztof Findeisen would like to review this ticket? You will notice the new dashboards at https://chronograf-demo.lsst.codes . I think queries and field names look less redundant now. I also included the "RUN CI ID" column in the tables which is a link to the CI runs and there's more metadata associated with the metric values as well. Look at the squash-demo database for those changes. I plan to recreate the squash-prod database after that.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            The code changes have only minor issues. I've also reviewed the new dashboards and they look very nice; the only confusing bit is that the HSC reprocessing dashboard links to a Jira issue instead of a Jenkins build page.

            Show
            krzys Krzysztof Findeisen added a comment - The code changes have only minor issues. I've also reviewed the new dashboards and they look very nice; the only confusing bit is that the HSC reprocessing dashboard links to a Jira issue instead of a Jenkins build page.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            Thank you Krzysztof Findeisen for the review and useful comments on the notebook that exercises the mapping. Keep in mind that's prototype code and the actual implementation of the `job_to_inluxdb` task is going to be done in DM-18706

            As for the HSC reprocessing dashboard, those are "manual" runs at LDF and we don't have a RUN ID, the ticket number is the closest information and we (Hsin-Fang Chiang and I) decided to use that.

             

             

             

             

            Show
            afausti Angelo Fausti added a comment - Thank you  Krzysztof Findeisen for the review and useful comments on the notebook that exercises the mapping. Keep in mind that's prototype code and the actual implementation of the `job_to_inluxdb` task is going to be done in DM-18706 As for the HSC reprocessing dashboard, those are "manual" runs at LDF and we don't have a RUN ID, the ticket number is the closest information and we ( Hsin-Fang Chiang and I) decided to use that.        

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Reviewers:
              Krzysztof Findeisen
              Watchers:
              Angelo Fausti, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.