Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17767

Mapping lsst.verify run time to InfluxDB

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • squash
    • None

    Description

      There are at least three options of timestamps that we can use to map the lsst.verify run time to InfluxDB.

      • CI pipeline run time
      • Verification Job time
      • InfluxDB creation time

      We are currently using the "Verification Job time" which turns not to be the right choice because we are running a different verification job for each dataID. As consequence, results that should have the same timestamp get inserted in InfluxDB at different times. A better choice for the InfluxDB timestamp seems to be the CI pipeline run time.

      Attachments

        Issue Links

          Activity

            In the PR https://github.com/lsst-sqre/squash-restful-api/pull/45 we updated the influxdb_data_model.ipynb notebook to use the CI pipeline runtime as the InfluxDB timestamp.

            The squash-demo database was re-created and we showed that aggregation of metric values across multiple data ids work just fine, using mean() and GROUP BY time() InfluxQL functions.

            afausti Angelo Fausti added a comment - In the PR https://github.com/lsst-sqre/squash-restful-api/pull/45 we updated the influxdb_data_model.ipynb notebook to use the CI pipeline runtime as the InfluxDB timestamp. The squash-demo database was re-created and we showed that aggregation of metric values across multiple data ids work just fine, using mean() and GROUP BY time() InfluxQL functions.
            afausti Angelo Fausti added a comment - - edited

            For the record the query to display metric values per ccd:

            SELECT "ap_association.totalUnassociatedDiaObjects" FROM "squash-demo"."autogen"."ap_association" WHERE time > :dashboardTime: AND ("ccdnum"='10' OR "ccdnum"='5' OR "ccdnum"='56' OR "ccdnum"='60') GROUP BY "ccdnum"
            

            and the query to aggregate metric values across multiple ccds:

            SELECT mean("ap_association.totalUnassociatedDiaObjects") AS "mean_ap_association.totalUnassociatedDiaObjects" FROM "squash-demo"."autogen"."ap_association" WHERE time > :dashboardTime: GROUP BY time(1d)
            

            Results for this aggregation test are currently available under the SQuaRE organization in Chronograf.

            afausti Angelo Fausti added a comment - - edited For the record the query to display metric values per ccd: SELECT "ap_association.totalUnassociatedDiaObjects" FROM "squash-demo"."autogen"."ap_association" WHERE time > :dashboardTime: AND ("ccdnum"='10' OR "ccdnum"='5' OR "ccdnum"='56' OR "ccdnum"='60') GROUP BY "ccdnum" and the query to aggregate metric values across multiple ccds: SELECT mean("ap_association.totalUnassociatedDiaObjects") AS "mean_ap_association.totalUnassociatedDiaObjects" FROM "squash-demo"."autogen"."ap_association" WHERE time > :dashboardTime: GROUP BY time(1d) Results for this aggregation test are currently available under the SQuaRE organization in Chronograf.

            krughoff would like to have a look at these results in Chronograf?

            afausti Angelo Fausti added a comment - krughoff would like to have a look at these results in Chronograf?

            I will have a look.

            krughoff Simon Krughoff (Inactive) added a comment - I will have a look.

            Looks great.

            krughoff Simon Krughoff (Inactive) added a comment - Looks great.

            People

              afausti Angelo Fausti
              afausti Angelo Fausti
              Simon Krughoff (Inactive)
              Angelo Fausti, John Swinbank, Simon Krughoff (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.