Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16399

Backfill ap_verify CI jobs into SQuaSH demo instance

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Related to DM-16333, we want to retroactively load the ap_verify CI jobs into SQuaSH to have the complete history of these runs.

      With this ticket we'll also test a change introduced by DM-16300 into the SQuaSH API. Now verify metrics are also sent to InfluxDB and can be visualized using the Chronograf UI.

      We'll first do this on the SQuaSH demo instance and then replicate to production in another ticket when DM-16333 is complete.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment -

            Ignoring metrics that are not defined in verify_metrics more info in DM-16319.

            Show
            afausti Angelo Fausti added a comment - Ignoring metrics that are not defined in verify_metrics more info in DM-16319 .
            Hide
            afausti Angelo Fausti added a comment - - edited

            Simon Krughoff Data is flowing from ap_verify CI job to InfluxDB through the SQuaSH API.

            I've made a simple dashboard in Chronograf (screenshot attached) to illustrate the kind of aggregations we can now do in InfluxDB since we annotate the metric values with data ID tags (visit, ccdnum, filter).

            The dashboard can be reached here.

            Show
            afausti Angelo Fausti added a comment - - edited Simon Krughoff Data is flowing from ap_verify CI job to InfluxDB through the SQuaSH API. I've made a simple dashboard in Chronograf (screenshot attached) to illustrate the kind of aggregations we can now do in InfluxDB since we annotate the metric values with data ID tags (visit, ccdnum, filter). The dashboard can be reached here .
            Hide
            afausti Angelo Fausti added a comment -

            Added example notebook to reconstruct ap_verify verification jobs from Jenkins runs. This is how the data shown in Chronograf was loaded.

            See PR https://github.com/lsst-sqre/squash-restful-api/pull/39

            Show
            afausti Angelo Fausti added a comment - Added example notebook to reconstruct ap_verify verification jobs from Jenkins runs. This is how the data shown in Chronograf was loaded. See PR https://github.com/lsst-sqre/squash-restful-api/pull/39
            Hide
            afausti Angelo Fausti added a comment -

            Simon Krughoff this was the notebook I used to reconstruct the ap_verify verification jobs from Jenkins runs and send to SQuaSH.

            I've used the SQuaSH demo instance which is connected to InlfuxDB. Thus, the results are also available in Chronograf.

            We can repeat this in production to retroactively load the ap_verify runs in CI when DM-16333 is complete.

            Show
            afausti Angelo Fausti added a comment - Simon Krughoff this was the notebook I used to reconstruct the ap_verify verification jobs from Jenkins runs and send to SQuaSH. I've used the SQuaSH demo instance which is connected to InlfuxDB. Thus, the results are also available in Chronograf. We can repeat this in production to retroactively load the ap_verify runs in CI when DM-16333 is complete.
            Hide
            krughoff Simon Krughoff added a comment -

            Looks great!

            Show
            krughoff Simon Krughoff added a comment - Looks great!

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Reviewers:
                Simon Krughoff
                Watchers:
                Angelo Fausti, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel