Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13236

Create the /monitor resource in the Flask REST API to feed the monitor app

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Now that the new API is alive we need the following resources that are consumed by the SQuaSH bokeh apps:

      • measurements -> monitor app
      • code changes -> monitor app
      • stats -> dashboard initial page
      • blobs > AM1, PA1, etc apps

      NOTE: the performance on retrieving the data from the new implementation will be evaluate only after DM-12604 is done

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment -

            The returned by the monitor resource can be constrained by dataset and period, those are the main queries. Other information like package name and metric name are loaded to memory and filtered by the bokeh application.

            Tried to optimize queries in SQLAlchemy as much as possible. For instance:

             
            http://localhost:5000/monitor?ci_dataset=cfht
            
            

            produces an SQL query like this:

            SELECT measurement.value AS measurement_value, measurement.metric_name AS measurement_metric_name, metric.package AS metric_package, job.date_created AS job_date_created, job.ci_dataset AS job_ci_dataset
            FROM measurement INNER JOIN job ON job.id = measurement.job_id INNER JOIN metric ON metric.id = measurement.metric_id
            WHERE job.ci_dataset =  %(ci_dataset_1)s ORDER BY job.date_created ASC
             
            {'ci_dataset_1': 'cfht'}
            

            and

            http://localhost:5000/monitor?ci_dataset=cfht&period=Last%20Month
            
            

            produces an SQL query like this:

            SELECT measurement.value AS measurement_value, measurement.metric_name AS measurement_metric_name, metric.package AS metric_package, job.date_created AS job_date_created, job.ci_dataset AS job_ci_dataset
            FROM measurement INNER JOIN job ON job.id = measurement.job_id INNER JOIN metric ON metric.id = measurement.metric_id
            WHERE job.ci_dataset = %(ci_dataset_1)s AND job.date_created > %(date_created_1)s ORDER BY job.date_created ASC
             
            {'ci_dataset_1': 'cfht', 'date_created_1': datetime.datetime(2018, 1, 8, 8, 10, 57, 578279)}
            

            Show
            afausti Angelo Fausti added a comment - The returned by the monitor resource can be constrained by dataset and period, those are the main queries. Other information like package name and metric name are loaded to memory and filtered by the bokeh application. Tried to optimize queries in SQLAlchemy as much as possible. For instance:   http: //localhost:5000/monitor?ci_dataset=cfht produces an SQL query like this: SELECT measurement.value AS measurement_value, measurement.metric_name AS measurement_metric_name, metric.package AS metric_package, job.date_created AS job_date_created, job.ci_dataset AS job_ci_dataset FROM measurement INNER JOIN job ON job.id = measurement.job_id INNER JOIN metric ON metric.id = measurement.metric_id WHERE job.ci_dataset = %(ci_dataset_1)s ORDER BY job.date_created ASC   { 'ci_dataset_1' : 'cfht' } and http: //localhost:5000/monitor?ci_dataset=cfht&period=Last%20Month produces an SQL query like this: SELECT measurement.value AS measurement_value, measurement.metric_name AS measurement_metric_name, metric.package AS metric_package, job.date_created AS job_date_created, job.ci_dataset AS job_ci_dataset FROM measurement INNER JOIN job ON job.id = measurement.job_id INNER JOIN metric ON metric.id = measurement.metric_id WHERE job.ci_dataset = %(ci_dataset_1)s AND job.date_created > %(date_created_1)s ORDER BY job.date_created ASC   { 'ci_dataset_1' : 'cfht' , 'date_created_1' : datetime.datetime(2018, 1, 8, 8, 10, 57, 578279)}
            Show
            afausti Angelo Fausti added a comment - See PR https://github.com/lsst-sqre/squash-rest-api/pull/14

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Watchers:
                Angelo Fausti
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel