Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10259

SQuaSH API optmization using Django querysets with Pandas

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

        Issue Links

          Activity

          Hide
          afausti Angelo Fausti added a comment - - edited

          The following method is used to optimize the queries in the SQuaSH API and to convert the django queryset to a pandas dataframe so that it can be used directly to make the datasources for the bokeh apps.

          https://www.iwoca.co.uk/blog/2016/09/02/using-pandas-django-faster/

          Show
          afausti Angelo Fausti added a comment - - edited The following method is used to optimize the queries in the SQuaSH API and to convert the django queryset to a pandas dataframe so that it can be used directly to make the datasources for the bokeh apps. https://www.iwoca.co.uk/blog/2016/09/02/using-pandas-django-faster/
          Hide
          afausti Angelo Fausti added a comment - - edited

          That's the performance results for the API endpoint that return data to feed the bokeh apps in squash, the data blobs (~50k data points) and measurement metadata after filtering by ci_id, ci_dataset and metric.

          Show
          afausti Angelo Fausti added a comment - - edited That's the performance results for the API endpoint that return data to feed the bokeh apps in squash, the data blobs (~50k data points) and measurement metadata after filtering by ci_id , ci_dataset and metric .
          Hide
          afausti Angelo Fausti added a comment -
          Show
          afausti Angelo Fausti added a comment - Example of API URLs, using default values: http://localhost:8000/dashboard/api/apps/?bokeh_app=pax and filtering by ci_id and ci_dataset http://localhost:8000/dashboard/api/apps/?bokeh_app=pax&ci_id=841&ci_dataset=hsc
          Hide
          afausti Angelo Fausti added a comment -

          Doesn't make sent anymore since the SQuaSH RETful API was reimplemented in Flask. However a similar method was implemented for some resources of the new API (DM-13236), they return arrays in JSON that can be converted do Pandas dataframe.

          Show
          afausti Angelo Fausti added a comment - Doesn't make sent anymore since the SQuaSH RETful API was reimplemented in Flask. However a similar method was implemented for some resources of the new API ( DM-13236 ), they return arrays in JSON that can be converted do Pandas dataframe.

            People

            • Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel