Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31998

Investigate rubin-tv performance issues

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      rubin-tv seems to have very slow (15 s, and more) response times when deployed to Roundtable. The purpose of this work is to investigate and hopefully resolve rubin-tv's performance.

        Attachments

          Activity

          Hide
          Permalink
          jsick Jonathan Sick added a comment -

          This PR resolves performance issues seen in Rubin TV's Roundtable deployment. Because querying all objects in the GCS bucket is fairly slow (>1 second), multiple requests can pile up and easily trigger what is effectively denial-of-service attack. The fixes:

          • The auto-refresh JavaScript was set to 0.5 seconds. Here I've backed off the refresh period to 10 seconds to prevent a single client from creating a backlog of requests.
          • I've updated the Kubernetes deployment with a replication factor of 5. More replicas mean that more pods can service requests at the same time.
          • I noticed that rubintv wasn't logging anything; I've added a simple Timer context manager and a log statement to each request handler to make it easier to monitor requests and response times.
          Show
          jsick Jonathan Sick added a comment - This PR resolves performance issues seen in Rubin TV's Roundtable deployment. Because querying all objects in the GCS bucket is fairly slow (>1 second), multiple requests can pile up and easily trigger what is effectively denial-of-service attack. The fixes: The auto-refresh JavaScript was set to 0.5 seconds. Here I've backed off the refresh period to 10 seconds to prevent a single client from creating a backlog of requests. I've updated the Kubernetes deployment with a replication factor of 5. More replicas mean that more pods can service requests at the same time. I noticed that rubintv wasn't logging anything; I've added a simple Timer context manager and a log statement to each request handler to make it easier to monitor requests and response times.

            People

            Assignee:
            jsick Jonathan Sick
            Reporter:
            jsick Jonathan Sick
            Reviewers:
            Simon Krughoff
            Watchers:
            Jonathan Sick, Simon Krughoff
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.