Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9875

Mitigation for memory issues on Kubernetes

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      ltdstatus (and, when kubified, probably the indexer) hang on GKE. I think it's a memory condition, because they're doing a lot of threads in parallel.

      Two things I want to try: 1) increase the size of the cluster instances a notch, 2) put a limiter on number of simultaneous threads spawned in our scatter/gather.

        Attachments

          Activity

          Hide
          athornton Adam Thornton added a comment -

          I've respun the cluster on 4GB rather than 1.7GB instances, which lets ltdstatus complete successfully. I am going to experiment whether that, on its own, is enough to let us move ccutter to uwsgi again.

          I don't think I'm going to try a threadpool implementation at this point, because it looks like it's going to be tricky to get right and this may be enough amelioration.

          Show
          athornton Adam Thornton added a comment - I've respun the cluster on 4GB rather than 1.7GB instances, which lets ltdstatus complete successfully. I am going to experiment whether that, on its own, is enough to let us move ccutter to uwsgi again. I don't think I'm going to try a threadpool implementation at this point, because it looks like it's going to be tricky to get right and this may be enough amelioration.
          Hide
          athornton Adam Thornton added a comment -

          All services running under uwsgi and completing now, even for many-threaded request storms.

          Show
          athornton Adam Thornton added a comment - All services running under uwsgi and completing now, even for many-threaded request storms.
          Hide
          frossie Frossie Economou added a comment -


          That should do it, thanks.

          Show
          frossie Frossie Economou added a comment - That should do it, thanks.

            People

            • Assignee:
              athornton Adam Thornton
              Reporter:
              athornton Adam Thornton
              Reviewers:
              Frossie Economou
              Watchers:
              Adam Thornton, Frossie Economou
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel