Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9823

Report: Nebula VM Filesystem I/O problem (logging.lsst.codes es-[1-4,6].lsst.codes)

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Team:
      Data Facility

      Description

      The Panopticon ES cluster had issues with their filesystem I/O yesterday and today. I did a hard reboot on each machine followed by an Ansible role to restart the cluster. This eventually resolved this issue.

      Machines involved:
      p-es-1-jmatt b16f725f-f86b-4349-8d9e-f07e63665029
      p-es-2-jmatt 5593f930-4613-4420-831c-2029fddd92cc
      p-es-3-jmatt 62cda1e4-7c00-4817-9bea-8deb057b3032
      p-es-4-jmatt d0f1f1be-3e26-4d4e-b821-244e7d903b5b
      p-es-k-jmatt (p-es-5-jmatt) 6dfaab48-aadf-4135-b933-23844685922b
      p-es-6-jmatt 1e7c2b7d-354f-45e0-b17c-bb761f03a2c0

      han:vagrant-nebula-panopticon jmatt$ vagrant ssh p-es-k
      Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-21-generic x86_64)
       
       * Documentation:  https://help.ubuntu.com
       * Management:     https://landscape.canonical.com
       * Support:        https://ubuntu.com/advantage
      Last login: Mon Feb 27 05:44:14 2017 from 24.255.16.161
      vagrant@logging:~$ sudo su -
      sudo: unable to execute /bin/su: Input/output error
      vagrant@logging:~$ df
      -bash: /bin/df: Input/output error
      vagrant@logging:~$ du
      -bash: /usr/bin/du: Input/output error
      vagrant@logging:~$ dig logging.lsst.codes
      -bash: /usr/bin/dig: Input/output error
      vagrant@logging:~$ exit
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jmatt J Matt Peterson [X] (Inactive)
                Reviewers:
                J Matt Peterson [X] (Inactive)
                Watchers:
                Frossie Economou, J Matt Peterson [X] (Inactive), Michelle Butler
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel