Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12782

travissync pull on el7-4 hung

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      The pull of the travissync image under the travissync job has been in progress on el7-4 for over 16mins while it is typically under 1min.

      There are multiple hung docker-gc (the nightly cleanup script) on that node:

      [vagrant@jenkins-el7-4 ~]$ ps -eaf | grep docker
      root       859     1  0 Sep19 ?        10:57:31 dockerd -H unix:///var/run/docker.sock --ip-forward=true --iptables=true --ip-masq=true --storage-driver=devicemapper
      jenkins+   862   856  0 Sep19 ?        12:07:29 /usr/bin/java -jar /home/jenkins-slave/swarm-client-3.3.jar -mode normal -executors 1 -username jhoblitt -passwordEnvVariable JENKINS_PASSWORD -name jenkins-el7-4 -master http://jenkins-master:8080 -labels jenkins-el7-4 centos centos-7 docker  -fsroot /home/jenkins-slave -disableClientsUniqueId -deleteExistingClients
      root       919   859  0 Sep19 ?        00:52:23 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
      root      1193  1192  0 Nov20 ?        00:00:00 /bin/bash /usr/local/bin/docker-gc
      root      1207  1193  0 Nov20 ?        00:00:00 docker ps -a -q --no-trunc
      root     20952 20950  0 04:00 ?        00:00:00 /bin/bash /usr/local/bin/docker-gc
      root     20964 20952  0 04:00 ?        00:00:00 docker ps -a -q --no-trunc
      jenkins+ 21804 21803  0 10:47 ?        00:00:00 docker pull lsstsqre/travissync:latest
      vagrant  21953 21818  0 10:54 pts/0    00:00:00 grep --color=auto docker
      

      Looking at the dmesg on that node shows xfs errors from dockerd

       
      [5320390.003393] XFS (dm-2): Unmounting Filesystem
      [5320564.453114] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5320564.457138] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5320564.461353] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5320564.466067]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5320564.470534]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5320564.474982]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5320564.479549] Call Trace:
      [5320564.481490]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5320564.484955]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5320564.488601]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5320564.492226]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5320564.495737]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5320564.499969]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5320564.503410]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5320564.507096]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5320564.510452]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5320564.513949]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5320564.517501]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5320564.520533]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5320564.523678]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5320564.526982]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5320564.530353]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5320684.533121] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5320684.537007] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5320684.540897] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5320684.544575]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5320684.548474]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5320684.552529]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5320684.556512] Call Trace:
      [5320684.558401]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5320684.561227]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5320684.564882]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5320684.568174]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5320684.571398]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5320684.574985]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5320684.578480]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5320684.581803]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5320684.584923]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5320684.588374]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5320684.591456]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5320684.594381]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5320684.597451]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5320684.600593]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5320684.603664]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5320804.606095] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5320804.611726] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5320804.616770] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5320804.621122]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5320804.625900]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5320804.630448]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5320804.637867] Call Trace:
      [5320804.641121]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5320804.645511]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5320804.650775]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5320804.655044]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5320804.659818]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5320804.664559]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5320804.668220]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5320804.671798]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5320804.675536]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5320804.679542]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5320804.682965]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5320804.686205]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5320804.689835]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5320804.693239]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5320804.696667]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5320924.699101] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5320924.702861] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5320924.706645] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5320924.710301]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5320924.714236]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5320924.718255]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5320924.722270] Call Trace:
      [5320924.724100]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5320924.726800]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5320924.730180]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5320924.733333]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5320924.736461]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5320924.739795]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5320924.743067]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5320924.746418]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5320924.749414]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5320924.752633]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5320924.755775]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5320924.758721]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5320924.761660]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5320924.764666]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5320924.767689]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5321044.770112] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5321044.773723] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5321044.777479] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5321044.781129]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5321044.785019]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5321044.788964]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5321044.792874] Call Trace:
      [5321044.794714]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5321044.797442]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5321044.800895]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5321044.804002]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5321044.807082]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5321044.810519]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5321044.813774]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5321044.816945]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5321044.819897]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5321044.823192]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5321044.826215]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5321044.828987]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5321044.831845]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5321044.834796]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5321044.837736]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5321164.840106] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5321164.843692] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5321164.847540] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5321164.851123]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5321164.854954]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5321164.858791]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5321164.862696] Call Trace:
      [5321164.864473]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5321164.867270]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5321164.870579]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5321164.873695]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5321164.876896]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5321164.882302]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5321164.887530]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5321164.890738]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5321164.893794]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5321164.897346]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5321164.900842]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5321164.903705]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5321164.906619]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5321164.909912]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5321164.913760]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5321284.917127] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5321284.921715] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5321284.926345] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5321284.930539]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5321284.934982]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5321284.939467]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5321284.943481] Call Trace:
      [5321284.945340]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5321284.948826]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5321284.952763]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5321284.957021]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5321284.961414]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5321284.966515]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5321284.970806]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5321284.974832]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5321284.978446]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5321284.982096]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5321284.986265]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5321284.990072]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5321284.994060]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5321284.997941]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5321285.002136]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5321405.005127] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5321405.010691] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5321405.015366] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5321405.019352]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5321405.024090]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5321405.028912]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5321405.034680] Call Trace:
      [5321405.036505]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5321405.039236]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5321405.042655]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5321405.046195]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5321405.049505]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5321405.053169]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5321405.056663]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5321405.061093]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5321405.065012]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5321405.069456]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5321405.072389]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5321405.075110]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5321405.077951]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5321405.080847]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5321405.083746]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5321525.086104] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5321525.089675] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5321525.093531] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5321525.097259]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5321525.101176]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5321525.104952]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5321525.108851] Call Trace:
      [5321525.110677]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5321525.113500]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5321525.116889]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5321525.120007]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5321525.123178]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5321525.126568]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5321525.129696]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5321525.133014]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5321525.135979]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5321525.139167]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5321525.142120]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5321525.144974]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5321525.147931]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5321525.150751]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5321525.153813]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      [5321645.157105] INFO: task dockerd:21004 blocked for more than 120 seconds.
      [5321645.160791] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      [5321645.164579] dockerd         D ffff880104eca690     0 21004      1 0x00000080
      [5321645.168155]  ffff8800ded8fd78 0000000000000086 ffff8803a9f18fd0 ffff8800ded8ffd8
      [5321645.171961]  ffff8800ded8ffd8 ffff8800ded8ffd8 ffff8803a9f18fd0 ffff880104eca680
      [5321645.175856]  ffff880104eca6c0 ffff8800343af000 ffff880104eca6e8 ffff880104eca690
      [5321645.179745] Call Trace:
      [5321645.181498]  [<ffffffff816a94e9>] schedule+0x29/0x70
      [5321645.184232]  [<ffffffffc01882ea>] xfs_ail_push_all_sync+0xba/0x110 [xfs]
      [5321645.187768]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
      [5321645.190785]  [<ffffffffc01712e1>] xfs_unmountfs+0x71/0x1c0 [xfs]
      [5321645.193830]  [<ffffffffc0171ded>] ? xfs_mru_cache_destroy+0x6d/0xa0 [xfs]
      [5321645.197306]  [<ffffffffc0173e92>] xfs_fs_put_super+0x32/0x90 [xfs]
      [5321645.200573]  [<ffffffff81203722>] generic_shutdown_super+0x72/0x100
      [5321645.203726]  [<ffffffff81203b67>] kill_block_super+0x27/0x70
      [5321645.206806]  [<ffffffff81203ea9>] deactivate_locked_super+0x49/0x60
      [5321645.210196]  [<ffffffff81204616>] deactivate_super+0x46/0x60
      [5321645.213207]  [<ffffffff8122184f>] cleanup_mnt+0x3f/0x80
      [5321645.216024]  [<ffffffff812218e2>] __cleanup_mnt+0x12/0x20
      [5321645.219053]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
      [5321645.222011]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
      [5321645.224966]  [<ffffffff816b52bd>] int_signal+0x12/0x17
      

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            This is either some sort of EBS / EBS optimized networking lurch or the kernel state has corrupted (or the prior caused the later). I'm going to try rebooting the node.

            Show
            jhoblitt Joshua Hoblitt added a comment - This is either some sort of EBS / EBS optimized networking lurch or the kernel state has corrupted (or the prior caused the later). I'm going to try rebooting the node.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            And it won't reboot from the CLI via reboot --force. I'm going to restart the EC2 instance.

            Show
            jhoblitt Joshua Hoblitt added a comment - And it won't reboot from the CLI via reboot --force . I'm going to restart the EC2 instance.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            There are similar xfs errors in the dmesg on el7-1 and el7-3. el6-2 has the OOM being invoked in the dmesg but I don't think that is related.

            Show
            jhoblitt Joshua Hoblitt added a comment - There are similar xfs errors in the dmesg on el7-1 and el7-3 . el6-2 has the OOM being invoked in the dmesg but I don't think that is related.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            All 3 el7 nodes with the xfs dmesg entries have had the ec2 instance forcibly stopped and re-launched.

            Show
            jhoblitt Joshua Hoblitt added a comment - All 3 el7 nodes with the xfs dmesg entries have had the ec2 instance forcibly stopped and re-launched.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I manually initiated a build of travissync, which completed in 2s. Hopefully, the docker (layer)/ xfs issues are resolved. If this reoccurs, the next step is likely to move to a newer kernel.

            Show
            jhoblitt Joshua Hoblitt added a comment - I manually initiated a build of travissync , which completed in 2s. Hopefully, the docker (layer)/ xfs issues are resolved. If this reoccurs, the next step is likely to move to a newer kernel.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel