Joshua Hoblitt added a comment - 24/Jun/19 4:32 PM Trying to delete the pod had no change in status after ~5minutes. Force deleting it got it into the pending state, but the pod is unschedulable as not enough usable nodes/capacity is online. IHS-2333 has been opened to report that lsst-kub016 is down.   [jhoblitt @lsst -bastion01 ~]$ kubectl -n jenkins-prod delete pod agent-ldfc- 0 --force --grace-period 0 warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "agent-ldfc-0" force deleted [jhoblitt @lsst -bastion01 ~]$ kubectl -n jenkins-prod get pod NAME READY STATUS RESTARTS AGE agent-ldfc- 0 0 / 3 Pending 0 6s agent-ldfc- 1 3 / 3 Running 0 22h agent-ldfc- 2 3 / 3 Running 0 22h agent-ldfc- 3 3 / 3 Running 0 22h agent-ldfc- 4 3 / 3 Running 0 22h agent-ldfc- 5 3 / 3 Running 0 22h [jhoblitt @lsst -bastion01 ~]$ kubectl -n jenkins-prod describe pod agent-ldfc- 0 Name: agent-ldfc- 0 Namespace: jenkins-prod Priority: 0 PriorityClassName: <none> Node: <none> Labels: app.k8s.io/component=agent app.k8s.io/instance=prod app.k8s.io/managed-by=terraform app.k8s.io/name=agent-ldfc app.k8s.io/part-of=jenkins app.k8s.io/version= 1.0 . 0 controller-revision-hash=agent-ldfc-748c96c6f6 statefulset.kubernetes.io/pod-name=agent-ldfc- 0 Annotations: kubernetes.io/psp: privileged Status: Pending IP: Controlled By: StatefulSet/agent-ldfc Containers: dind: Image: lsstsqre/dind: 18.09 . 5 Port: <none> Host Port: <none> Command: /usr/local/bin/dockerd Args: --host=tcp: //localhost:2375 --mtu= 1376 Limits: cpu: 32 memory: 96Gi Requests: cpu: 6 memory: 12Gi Liveness: exec [wget --spider -q http: //localhost:2375/_ping] delay=5s timeout=1s period=5s #success=1 #failure=2 Readiness: exec [wget --spider -q http: //localhost:2375/_ping] delay=5s timeout=1s period=5s #success=1 #failure=2 Environment: DOCKER_HOST: tcp: //localhost:2375 Mounts: /j from ws (rw) /var/lib/docker from docker-graph-storage (rw) /var/run/secrets/kubernetes.io/serviceaccount from default -token-nkqvn (ro) docker-gc: Image: lsstsqre/docker-gc:latest Port: <none> Host Port: <none> Command: sh -c while true ; do /usr/local/bin/docker-gc; sleep $GRACE_PERIOD_SECONDS; done Limits: cpu: 500m memory: 512Mi Requests: cpu: 200m memory: 100Mi Environment: DOCKER_HOST: tcp: //localhost:2375 GRACE_PERIOD_SECONDS: 3600 MINIMUM_IMAGES_TO_SAVE: 5 REMOVE_VOLUMES: 1 FORCE_CONTAINER_REMOVAL: 1 FORCE_IMAGE_REMOVAL: 1 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default -token-nkqvn (ro) swarm: Image: lsstsqre/jenkins-swarm-client: 3.15 -ldfc Port: <none> Host Port: <none> Limits: cpu: 2 memory: 3Gi Requests: cpu: 1 memory: 2Gi Liveness: exec [wget --spider -q http: //localhost:8080/metrics] delay=5s timeout=1s period=5s #success=1 #failure=2 Readiness: exec [wget --spider -q http: //localhost:8080/metrics] delay=5s timeout=1s period=5s #success=1 #failure=2 Environment: DOCKER_HOST: tcp: //localhost:2375 JSWARM_MASTER_URL: https: //ci.lsst.codes JSWARM_MODE: normal JSWARM_LABELS: docker ldfc JSWARM_EXECUTORS: 1 JSWARM_AGENT_NAME: agent-ldfc- 0 (v1:metadata.name) JSWARM_DISABLE_CLIENTS_UNIQUE_ID: true JSWARM_DELETE_EXISTING_CLIENTS: true JSWARM_USERNAME: <set to the key 'JSWARM_USERNAME' in secret 'agent-ldfc' > Optional: false JSWARM_PASSWORD: <set to the key 'JSWARM_PASSWORD' in secret 'agent-ldfc' > Optional: false Mounts: /j from ws (rw) /var/run/secrets/kubernetes.io/serviceaccount from default -token-nkqvn (ro) Conditions: Type Status PodScheduled False Volumes: ws: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: ws-agent-ldfc- 0 ReadOnly: false docker-graph-storage: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> default -token-nkqvn: Type: Secret (a volume populated by a Secret) SecretName: default -token-nkqvn Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 73s (x10 over 6m44s) default -scheduler 0 / 42 nodes are available: 1 node(s) didn 't match node selector, 36 node(s) had taints that the pod didn' t tolerate, 5 Insufficient cpu.