Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20252

jenkins docker builds failing on agent-ldfc-3/lsst-kub017

    XMLWordPrintable

    Details

      Description

      Per Adam Thornton, builds on agent-ldfc-3 are failing with:

      https://ci.lsst.codes/job/sqre/job/infra/job/build-sciplatlab/179/console

      OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:297: getting the final child's pid from pipe caused \"EOF\"": unknown

        Attachments

          Activity

          Descending order - Click to sort in ascending order
          Hide
          Permalink
          jhoblitt Joshua Hoblitt added a comment -

          The rolling update of the statefulset has completed.

          $ kubectl -n jenkins-prod get pods 
          NAME           READY   STATUS    RESTARTS   AGE
          		 
          agent-ldfc-0   3/3     Running   0          107s
          		 
          agent-ldfc-1   3/3     Running   0          4m3s
          		 
          agent-ldfc-2   3/3     Running   0          10m
          		 
          agent-ldfc-3   3/3     Running   0          12m
          		 
          agent-ldfc-4   3/3     Running   0          36m
          		 
          agent-ldfc-5   3/3     Running   0          41m

          Show
          jhoblitt Joshua Hoblitt added a comment - The rolling update of the statefulset has completed. $ kubectl -n jenkins-prod get pods NAME READY STATUS RESTARTS AGE agent-ldfc- 0 3 / 3 Running 0 107s agent-ldfc- 1 3 / 3 Running 0 4m3s agent-ldfc- 2 3 / 3 Running 0 10m agent-ldfc- 3 3 / 3 Running 0 12m agent-ldfc- 4 3 / 3 Running 0 36m agent-ldfc- 5 3 / 3 Running 0 41m
          Hide
          Permalink
          jhoblitt Joshua Hoblitt added a comment -

          It appears that the jenkins pods are restarted (for unknown reasons) frequently enough that merely leaving the agent-ldfc-3 agent manually offline isn't likely to be a robust fix. I'm adding anti-affinity for lsst-kub017. However, this is unfortunately going to a kubernetes to restart all of the pods spawned by the statefulset.

          $ kubectl -n jenkins-prod get pods -w
          		 
          NAME READY STATUS RESTARTS AGE
          		 
          agent-ldfc-0 3/3 Running 2 29d
          		 
          agent-ldfc-1 3/3 Running 0 19d
          		 
          agent-ldfc-2 3/3 Running 1 29d
          		 
          agent-ldfc-3 3/3 Running 3 29d
          		 
          agent-ldfc-4 3/3 Running 5 29d
          		 
          agent-ldfc-5 3/3 Terminating 2 29d

          Show
          jhoblitt Joshua Hoblitt added a comment - It appears that the jenkins pods are restarted (for unknown reasons) frequently enough that merely leaving the agent-ldfc-3 agent manually offline isn't likely to be a robust fix. I'm adding anti-affinity for lsst-kub017 . However, this is unfortunately going to a kubernetes to restart all of the pods spawned by the statefulset. $ kubectl -n jenkins-prod get pods -w NAME READY STATUS RESTARTS AGE agent-ldfc- 0 3 / 3 Running 2 29d agent-ldfc- 1 3 / 3 Running 0 19d agent-ldfc- 2 3 / 3 Running 1 29d agent-ldfc- 3 3 / 3 Running 3 29d agent-ldfc- 4 3 / 3 Running 5 29d agent-ldfc- 5 3 / 3 Terminating 2 29d

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jhoblitt Joshua Hoblitt
            Watchers:
            Joshua Hoblitt
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: