Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20582

unable to rebuild jenkins builds

    Details

      Description

      Per Nate Lust

      https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30116/pipeline
       
      that is the failed job
      if I click rebuild in there it does not work, those it seems to indicate it does
      same if I go here https://ci.lsst.codes/job/stack-os-matrix/30116/rebuild/parameterized
      

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          INFO: stack-os-matrix #30122 completed: SUCCESS
          Jul 16, 2019 12:41:19 PM hudson.node_monitors.AbstractDiskSpaceMonitor markNodeOfflineIfDiskspaceIsTooLow
          WARNING: Making master offline temporarily due to the lack of disk space
          Jul 16, 2019 12:41:19 PM hudson.node_monitors.AbstractDiskSpaceMonitor markNodeOfflineIfDiskspaceIsTooLow
          WARNING: Making jenkins-master offline temporarily due to the lack of disk space
          Jul 16, 2019 12:58:02 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
          INFO: stack-os-matrix #30116 completed: FAILURE
          Jul 16, 2019 1:50:06 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@1d333371[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 1:51:31 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@172cd276[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 1:52:13 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@29e1af3c[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 1:52:30 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@7eb456fa[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 1:52:44 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@1b854aba[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 1:52:57 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@3a6f6230[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 1:53:48 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@45167b61[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 1:55:03 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@c9b658a[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 2:02:13 PM hudson.util.Secret toString
          WARNING: Use of toString() on hudson.util.Secret from java.lang.String.valueOf(String.java:2994). Prefer getPlainText() or getEncryptedValue() depending your needs. see https://jenkins.io/redirect/hudson.util.Secret/
          Jul 16, 2019 2:02:13 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
          WARNING: Invalid cookie header: "Set-Cookie: has_recent_activity=1; path=/; expires=Tue, 16 Jul 2019 22:02:13 -0000". Invalid 'expires' attribute: Tue, 16 Jul 2019 22:02:13 -0000
          Jul 16, 2019 2:02:36 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@553f8d10[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 2:03:11 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@537f7d19[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 2:08:42 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
          INFO: stack-os-matrix #30120 completed: SUCCESS
          Jul 16, 2019 2:09:09 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@9d6aa3c[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 2:09:34 PM hudson.model.Executor resetWorkUnit
          WARNING: Executor #-1 for master grabbed hudson.model.queue.WorkUnit@36415a28[work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available.
          Jul 16, 2019 2:10:05 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
          INFO: stack-os-matrix #30121 completed: SUCCESS
          
          

          Show
          jhoblitt Joshua Hoblitt added a comment - INFO: stack-os-matrix # 30122 completed: SUCCESS Jul 16 , 2019 12 : 41 : 19 PM hudson.node_monitors.AbstractDiskSpaceMonitor markNodeOfflineIfDiskspaceIsTooLow WARNING: Making master offline temporarily due to the lack of disk space Jul 16 , 2019 12 : 41 : 19 PM hudson.node_monitors.AbstractDiskSpaceMonitor markNodeOfflineIfDiskspaceIsTooLow WARNING: Making jenkins-master offline temporarily due to the lack of disk space Jul 16 , 2019 12 : 58 : 02 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: stack-os-matrix # 30116 completed: FAILURE Jul 16 , 2019 1 : 50 : 06 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @1d333371 [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 1 : 51 : 31 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @172cd276 [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 1 : 52 : 13 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @29e1af3c [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 1 : 52 : 30 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @7eb456fa [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 1 : 52 : 44 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @1b854aba [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 1 : 52 : 57 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @3a6f6230 [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 1 : 53 : 48 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @45167b61 [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 1 : 55 : 03 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @c9b658a [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 2 : 02 : 13 PM hudson.util.Secret toString WARNING: Use of toString() on hudson.util.Secret from java.lang.String.valueOf(String.java: 2994 ). Prefer getPlainText() or getEncryptedValue() depending your needs. see https: //jenkins.io/redirect/hudson.util.Secret/ Jul 16 , 2019 2 : 02 : 13 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies WARNING: Invalid cookie header: "Set-Cookie: has_recent_activity=1; path=/; expires=Tue, 16 Jul 2019 22:02:13 -0000" . Invalid 'expires' attribute: Tue, 16 Jul 2019 22 : 02 : 13 - 0000 Jul 16 , 2019 2 : 02 : 36 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @553f8d10 [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 2 : 03 : 11 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @537f7d19 [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 2 : 08 : 42 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: stack-os-matrix # 30120 completed: SUCCESS Jul 16 , 2019 2 : 09 : 09 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @9d6aa3c [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 2 : 09 : 34 PM hudson.model.Executor resetWorkUnit WARNING: Executor #- 1 for master grabbed hudson.model.queue.WorkUnit @36415a28 [work=stack-os-matrix] from queue but master went off-line before the task's worker thread started. No termination trace available. Jul 16 , 2019 2 : 10 : 05 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: stack-os-matrix # 30121 completed: SUCCESS
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The issue was systemic for all builds as the jenkins master was low on disk space. As a short term fix, I've removed some of the build history for the s3backup-eups-daily-cron job.

          [root@jenkins-master jobs]# du -sh *
          20M	build-ec2-snapshot
          9.9M	build-mysqldump-to-s3
          15M	build-s3backup
          21M	build-sqre-github-snapshot
          224M	jenkins-ebs-snapshot
          431M	nightly-sqre-github-snapshot
          76G	s3backup-eups-daily-cron
          2.6G	s3backup-eups-monthly-cron
           
          [root@jenkins-master jobs]# du -sh *
          20M	build-ec2-snapshot
          9.9M	build-mysqldump-to-s3
          15M	build-s3backup
          21M	build-sqre-github-snapshot
          224M	jenkins-ebs-snapshot
          431M	nightly-sqre-github-snapshot
          7.1G	s3backup-eups-daily-cron
          2.6G	s3backup-eups-monthly-cron
          
          

          Show
          jhoblitt Joshua Hoblitt added a comment - The issue was systemic for all builds as the jenkins master was low on disk space. As a short term fix, I've removed some of the build history for the s3backup-eups-daily-cron job. [root @jenkins -master jobs]# du -sh * 20M build-ec2-snapshot 9 .9M build-mysqldump-to-s3 15M build-s3backup 21M build-sqre-github-snapshot 224M jenkins-ebs-snapshot 431M nightly-sqre-github-snapshot 76G s3backup-eups-daily-cron 2 .6G s3backup-eups-monthly-cron   [root @jenkins -master jobs]# du -sh * 20M build-ec2-snapshot 9 .9M build-mysqldump-to-s3 15M build-s3backup 21M build-sqre-github-snapshot 224M jenkins-ebs-snapshot 431M nightly-sqre-github-snapshot 7 .1G s3backup-eups-daily-cron 2 .6G s3backup-eups-monthly-cron
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          This was resolved by freeing up disk space.

          Show
          jhoblitt Joshua Hoblitt added a comment - This was resolved by freeing up disk space.

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Watchers:
              Joshua Hoblitt
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: