Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18516

CLONE - TSSW Jenkins executor is offline

    Details

      Description

      Hi Josh,

      It seems that Jenkins out of space again, but I think we got the to bottom of the problem (please see the email in the comments below).

      Is there a way to clear this up and get it working again?

      If you need anything from me or Eric Coughlin, let us know.  Rob Bovill is out this week.

      Thanks!

      Andy

       

      The TSSW Jenkins agent went offline sometime overnight when the 50GB disk space alert triggered.

       Disk space is too low. Only 51.141GB left on /j.

       

       

        Attachments

          Issue Links

            Activity

            Hide
            aclements Andy Clements added a comment - - edited

            Hi Andy and Rob,

            I am deeply sorry for this, the polling strategy was every minute because the Job was supposed to be triggered only when there is a new commit in the repo.
            Anyway, we reduced the poll frequency to daily on Friday and now we removed the poll trigger from all our repos. During the day we will change the configuration and use Github hooks as Rob suggested.
             
            However, right now the jobs are not running (there a re a few in queue). It seems the machine's disk is full:
             
            Unable to render embedded object: File (Hl01LkkAAAAAElFTkSuQmCC) not found.
            I understand Rob is on vacation, so Andy please let me know if I need to contact someone about this, or if you can do it.
             
            Sorry for all the trouble.
            Let me know if I can help in any way.

            Kind regards,

             
             

            Sebastián Fehlandt
            Show
            aclements Andy Clements added a comment - - edited Hi Andy and Rob, I am deeply sorry for this, the polling strategy was every minute because the Job was supposed to be triggered only when there is a new commit in the repo. Anyway, we reduced the poll frequency to daily on Friday and now we removed the poll trigger from all our repos. During the day we will change the configuration and use Github hooks as Rob suggested.   However, right now the jobs are not running (there a re a few in queue). It seems the machine's disk is full:   Unable to render embedded object: File (Hl01LkkAAAAAElFTkSuQmCC) not found. I understand Rob is on vacation, so Andy please let me know if I need to contact someone about this, or if you can do it.   Sorry for all the trouble. Let me know if I can help in any way. Kind regards,     Sebastián Fehlandt
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've cleaned up el7-1 again. These were the workspaces on disk:

            SAL_Release    ts_sal_ws-cleanup_1552434293109  ts_sal_ws-cleanup_1552496238573  ts_sal_ws-cleanup_1552517957310  ts_sal_ws-cleanup_1552594917056
            TMA_Simulator  ts_sal_ws-cleanup_1552444794218  ts_sal_ws-cleanup_1552496712674  ts_sal_ws-cleanup_1552519920919  ts_sal_ws-cleanup_1552669643187
            ts_sal         ts_sal_ws-cleanup_1552495340510  ts_sal_ws-cleanup_1552497953258  ts_sal_ws-cleanup_1552583005429  ts_sal_ws-cleanup_1552671917899
            

            DM-18114 should prevent this issue from happening.

            Show
            jhoblitt Joshua Hoblitt added a comment - I've cleaned up el7-1 again. These were the workspaces on disk: SAL_Release ts_sal_ws-cleanup_1552434293109 ts_sal_ws-cleanup_1552496238573 ts_sal_ws-cleanup_1552517957310 ts_sal_ws-cleanup_1552594917056 TMA_Simulator ts_sal_ws-cleanup_1552444794218 ts_sal_ws-cleanup_1552496712674 ts_sal_ws-cleanup_1552519920919 ts_sal_ws-cleanup_1552669643187 ts_sal ts_sal_ws-cleanup_1552495340510 ts_sal_ws-cleanup_1552497953258 ts_sal_ws-cleanup_1552583005429 ts_sal_ws-cleanup_1552671917899 DM-18114 should prevent this issue from happening.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                aclements Andy Clements
                Reviewers:
                Rob Bovill
                Watchers:
                Andy Clements, Eric Coughlin, Jose Galaz, Joshua Hoblitt, Maria Grazia Prato, Rob Bovill, Sebastian Fehlandt, Sebastian Pereira, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: