Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11746

jenkins remoting error on centos-6 nodes

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      This exception has been observed several times now. It seems to be related to archiving files from an centos-6 agent.

      [Pipeline] End of Pipeline
      		 
      java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-2.b16.el6_9.x86_64/jre/lib/resources.jar
      		 
      	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:928)
      		 
      	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
      		 
      	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
      		 
      	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
      		 
      	at java.security.AccessController.doPrivileged(Native Method)
      		 
      	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
      		 
      	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1038)
      		 
      	at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
      		 
      	at sun.misc.URLClassPath.getResource(URLClassPath.java:292)
      		 
      	at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1264)
      		 
      	at java.lang.ClassLoader.getResource(ClassLoader.java:1093)
      		 
      	at java.lang.ClassLoader.getResource(ClassLoader.java:1091)
      		 
      	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:232)
      		 
      	at java.util.ResourceBundle$RBClassLoader.getResourceAsStream(ResourceBundle.java:515)
      		 
      	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2677)
      		 
      	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2662)
      		 
      	at java.security.AccessController.doPrivileged(Native Method)
      		 
      	at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2661)
      		 
      	at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501)
      		 
      	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465)
      		 
      	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
      		 
      	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
      		 
      	at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361)
      		 
      	at java.util.ResourceBundle.getBundle(ResourceBundle.java:845)
      		 
      	at com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:169)
      		 
      	at com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:166)
      		 
      	at java.security.AccessController.doPrivileged(Native Method)
      		 
      	at com.sun.org.apache.xerces.internal.utils.SecuritySupport.getResourceBundle(SecuritySupport.java:166)
      		 
      	at com.sun.org.apache.xerces.internal.util.SAXMessageFormatter.formatMessage(SAXMessageFormatter.java:58)
      		 
      	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.getProperty(AbstractSAXParser.java:2063)
      		 
      	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.setProperty(SAXParserImpl.java:574)
      		 
      	at org.dom4j.io.SAXHelper.setParserProperty(SAXHelper.java:34)
      		 
      	at org.dom4j.io.SAXReader.configureReader(SAXReader.java:911)
      		 
      	at org.dom4j.io.SAXReader.read(SAXReader.java:463)
      		 
      	at org.dom4j.io.SAXReader.read(SAXReader.java:343)
      		 
      	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:132)
      		 
      	at hudson.tasks.junit.TestResult.parse(TestResult.java:302)
      		 
      	at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:244)
      		 
      	at hudson.tasks.junit.TestResult.parse(TestResult.java:175)
      		 
      	at hudson.tasks.junit.TestResult.parse(TestResult.java:154)
      		 
      	at hudson.tasks.junit.TestResult.<init>(TestResult.java:126)
      		 
      	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:132)
      		 
      	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:107)
      		 
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2749)
      		 
      	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
      		 
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      		 
      	at hudson.remoting.Request$2.run(Request.java:336)
      		 
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      		 
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		 
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      		 
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      		 
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      		 
      Caused: java.lang.InternalError
      		 
      	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1040)
      		 
      	at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
      		 
      	at sun.misc.URLClassPath.getResource(URLClassPath.java:292)
      		 
      	at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1264)
      		 
      	at java.lang.ClassLoader.getResource(ClassLoader.java:1093)
      		 
      	at java.lang.ClassLoader.getResource(ClassLoader.java:1091)
      		 
      	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:232)
      		 
      	at java.util.ResourceBundle$RBClassLoader.getResourceAsStream(ResourceBundle.java:515)
      		 
      	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2677)
      		 
      	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2662)
      		 
      	at java.security.AccessController.doPrivileged(Native Method)
      		 
      	at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2661)
      		 
      	at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501)
      		 
      	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465)
      		 
      	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
      		 
      	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
      		 
      	at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361)
      		 
      	at java.util.ResourceBundle.getBundle(ResourceBundle.java:845)
      		 
      	at com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:169)
      		 
      	at com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:166)
      		 
      	at java.security.AccessController.doPrivileged(Native Method)
      		 
      	at com.sun.org.apache.xerces.internal.utils.SecuritySupport.getResourceBundle(SecuritySupport.java:166)
      		 
      	at com.sun.org.apache.xerces.internal.util.SAXMessageFormatter.formatMessage(SAXMessageFormatter.java:58)
      		 
      	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.getProperty(AbstractSAXParser.java:2063)
      		 
      	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.setProperty(SAXParserImpl.java:574)
      		 
      	at org.dom4j.io.SAXHelper.setParserProperty(SAXHelper.java:34)
      		 
      	at org.dom4j.io.SAXReader.configureReader(SAXReader.java:911)
      		 
      	at org.dom4j.io.SAXReader.read(SAXReader.java:463)
      		 
      	at org.dom4j.io.SAXReader.read(SAXReader.java:343)
      		 
      	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:132)
      		 
      	at hudson.tasks.junit.TestResult.parse(TestResult.java:302)
      		 
      	at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:244)
      		 
      	at hudson.tasks.junit.TestResult.parse(TestResult.java:175)
      		 
      	at hudson.tasks.junit.TestResult.parse(TestResult.java:154)
      		 
      	at hudson.tasks.junit.TestResult.<init>(TestResult.java:126)
      		 
      	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:132)
      		 
      	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:107)
      		 
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2749)
      		 
      	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
      		 
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      		 
      	at hudson.remoting.Request$2.run(Request.java:336)
      		 
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      		 
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		 
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      		 
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      		 
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      		 
      	at java.lang.Thread.run(Thread.java:748)
      		 
      	at ......remote call to JNLP4-connect connection from ip-192-168-123-38.ec2.internal/192.168.123.38:39782(Native Method)
      		 
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
      		 
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
      		 
      	at hudson.remoting.Channel.call(Channel.java:830)
      		 
      Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-192-168-123-38.ec2.internal/192.168.123.38:39782 failed
      		 
      	at hudson.remoting.Channel.call(Channel.java:838)
      		 
      	at hudson.FilePath.act(FilePath.java:987)
      		 
      Caused: java.io.IOException: remote file operation failed: /home/jenkins-slave/workspace/stack-os-matrix at hudson.remoting.Channel@26bef38f:JNLP4-connect connection from ip-192-168-123-38.ec2.internal/192.168.123.38:39782
      		 
      	at hudson.FilePath.act(FilePath.java:994)
      		 
      	at hudson.FilePath.act(FilePath.java:976)
      		 
      	at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:103)
      		 
      	at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:128)
      		 
      	at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:149)
      		 
      	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
      		 
      	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
      		 
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
      		 
      	at hudson.security.ACL.impersonate(ACL.java:260)
      		 
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
      		 
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      		 
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		 
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      		 
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      		 
      	at java.lang.Thread.run(Thread.java:748)
      		 
      Finished: FAILURE

        Attachments

          Issue Links

          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11624 convert stack-os-matrix job to jenkins pipeline

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            There are no support plugin or regular log entries from today on the node (el6-2) the stack trace came from.

            Show
            jhoblitt Joshua Hoblitt added a comment - There are no support plugin or regular log entries from today on the node (el6-2) the stack trace came from.
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            The jenkins agent on that node reports that its java home is /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-2.b16.el6_9.x86_64/jre. However:

            [root@jenkins-el6-2 support]# find /usr -name resources.jar
            		 
            /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-3.b16.el6_9.x86_64/jre/lib/resources.jar
            		 
            /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.151.x86_64/jre/lib/resources.jar
            		 
            [root@jenkins-el6-2 support]# rpm -qa | grep java
            		 
            java-1.7.0-openjdk-devel-1.7.0.151-2.6.11.0.el6_9.x86_64
            		 
            java-1.8.0-openjdk-headless-1.8.0.141-3.b16.el6_9.x86_64
            		 
            tzdata-java-2017b-1.el6.noarch
            		 
            java-1.7.0-openjdk-1.7.0.151-2.6.11.0.el6_9.x86_64
            		 
            java-1.8.0-openjdk-1.8.0.141-3.b16.el6_9.x86_64

            My theory is that the agent wasn't restarted after the java update last week. After restarting the agent, the reported java home is now /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-3.b16.el6_9.x86_64/jre.

            Show
            jhoblitt Joshua Hoblitt added a comment - The jenkins agent on that node reports that its java home is /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-2.b16.el6_9.x86_64/jre . However: [root @jenkins -el6- 2 support]# find /usr -name resources.jar /usr/lib/jvm/java- 1.8 . 0 -openjdk- 1.8 . 0.141 - 3 .b16.el6_9.x86_64/jre/lib/resources.jar /usr/lib/jvm/java- 1.7 . 0 -openjdk- 1.7 . 0.151 .x86_64/jre/lib/resources.jar [root @jenkins -el6- 2 support]# rpm -qa | grep java java- 1.7 . 0 -openjdk-devel- 1.7 . 0.151 - 2.6 . 11.0 .el6_9.x86_64 java- 1.8 . 0 -openjdk-headless- 1.8 . 0.141 - 3 .b16.el6_9.x86_64 tzdata-java-2017b- 1 .el6.noarch java- 1.7 . 0 -openjdk- 1.7 . 0.151 - 2.6 . 11.0 .el6_9.x86_64 java- 1.8 . 0 -openjdk- 1.8 . 0.141 - 3 .b16.el6_9.x86_64 My theory is that the agent wasn't restarted after the java update last week. After restarting the agent, the reported java home is now /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-3.b16.el6_9.x86_64/jre .
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            I've checked the other linux nodes, the same problem was present on el6-1. The centos-7 nodes seem to all have been updated after the java update. Note that the el7 java version is slightly different (java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64).

            Show
            jhoblitt Joshua Hoblitt added a comment - I've checked the other linux nodes, the same problem was present on el6-1 . The centos-7 nodes seem to all have been updated after the java update. Note that the el7 java version is slightly different ( java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64 ).
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            The remoting errors seem to be resolved.

            Show
            jhoblitt Joshua Hoblitt added a comment - The remoting errors seem to be resolved.

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Watchers:
              Joshua Hoblitt, Pim Schellart [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.