码迷,mamicode.com
首页 > 编程语言 > 详细

Call From master/192.168.128.135 to master:8485 failed on connection exception: java.net.ConnectException: Connection refused

时间:2017-08-29 12:44:36      阅读:196      评论:0      收藏:0      [点我收藏+]

标签:datanode   比较   ret   htm   async   already   org   ogg   war   

hadoop集群搭建了ha,初次启动正常,最近几天启动时偶尔发现,namenode1节点启动后一段时间(大约10几秒-半分钟左右),namenode1上namenode进程停掉,查看日志:

 1 2017-08-28 21:54:37,617 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: slave1/192.168.128.136:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
 2 2017-08-28 21:54:37,629 WARN org.apache.hadoop.hdfs.server.namenode.FSEditLog: Unable to determine input streams from QJM to [192.168.128.135:8485, 192.168.128.136:8485, 192.168.128.137:8485]. Skipping.
 3 org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 1 successful responses:
 4 192.168.128.137:8485: [[1557,1558], [1559,1560], [1561,1562], [1563,1564], [1565,1566], [1567,1568], [1569,1570], [1571,1572], [1573,1574], [1575,1576], [1577,1578], [1579,1580], [1581,1582], [1583,1584], [1585,1586], [1587,1588], [1589,1589], [1590,1591]]
 5 2 exceptions thrown:
 6 192.168.128.136:8485: Call From master/192.168.128.135 to slave1:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
 7 192.168.128.135:8485: Call From master/192.168.128.135 to master:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
 8         at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
 9         at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
10         at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
11         at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:471)

 

一、查阅资料后找到原因:

我是用start-al.sh启动的集群,journalnode(端口8485)是在namenode后启动的。默认情况下namenode启动10s(maxRetries=10, sleepTime=1000)后journalnode还没有启动,就会报上述错误。

二、解决方案:

1.  修改core-site.xml中的ipc参数

 1 <!--修改core-site.xml中的ipc参数,防止出现连接journalnode服务ConnectException-->
 2 <property>
 3     <name>ipc.client.connect.max.retries</name>
 4     <value>100</value>
 5     <description>Indicates the number of retries a client will make to establish a server connection.</description>
 6 </property>
 7 <property>
 8     <name>ipc.client.connect.retry.interval</name>
 9     <value>10000</value>
10     <description>Indicates the number of milliseconds a client will wait for before retrying to establish a server connection.</description>
11 </property>

注意:仅对于这种由于服务没有启动完成造成连接超时的问题,都可以调整core-site.xml中的ipc参数来解决。如果目标服务本身没有启动成功,这边调整ipc参数是无效的。

2.  手动分步启动

1 #启动hadfs,注意有的是在多个节点执行的。
2 hadoop-daemons.sh start journalnode
3 hadoop-daemon.sh start namenode  #每个namenode都要执行
4 hadoop-daemon.sh start zkfc  #每个namenode都要执行
5 hadoop-daemons.sh start datanode
6 #启动yarn
7 start-yarn.sh

分步启动集群的方式,因为journalnode是在namenode之前启动的,所以正常情况下一次就会连接成功,不会重试多次。

三、错误再次分析

  由于部署好ha后,首次启动我是分步启动的,没有遇到该问题。之后都是start-all.sh启动,大约70%情况下会有该问题,30%左右的启动是正常的,究其原因,我想70%的时候journalnode启动比较慢,另有个别时候是启动比较快。实测中确实发现集群主机刚刚开机,就启动hadoop,会比较慢;等一段时间或者首次启动hadoop后停止,然后再重新启动,这两种情况下hadoop启动会比较快。我是虚拟机,通常会第一时间启动hadoop,所以遇到这个坑的时候比较多。当然,找到了根本原因,无论hadoop启动快慢namenode都不会挂掉了。

  另外namenode启动后有退出有多种原因,本文只针对启动的一种,具体情况需要查看日志并寻找合适解决方案。

 

致谢:http://www.linuxidc.com/Linux/2016-03/129437.htm

 

Call From master/192.168.128.135 to master:8485 failed on connection exception: java.net.ConnectException: Connection refused

标签:datanode   比较   ret   htm   async   already   org   ogg   war   

原文地址:http://www.cnblogs.com/tibit/p/7447190.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!