博客

现象

spark-submit on yarn 时出现异常。

22/07/20 15:59:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/07/20 15:59:53 INFO client.RMProxy: Connecting to ResourceManager at sh01/172.16.99.215:8010
22/07/20 15:59:54 INFO ipc.Client: Retrying connect to server: sh01/172.17.99.215:8010. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/20 15:59:55 INFO ipc.Client: Retrying connect to server: sh01/172.17.99.215:8010. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/20 15:59:56 INFO ipc.Client: Retrying connect to server: sh01/172.17.99.215:8010. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

根据日志是链接 ResourceManager 时一直在尝试… 也是就说客户端连不上 yarn。

问题查找

1. 确认平台上 yarn 是不是已经启动了，（我遇到的就是 yarn 没有启动）

   jps
   
   8144 SecondaryNameNode
   9251 Master
   7732 NameNode
   3215205 Jps
   3214887 ResourceManager #（能看到这个表示 yarn 启动了）
   2748510 Master
   

2. 看下 yarn-site.xml 中是否配置正确。

    <property>
      <name>yarn.resourcemanager.address</name>
      <value>sh01:8010</value>
    </property>
   

3. 可能是解析不了自定与域名，在
   
/etc/hosts

   要配置域名映射。

   127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
   ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
   172.17.99.215 sh01
   

4. 确认hadoop环境变量是否配置正确。

   echo ${HADOOP_HOME}
   /usr/local/hadoop