版权声明：本文为博主原创文章，遵循
CC 4.0 BY-SA
版权协议，转载请附上原文出处链接和本声明。
本文链接:
https://blog.csdn.net/cmqzyd2222/article/details/102461358
参考文章:
https://www.iteye.com/blog/duguyiren3476-2214549
https://www.iteye.com/blog/vase-2090320
https://blog.csdn.net/Shea1992/article/details/101041244
http://www.luyixian.cn/news_show_6130.aspx
https://blog.csdn.net/you23hai45/article/details/70197383
本文使用的cdh版本为6.0.1,同时hadoop配置了lzo压缩,如使用其他版本可能会有问题,同时,文中部分你会发现内容可能没有,如果存在这一现象,是因为hadoop没有配置lzo的关系,博主这里找了一篇比较好的博文,里面详细介绍了对cdh的hadoop如何配置lzo:
https://blog.csdn.net/wwwzydcom/article/details/96735721
准备环境:(3台虚拟机,博主为master,slave1,slave2)
1.cdh版本6.0.1且处于搭建成功状态
2.maven:
apache-maven-3.3.9-bin.tar.gz
3.jdk:
oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
4.protobuf2.5.0.tar.gz:
https://download.csdn.net/download/cmqzyd2222/11835172
ps:这个资源强制5积分,无法修改为0积分,可自行百度下载
5.tez-0.9.1源码:
http://apache.01link.hk/tez/0.9.1/
6.没有时间编译的小伙伴,这里已经有编译完毕的tez文件供使用
https://download.csdn.net/download/cmqzyd2222/11835283
安装jdk(cdh安装成功的可以略过)
[root@master software]# rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
[root@master software]# vim /etc/profile
在最后添加
#JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export PATH=$PATH:$JAVA_HOME/bin
执行
[root@master software]# source /etc/profile
分发并在其他机器执行是否安装成功
[root@master software]# scp -r /etc/profile root@slave1:/etc/profile
[root@master software]# scp -r /etc/profile root@slave2:/etc/profile
[root@slave1 ~]# java -version
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)
安装maven(只需要在一台机器安装)
解压
[root@master software]# tar -zxvf apache-maven-3.3.9-bin.tar.gz -C /opt/module/
[root@master software]# cd /opt/module/apache-maven-3.3.9/conf
[root@master conf]# vim settings.xml
找到下面配置注释并添加下面的内容:修改配置阿里云仓库(如不配置亦可,从国外仓库下载,编译将十分慢)
<localRepository>/opt/modules/apache-maven-3.3.9/MvnRep</localRepository>
-----------------------------------------------
<mirror>
<id>nexus-aliyun</id>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
--------------------------------------------------------------
<profile>
<id>jdk-1.8</id>
<activation>
<activeByDefault>true</activeByDefault>
<jdk>1.8</jdk>
</activation>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.compilerVersion>1.8</maven.compiler.compilerVersion>
</properties>
</profile>
/etc/profile配置并记得source
#MVN_HOME
export MVN_HOME=/opt/module/apache-maven-3.3.9
export PATH=$MVN_HOME/bin:$PATH
[root@master software]# source /etc/profile
检查是否安装成功:可能会遇到以下问题
[root@master ~]# mvn -v
-bash: /opt/module/apache-maven-3.3.9/bin/mvn: Permission denied
[root@master ~]# chmod 755 /opt/module/apache-maven-3.3.9/bin/mvn
[root@master ~]# mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/module/apache-maven-3.3.9
Java version: 1.8.0_141, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_141-cloudera/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-957.21.3.el7.x86_64", arch: "amd64", family: "unix"
安装protobuf
[root@master software]# tar -zxvf protobuf-2.5.0.tar.gz -C /opt/module/
[root@master software]# cd /opt/module/protobuf-2.5.0/
[root@master protobuf-2.5.0]# ./configure
[root@master protobuf-2.5.0]# make
[root@master protobuf-2.5.0]# make install
[root@master protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
出现libprotoc 2.5.0则说明安装成功
编译tez使其支持cdh
[root@master software]# tar -zxvf apache-tez-0.9.1-src.tar.gz -C /opt/software
[root@master software]# cd apache-tez-0.9.1-src/
[root@master apache-tez-0.9.1-src]# vim pom.xml
按照如图所示逐一修改pom文件
全部替换properties为如下内容
<properties>
<maven.test.redirectTestOutputToFile>true</maven.test.redirectTestOutputToFile>
<clover.license>${user.home}/clover.license</clover.license>
<hadoop.version>3.0.0-cdh6.0.1</hadoop.version>
<jetty.version>6.1.26</jetty.version>
<netty.version>3.6.2.Final</netty.version>
<pig.version>0.13.0</pig.version>
<javac.version>1.8</javac.version>
<slf4j.version>1.7.10</slf4j.version>
<enforced.java.version>[${javac.version},)</enforced.java.version>
<distMgmtSnapshotsId>apache.snapshots.https</distMgmtSnapshotsId>
<distMgmtSnapshotsName>Apache Development Snapshot Repository</distMgmtSnapshotsName>
<distMgmtSnapshotsUrl>https://repository.apache.org/content/repositories/snapshots</distMgmtSnapshotsUrl>
<distMgmtStagingId>apache.staging.https</distMgmtStagingId>
<distMgmtStagingName>Apache Release Distribution Repository</distMgmtStagingName>
<distMgmtStagingUrl>https://repository.apache.org/service/local/staging/deploy/maven2</distMgmtStagingUrl>
<failIfNoTests>false</failIfNoTests>
<protobuf.version>2.5.0</protobuf.version>
<protoc.path>${env.PROTOC_PATH}</protoc.path>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<scm.url>scm:git:https://git-wip-us.apache.org/repos/asf/tez.git</scm.url>
<build.time>${maven.build.timestamp}</build.time>
<frontend-maven-plugin.version>1.4</frontend-maven-plugin.version>
<findbugs-maven-plugin.version>3.0.1</findbugs-maven-plugin.version>
<javadoc-maven-plugin.version>2.10.4</javadoc-maven-plugin.version>
<shade-maven-plugin.version>2.4.3</shade-maven-plugin.version>
</properties>
在repositories标签内部增加以下内容
<repository>
<id>nexus</id>
<name>Team Nexus Repository</name>
<url>http://192.168.0.203:8081/repository/maven-public/</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
<name>Cloudera Repositories</name>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
pluginRepositories标签内部增加cdh依赖
<pluginRepository>
<id>cloudera</id>
<name>Cloudera Repositories</name>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</pluginRepository>
modules标签内部注释tez-ext-service-tests和tez-ui两个模块
<modules>
<module>hadoop-shim</module>
<module>tez-api</module>
<module>tez-common</module>
<module>tez-runtime-library</module>
<module>tez-runtime-internals</module>
<module>tez-mapreduce</module>
<module>tez-examples</module>
<module>tez-tests</module>
<module>tez-dag</module>
<!--
<module>tez-ext-service-tests</module>
<module>tez-ui</module> -->
<module>tez-plugins</module>
<module>tez-tools</module>
<module>hadoop-shim-impls</module>
<module>tez-dist</module>
<module>docs</module>
</modules>
以上工作准备完毕,保存pom.xml文件,就可以开始编译了
[root@master apache-tez-0.9.1-src]# mvn clean package -Dmaven.javadoc.skip=true -Dmaven.test.skip=true
......经过漫长的编译时间之后.....
[root@master apache-tez-0.9.1-src]# cd tez-dist/
[root@master tez-dist]# cp tez-0.9.1.tar.gz /opt/software
[root@master module]# cd /opt/module/
[root@master module]# mkdir tez-0.9.1
[root@master module]# cd /opt/software/
[root@master software]# tar -zxvf tez-0.9.1.tar.gz -C /opt/module/tez-0.9.1/
注意分发
[root@master software]# scp -r /opt/module/tez-0.9.1/ root@slave1:/opt/module/
[root@master software]# scp -r /opt/module/tez-0.9.1/ root@slave2:/opt/module/
上传tez到hdfs
[root@master software]# hdfs dfs -mkdir /tez
[root@master software]# hdfs dfs -put tez-0.9.1.tar.gz /tez
网上查询都说要新建文件配置tez-site.xml,其实可以不用,由于cdh部署客户端配置会导致配置文件缺失,所以为了以后维护集群方便,博主直接在cloudera manager进行配置了
配置部分
开始进行疯狂踩坑,直接上看看出啥新鲜bug
hive (gmall)> set hive.tez.container.size=2048;
hive (gmall)> set hive.execution.engine=tez;
hive (gmall)> select count(1) from ods_event_log
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/tez/runtime/api/Event
at org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsDependentOptimizations(TezCompiler.java:282)
at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:133)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:134)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11058)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10626)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:564)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1359)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1488)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
出现了以上bug
以上bug是由于hive lib里面没有tez的相关包导致,解决方案如下:
[root@master software]# cd /opt/module/tez-0.9.1/
[root@master tez-0.9.1]# mkdir tmptez(临时文件,存放tezjar包用)
[root@master tez-0.9.1]# scp *.jar tmptez/
[root@master tez-0.9.1]# scp lib/* tmptez/
[root@master tez-0.9.1]# cd tmptez/
[root@master tmptez]# mv * /opt/cloudera/parcels/CDH/lib/hive/lib
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/aopalliance-1.0.jar’? n
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/audience-annotations-0.5.0.jar’? n
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/commons-io-2.4.jar’? n
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/commons-lang-2.6.jar’? n
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/curator-client-2.12.0.jar’? n
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/curator-framework-2.12.0.jar’? n
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/curator-recipes-2.12.0.jar’? n
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/gson-2.2.4.jar’? n
mv: overwrite ‘/opt/cloudera/parcels/CDH/lib/hive/lib/httpclient-4.5.3.jar’? n
以防万一,全部选n
hive/lib下jar包分发
[root@master hive]# scp -r /opt/cloudera/parcels/CDH/lib/hive/lib root@slave1:/opt/cloudera/parcels/CDH/lib/hive
[root@master hive]# scp -r /opt/cloudera/parcels/CDH/lib/hive/lib root@slave2:/opt/cloudera/parcels/CDH/lib/hive
再启动hive看看有啥bug
[root@master tmptez]# hive
hive (gmall)> set hive.tez.container.size=2048;
hive (gmall)> set hive.execution.engine=tez;
hive (gmall)> select count(1) from ods_event_log
出现了下面的bug
解决方案如下:(另两台机器同样位置也进行相同操作,不可忘记)
[root@master tmptez]# cd /opt/cloudera/parcels/CDH/lib/hive/auxlib
[root@master auxlib]# mv hive-exec-2.1.1-cdh6.0.1-core.jar hive-exec-2.1.1-cdh6.0.1-core.jar.bak
[root@master auxlib]# mv hive-exec-core.jar hive-exec-core.jar.bak
再试试,看看又有啥bug
[root@master auxlib]# hive
hive (gmall)> set hive.tez.container.size=2048;
hive (gmall)> set hive.execution.engine=tez;
hive (gmall)> select count(1) from ods_event_log
出现了以下bug,由于博主配置了lzo的关系
执行如下命令:(其他机器同样操作)
[root@master auxlib]# cp /opt/cloudera/parcels/GPLEXTRAS/jars/hadoop-lzo-0.4.15-cdh6.0.1.jar /opt/cloudera/parcels/CDH/lib/hive/lib
[root@master auxlib]# cp /opt/cloudera/parcels/GPLEXTRAS/jars/hadoop-lzo-0.4.15-cdh6.0.1.jar /opt/module/tez-0.9.1/lib/
[root@master auxlib]# cp /opt/cloudera/parcels/CDH/jars/kryo-2.22.jar /opt/cloudera/parcels/CDH/lib/hive/lib
[root@master auxlib]# cp /opt/cloudera/parcels/CDH/jars/kryo-2.22.jar /opt/module/tez-0.9.1/lib/
再试看看是否还有bug
[root@master auxlib]# hive
hive (gmall)> set hive.tez.container.size=2048;
hive (gmall)> set hive.execution.engine=tez;
hive (gmall)> select count(1) from ods_event_log
会发现仍然还是存在一样的bug,是因为没有将tez包加入hive-env的关系
为了将tez加入hive的环境,编写脚本如下:
#!/bin/bash
TEZ_HOME=/opt/module/tez-0.9.1 #是你的tez的解压目录
TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
TEZ_JARS=$TEZ_JARS,$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
TEZ_JARS=$TEZ_JARS,$TEZ_HOME/lib/$jar
done
#加入lzo包
echo "TEZ_JARS=/opt/cloudera/parcels/GPLEXTRAS/jars/hadoop-lzo-0.4.15-cdh6.0.1.jar$TEZ_JARS"
[root@master rmrf回收站]# sh bug.sh
将"TEZ_JARS=" 后面的文本全部复制备用
修改cdh的hive配置文件查看变量
[root@master auxlib]# cd /etc/hive/conf
[root@master conf]# vim hive-env.sh
最后一行增加echo,其余不动
启动hive查看值
[root@master conf]# hive
HIVE_AUX_JARS_PATH=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.0.1.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop2-compat.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-server.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/lib/htrace-core.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop-compat.jar,/usr/share/java/mysql-connector-java.jar
将”HIVE_AUX_JARS_PATH=” 之后的文本复制出来与之前的结合,如下:
HIVE_AUX_JARS_PATH=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.0.1.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop2-compat.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-server.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/lib/htrace-core.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop-compat.jar,/usr/share/java/mysql-connector-java.jar,/opt/cloudera/parcels/GPLEXTRAS/jars/hadoop-lzo-0.4.15-cdh6.0.1.jar,/opt/module/tez-0.9.1/hadoop-shim-0.9.1.jar,/opt/module/tez-0.9.1/hadoop-shim-2.7-0.9.1.jar,/opt/module/tez-0.9.1/tez-api-0.9.1.jar,/opt/module/tez-0.9.1/tez-common-0.9.1.jar,/opt/module/tez-0.9.1/tez-dag-0.9.1.jar,/opt/module/tez-0.9.1/tez-examples-0.9.1.jar,/opt/module/tez-0.9.1/tez-history-parser-0.9.1.jar,/opt/module/tez-0.9.1/tez-javadoc-tools-0.9.1.jar,/opt/module/tez-0.9.1/tez-job-analyzer-0.9.1.jar,/opt/module/tez-0.9.1/tez-mapreduce-0.9.1.jar,/opt/module/tez-0.9.1/tez-runtime-internals-0.9.1.jar,/opt/module/tez-0.9.1/tez-runtime-library-0.9.1.jar,/opt/module/tez-0.9.1/tez-tests-0.9.1.jar,/opt/module/tez-0.9.1/tez-yarn-timeline-history-0.9.1.jar,/opt/module/tez-0.9.1/tez-yarn-timeline-history-with-acls-0.9.1.jar,/opt/module/tez-0.9.1/lib/accessors-smart-1.2.jar,/opt/module/tez-0.9.1/lib/aopalliance-1.0.jar,/opt/module/tez-0.9.1/lib/asm-5.0.4.jar,/opt/module/tez-0.9.1/lib/async-http-client-1.8.16.jar,/opt/module/tez-0.9.1/lib/audience-annotations-0.5.0.jar,/opt/module/tez-0.9.1/lib/avro-1.8.2-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/commons-beanutils-1.9.3.jar,/opt/module/tez-0.9.1/lib/commons-cli-1.2.jar,/opt/module/tez-0.9.1/lib/commons-codec-1.4.jar,/opt/module/tez-0.9.1/lib/commons-collections-3.2.2.jar,/opt/module/tez-0.9.1/lib/commons-collections4-4.1.jar,/opt/module/tez-0.9.1/lib/commons-compress-1.4.1.jar,/opt/module/tez-0.9.1/lib/commons-configuration2-2.1.1.jar,/opt/module/tez-0.9.1/lib/commons-io-2.4.jar,/opt/module/tez-0.9.1/lib/commons-lang-2.6.jar,/opt/module/tez-0.9.1/lib/commons-lang3-3.7.jar,/opt/module/tez-0.9.1/lib/commons-logging-1.1.3.jar,/opt/module/tez-0.9.1/lib/commons-math3-3.1.1.jar,/opt/module/tez-0.9.1/lib/commons-net-3.1.jar,/opt/module/tez-0.9.1/lib/curator-client-2.12.0.jar,/opt/module/tez-0.9.1/lib/curator-framework-2.12.0.jar,/opt/module/tez-0.9.1/lib/curator-recipes-2.12.0.jar,/opt/module/tez-0.9.1/lib/ehcache-3.3.1.jar,/opt/module/tez-0.9.1/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar,/opt/module/tez-0.9.1/lib/gson-2.2.4.jar,/opt/module/tez-0.9.1/lib/guava-11.0.2.jar,/opt/module/tez-0.9.1/lib/guice-4.0.jar,/opt/module/tez-0.9.1/lib/guice-servlet-4.0.jar,/opt/module/tez-0.9.1/lib/hadoop-annotations-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-auth-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-hdfs-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-hdfs-client-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-lzo-0.4.15-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-mapreduce-client-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-mapreduce-client-core-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-api-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-client-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-server-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-server-web-proxy-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/HikariCP-java7-2.4.12.jar,/opt/module/tez-0.9.1/lib/htrace-core4-4.1.0-incubating.jar,/opt/module/tez-0.9.1/lib/httpclient-4.5.3.jar,/opt/module/tez-0.9.1/lib/httpcore-4.4.6.jar,/opt/module/tez-0.9.1/lib/jackson-annotations-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-core-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-core-asl-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-databind-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-base-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-json-provider-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-mapper-asl-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-module-jaxb-annotations-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-xc-1.8.3.jar,/opt/module/tez-0.9.1/lib/javax.inject-1.jar,/opt/module/tez-0.9.1/lib/javax.servlet-api-3.1.0.jar,/opt/module/tez-0.9.1/lib/jaxb-api-2.2.11.jar,/opt/module/tez-0.9.1/lib/jaxb-impl-2.2.3-1.jar,/opt/module/tez-0.9.1/lib/jcip-annotations-1.0-1.jar,/opt/module/tez-0.9.1/lib/jersey-client-1.9.jar,/opt/module/tez-0.9.1/lib/jersey-core-1.19.jar,/opt/module/tez-0.9.1/lib/jersey-guice-1.19.jar,/opt/module/tez-0.9.1/lib/jersey-json-1.9.jar,/opt/module/tez-0.9.1/lib/jersey-servlet-1.19.jar,/opt/module/tez-0.9.1/lib/jettison-1.3.4.jar,/opt/module/tez-0.9.1/lib/jetty-6.1.26.jar,/opt/module/tez-0.9.1/lib/jetty-http-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-io-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-security-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-server-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-servlet-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-util-6.1.26.jar,/opt/module/tez-0.9.1/lib/jetty-util-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-util-ajax-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-webapp-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-xml-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jline-2.12.jar,/opt/module/tez-0.9.1/lib/json-smart-2.3.jar,/opt/module/tez-0.9.1/lib/jsr305-3.0.0.jar,/opt/module/tez-0.9.1/lib/jsr311-api-1.1.1.jar,/opt/module/tez-0.9.1/lib/kerb-admin-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-client-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-common-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-core-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-crypto-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-identity-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-server-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-simplekdc-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-util-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-asn1-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-config-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-pkix-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-util-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-xdr-1.0.0.jar,/opt/module/tez-0.9.1/lib/kryo-2.22.jar,/opt/module/tez-0.9.1/lib/leveldbjni-all-1.8.jar,/opt/module/tez-0.9.1/lib/log4j-1.2.17.jar,/opt/module/tez-0.9.1/lib/metrics-core-3.1.0.jar,/opt/module/tez-0.9.1/lib/mssql-jdbc-6.2.1.jre7.jar,/opt/module/tez-0.9.1/lib/netty-all-4.1.17.Final.jar,/opt/module/tez-0.9.1/lib/nimbus-jose-jwt-4.41.1.jar,/opt/module/tez-0.9.1/lib/okhttp-2.7.5.jar,/opt/module/tez-0.9.1/lib/okio-1.6.0.jar,/opt/module/tez-0.9.1/lib/paranamer-2.8.jar,/opt/module/tez-0.9.1/lib/protobuf-java-2.5.0.jar,/opt/module/tez-0.9.1/lib/re2j-1.1.jar,/opt/module/tez-0.9.1/lib/RoaringBitmap-0.4.9.jar,/opt/module/tez-0.9.1/lib/servlet-api-2.5-20081211.jar,/opt/module/tez-0.9.1/lib/servlet-api-2.5.jar,/opt/module/tez-0.9.1/lib/slf4j-api-1.7.10.jar,/opt/module/tez-0.9.1/lib/slf4j-log4j12-1.7.10.jar,/opt/module/tez-0.9.1/lib/snappy-java-1.1.4.jar,/opt/module/tez-0.9.1/lib/stax2-api-3.1.4.jar,/opt/module/tez-0.9.1/lib/stax-api-1.0.1.jar,/opt/module/tez-0.9.1/lib/woodstox-core-5.0.3.jar,/opt/module/tez-0.9.1/lib/xz-1.6.jar,/opt/module/tez-0.9.1/lib/zookeeper-3.4.5-cdh6.0.1.jar
打开cdh管理界面,粘贴上面代码,并部署客户端配置
再次启动hive会发现打印很多日志
[root@master auxlib]# hive
hive (gmall)> set hive.tez.container.size=2048;
19/10/09 18:25:45 INFO conf.HiveConf: Using the default value passed in for log id: 35214824-1711-4eca-ad5c-d880d8c9b9f8
19/10/09 18:25:45 INFO conf.HiveConf: Using the default value passed in for log id: 35214824-1711-4eca-ad5c-d880d8c9b9f8
19/10/09 18:25:45 INFO session.SessionState: Resetting thread name to main
hive (gmall)> set hive.execution.engine=tez;
19/10/09 18:26:31 INFO conf.HiveConf: Using the default value passed in for log id: 35214824-1711-4eca-ad5c-d880d8c9b9f8
19/10/09 18:26:31 INFO session.SessionState: Updating thread name to 35214824-1711-4eca-ad5c-d880d8c9b9f8 main
19/10/09 18:26:31 INFO conf.HiveConf: Using the default value passed in for log id: 35214824-1711-4eca-ad5c-d880d8c9b9f8
19/10/09 18:26:31 INFO session.SessionState: Resetting thread name to main
......日志太多,崩溃了.........
解决方案如下:干掉日志文件(其他机器也进行同样操作)
[root@master ~]# cd /opt/module/tez-0.9.1/lib/
[root@master lib]# rm slf4j-api-1.7.10.jar
[root@master lib]# rm slf4j-log4j12-1.7.10.jar
重新运行脚本
[root@master rmrf回收站]# sh bug.sh
TEZ_JARS=/opt/cloudera/parcels/GPLEXTRAS/jars/hadoop-lzo-0.4.15-cdh6.0.1.jar,/opt/module/tez-0.9.1/hadoop-shim-0.9.1.jar,/opt/module/tez-0.9.1/hadoop-shim-2.7-0.9.1.jar,/opt/module/tez-0.9.1/tez-api-0.9.1.jar,/opt/module/tez-0.9.1/tez-common-0.9.1.jar,/opt/module/tez-0.9.1/tez-dag-0.9.1.jar,/opt/module/tez-0.9.1/tez-examples-0.9.1.jar,/opt/module/tez-0.9.1/tez-history-parser-0.9.1.jar,/opt/module/tez-0.9.1/tez-javadoc-tools-0.9.1.jar,/opt/module/tez-0.9.1/tez-job-analyzer-0.9.1.jar,/opt/module/tez-0.9.1/tez-mapreduce-0.9.1.jar,/opt/module/tez-0.9.1/tez-runtime-internals-0.9.1.jar,/opt/module/tez-0.9.1/tez-runtime-library-0.9.1.jar,/opt/module/tez-0.9.1/tez-tests-0.9.1.jar,/opt/module/tez-0.9.1/tez-yarn-timeline-history-0.9.1.jar,/opt/module/tez-0.9.1/tez-yarn-timeline-history-with-acls-0.9.1.jar,/opt/module/tez-0.9.1/lib/accessors-smart-1.2.jar,/opt/module/tez-0.9.1/lib/aopalliance-1.0.jar,/opt/module/tez-0.9.1/lib/asm-5.0.4.jar,/opt/module/tez-0.9.1/lib/async-http-client-1.8.16.jar,/opt/module/tez-0.9.1/lib/audience-annotations-0.5.0.jar,/opt/module/tez-0.9.1/lib/avro-1.8.2-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/commons-beanutils-1.9.3.jar,/opt/module/tez-0.9.1/lib/commons-cli-1.2.jar,/opt/module/tez-0.9.1/lib/commons-codec-1.4.jar,/opt/module/tez-0.9.1/lib/commons-collections-3.2.2.jar,/opt/module/tez-0.9.1/lib/commons-collections4-4.1.jar,/opt/module/tez-0.9.1/lib/commons-compress-1.4.1.jar,/opt/module/tez-0.9.1/lib/commons-configuration2-2.1.1.jar,/opt/module/tez-0.9.1/lib/commons-io-2.4.jar,/opt/module/tez-0.9.1/lib/commons-lang-2.6.jar,/opt/module/tez-0.9.1/lib/commons-lang3-3.7.jar,/opt/module/tez-0.9.1/lib/commons-logging-1.1.3.jar,/opt/module/tez-0.9.1/lib/commons-math3-3.1.1.jar,/opt/module/tez-0.9.1/lib/commons-net-3.1.jar,/opt/module/tez-0.9.1/lib/curator-client-2.12.0.jar,/opt/module/tez-0.9.1/lib/curator-framework-2.12.0.jar,/opt/module/tez-0.9.1/lib/curator-recipes-2.12.0.jar,/opt/module/tez-0.9.1/lib/ehcache-3.3.1.jar,/opt/module/tez-0.9.1/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar,/opt/module/tez-0.9.1/lib/gson-2.2.4.jar,/opt/module/tez-0.9.1/lib/guava-11.0.2.jar,/opt/module/tez-0.9.1/lib/guice-4.0.jar,/opt/module/tez-0.9.1/lib/guice-servlet-4.0.jar,/opt/module/tez-0.9.1/lib/hadoop-annotations-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-auth-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-hdfs-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-hdfs-client-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-lzo-0.4.15-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-mapreduce-client-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-mapreduce-client-core-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-api-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-client-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-server-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-server-web-proxy-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/HikariCP-java7-2.4.12.jar,/opt/module/tez-0.9.1/lib/htrace-core4-4.1.0-incubating.jar,/opt/module/tez-0.9.1/lib/httpclient-4.5.3.jar,/opt/module/tez-0.9.1/lib/httpcore-4.4.6.jar,/opt/module/tez-0.9.1/lib/jackson-annotations-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-core-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-core-asl-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-databind-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-base-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-json-provider-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-mapper-asl-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-module-jaxb-annotations-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-xc-1.8.3.jar,/opt/module/tez-0.9.1/lib/javax.inject-1.jar,/opt/module/tez-0.9.1/lib/javax.servlet-api-3.1.0.jar,/opt/module/tez-0.9.1/lib/jaxb-api-2.2.11.jar,/opt/module/tez-0.9.1/lib/jaxb-impl-2.2.3-1.jar,/opt/module/tez-0.9.1/lib/jcip-annotations-1.0-1.jar,/opt/module/tez-0.9.1/lib/jersey-client-1.9.jar,/opt/module/tez-0.9.1/lib/jersey-core-1.19.jar,/opt/module/tez-0.9.1/lib/jersey-guice-1.19.jar,/opt/module/tez-0.9.1/lib/jersey-json-1.9.jar,/opt/module/tez-0.9.1/lib/jersey-servlet-1.19.jar,/opt/module/tez-0.9.1/lib/jettison-1.3.4.jar,/opt/module/tez-0.9.1/lib/jetty-6.1.26.jar,/opt/module/tez-0.9.1/lib/jetty-http-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-io-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-security-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-server-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-servlet-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-util-6.1.26.jar,/opt/module/tez-0.9.1/lib/jetty-util-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-util-ajax-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-webapp-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-xml-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jline-2.12.jar,/opt/module/tez-0.9.1/lib/json-smart-2.3.jar,/opt/module/tez-0.9.1/lib/jsr305-3.0.0.jar,/opt/module/tez-0.9.1/lib/jsr311-api-1.1.1.jar,/opt/module/tez-0.9.1/lib/kerb-admin-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-client-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-common-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-core-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-crypto-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-identity-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-server-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-simplekdc-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-util-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-asn1-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-config-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-pkix-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-util-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-xdr-1.0.0.jar,/opt/module/tez-0.9.1/lib/kryo-2.22.jar,/opt/module/tez-0.9.1/lib/leveldbjni-all-1.8.jar,/opt/module/tez-0.9.1/lib/log4j-1.2.17.jar,/opt/module/tez-0.9.1/lib/metrics-core-3.1.0.jar,/opt/module/tez-0.9.1/lib/mssql-jdbc-6.2.1.jre7.jar,/opt/module/tez-0.9.1/lib/netty-all-4.1.17.Final.jar,/opt/module/tez-0.9.1/lib/nimbus-jose-jwt-4.41.1.jar,/opt/module/tez-0.9.1/lib/okhttp-2.7.5.jar,/opt/module/tez-0.9.1/lib/okio-1.6.0.jar,/opt/module/tez-0.9.1/lib/paranamer-2.8.jar,/opt/module/tez-0.9.1/lib/protobuf-java-2.5.0.jar,/opt/module/tez-0.9.1/lib/re2j-1.1.jar,/opt/module/tez-0.9.1/lib/RoaringBitmap-0.4.9.jar,/opt/module/tez-0.9.1/lib/servlet-api-2.5-20081211.jar,/opt/module/tez-0.9.1/lib/servlet-api-2.5.jar,/opt/module/tez-0.9.1/lib/snappy-java-1.1.4.jar,/opt/module/tez-0.9.1/lib/stax2-api-3.1.4.jar,/opt/module/tez-0.9.1/lib/stax-api-1.0.1.jar,/opt/module/tez-0.9.1/lib/woodstox-core-5.0.3.jar,/opt/module/tez-0.9.1/lib/xz-1.6.jar,/opt/module/tez-0.9.1/lib/zookeeper-3.4.5-cdh6.0.1.jar
拿上述文档hive-env.sh打印出来的HIVE_AUX_JARS_PATH与TEZ_JARS重新结合,最终结果如下:
HIVE_AUX_JARS_PATH=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.0.1.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop2-compat.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-server.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/lib/htrace-core.jar,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop-compat.jar,/usr/share/java/mysql-connector-java.jar,/opt/cloudera/parcels/GPLEXTRAS/jars/hadoop-lzo-0.4.15-cdh6.0.1.jar,/opt/module/tez-0.9.1/hadoop-shim-0.9.1.jar,/opt/module/tez-0.9.1/hadoop-shim-2.7-0.9.1.jar,/opt/module/tez-0.9.1/tez-api-0.9.1.jar,/opt/module/tez-0.9.1/tez-common-0.9.1.jar,/opt/module/tez-0.9.1/tez-dag-0.9.1.jar,/opt/module/tez-0.9.1/tez-examples-0.9.1.jar,/opt/module/tez-0.9.1/tez-history-parser-0.9.1.jar,/opt/module/tez-0.9.1/tez-javadoc-tools-0.9.1.jar,/opt/module/tez-0.9.1/tez-job-analyzer-0.9.1.jar,/opt/module/tez-0.9.1/tez-mapreduce-0.9.1.jar,/opt/module/tez-0.9.1/tez-runtime-internals-0.9.1.jar,/opt/module/tez-0.9.1/tez-runtime-library-0.9.1.jar,/opt/module/tez-0.9.1/tez-tests-0.9.1.jar,/opt/module/tez-0.9.1/tez-yarn-timeline-history-0.9.1.jar,/opt/module/tez-0.9.1/tez-yarn-timeline-history-with-acls-0.9.1.jar,/opt/module/tez-0.9.1/lib/accessors-smart-1.2.jar,/opt/module/tez-0.9.1/lib/aopalliance-1.0.jar,/opt/module/tez-0.9.1/lib/asm-5.0.4.jar,/opt/module/tez-0.9.1/lib/async-http-client-1.8.16.jar,/opt/module/tez-0.9.1/lib/audience-annotations-0.5.0.jar,/opt/module/tez-0.9.1/lib/avro-1.8.2-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/commons-beanutils-1.9.3.jar,/opt/module/tez-0.9.1/lib/commons-cli-1.2.jar,/opt/module/tez-0.9.1/lib/commons-codec-1.4.jar,/opt/module/tez-0.9.1/lib/commons-collections-3.2.2.jar,/opt/module/tez-0.9.1/lib/commons-collections4-4.1.jar,/opt/module/tez-0.9.1/lib/commons-compress-1.4.1.jar,/opt/module/tez-0.9.1/lib/commons-configuration2-2.1.1.jar,/opt/module/tez-0.9.1/lib/commons-io-2.4.jar,/opt/module/tez-0.9.1/lib/commons-lang-2.6.jar,/opt/module/tez-0.9.1/lib/commons-lang3-3.7.jar,/opt/module/tez-0.9.1/lib/commons-logging-1.1.3.jar,/opt/module/tez-0.9.1/lib/commons-math3-3.1.1.jar,/opt/module/tez-0.9.1/lib/commons-net-3.1.jar,/opt/module/tez-0.9.1/lib/curator-client-2.12.0.jar,/opt/module/tez-0.9.1/lib/curator-framework-2.12.0.jar,/opt/module/tez-0.9.1/lib/curator-recipes-2.12.0.jar,/opt/module/tez-0.9.1/lib/ehcache-3.3.1.jar,/opt/module/tez-0.9.1/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar,/opt/module/tez-0.9.1/lib/gson-2.2.4.jar,/opt/module/tez-0.9.1/lib/guava-11.0.2.jar,/opt/module/tez-0.9.1/lib/guice-4.0.jar,/opt/module/tez-0.9.1/lib/guice-servlet-4.0.jar,/opt/module/tez-0.9.1/lib/hadoop-annotations-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-auth-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-hdfs-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-hdfs-client-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-lzo-0.4.15-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-mapreduce-client-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-mapreduce-client-core-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-api-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-client-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-server-common-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/hadoop-yarn-server-web-proxy-3.0.0-cdh6.0.1.jar,/opt/module/tez-0.9.1/lib/HikariCP-java7-2.4.12.jar,/opt/module/tez-0.9.1/lib/htrace-core4-4.1.0-incubating.jar,/opt/module/tez-0.9.1/lib/httpclient-4.5.3.jar,/opt/module/tez-0.9.1/lib/httpcore-4.4.6.jar,/opt/module/tez-0.9.1/lib/jackson-annotations-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-core-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-core-asl-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-databind-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-base-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-jaxrs-json-provider-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-mapper-asl-1.8.3.jar,/opt/module/tez-0.9.1/lib/jackson-module-jaxb-annotations-2.9.5.jar,/opt/module/tez-0.9.1/lib/jackson-xc-1.8.3.jar,/opt/module/tez-0.9.1/lib/javax.inject-1.jar,/opt/module/tez-0.9.1/lib/javax.servlet-api-3.1.0.jar,/opt/module/tez-0.9.1/lib/jaxb-api-2.2.11.jar,/opt/module/tez-0.9.1/lib/jaxb-impl-2.2.3-1.jar,/opt/module/tez-0.9.1/lib/jcip-annotations-1.0-1.jar,/opt/module/tez-0.9.1/lib/jersey-client-1.9.jar,/opt/module/tez-0.9.1/lib/jersey-core-1.19.jar,/opt/module/tez-0.9.1/lib/jersey-guice-1.19.jar,/opt/module/tez-0.9.1/lib/jersey-json-1.9.jar,/opt/module/tez-0.9.1/lib/jersey-servlet-1.19.jar,/opt/module/tez-0.9.1/lib/jettison-1.3.4.jar,/opt/module/tez-0.9.1/lib/jetty-6.1.26.jar,/opt/module/tez-0.9.1/lib/jetty-http-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-io-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-security-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-server-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-servlet-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-util-6.1.26.jar,/opt/module/tez-0.9.1/lib/jetty-util-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-util-ajax-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-webapp-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jetty-xml-9.3.20.v20170531.jar,/opt/module/tez-0.9.1/lib/jline-2.12.jar,/opt/module/tez-0.9.1/lib/json-smart-2.3.jar,/opt/module/tez-0.9.1/lib/jsr305-3.0.0.jar,/opt/module/tez-0.9.1/lib/jsr311-api-1.1.1.jar,/opt/module/tez-0.9.1/lib/kerb-admin-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-client-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-common-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-core-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-crypto-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-identity-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-server-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-simplekdc-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerb-util-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-asn1-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-config-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-pkix-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-util-1.0.0.jar,/opt/module/tez-0.9.1/lib/kerby-xdr-1.0.0.jar,/opt/module/tez-0.9.1/lib/kryo-2.22.jar,/opt/module/tez-0.9.1/lib/leveldbjni-all-1.8.jar,/opt/module/tez-0.9.1/lib/log4j-1.2.17.jar,/opt/module/tez-0.9.1/lib/metrics-core-3.1.0.jar,/opt/module/tez-0.9.1/lib/mssql-jdbc-6.2.1.jre7.jar,/opt/module/tez-0.9.1/lib/netty-all-4.1.17.Final.jar,/opt/module/tez-0.9.1/lib/nimbus-jose-jwt-4.41.1.jar,/opt/module/tez-0.9.1/lib/okhttp-2.7.5.jar,/opt/module/tez-0.9.1/lib/okio-1.6.0.jar,/opt/module/tez-0.9.1/lib/paranamer-2.8.jar,/opt/module/tez-0.9.1/lib/protobuf-java-2.5.0.jar,/opt/module/tez-0.9.1/lib/re2j-1.1.jar,/opt/module/tez-0.9.1/lib/RoaringBitmap-0.4.9.jar,/opt/module/tez-0.9.1/lib/servlet-api-2.5-20081211.jar,/opt/module/tez-0.9.1/lib/servlet-api-2.5.jar,/opt/module/tez-0.9.1/lib/snappy-java-1.1.4.jar,/opt/module/tez-0.9.1/lib/stax2-api-3.1.4.jar,/opt/module/tez-0.9.1/lib/stax-api-1.0.1.jar,/opt/module/tez-0.9.1/lib/woodstox-core-5.0.3.jar,/opt/module/tez-0.9.1/lib/xz-1.6.jar,/opt/module/tez-0.9.1/lib/zookeeper-3.4.5-cdh6.0.1.jar
cdh管理界面再一次修改并部署客户端配置
这一次再次测试:
[root@master lib]# hive
hive (gmall)> set hive.tez.container.size=2048;
hive (gmall)> set hive.execution.engine=tez;
hive (gmall)> select count(1) from ods_event_log
Query ID = root_20191009184344_333e044c-196c-4d95-9a4e-265228b8113f
Total jobs = 1
Launching Job 1 out of 1
发现日志消失
出现了以下界面,说明安装成功!!!
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 17.69 s
----------------------------------------------------------------------------------------------
OK
_c0
1978
Time taken: 67.523 seconds, Fetched: 1 row(s)
hive (gmall)>
其他机器测试:
[root@slave1 lib]# hive
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 33.05 s
----------------------------------------------------------------------------------------------
OK
_c0
1978
Time taken: 80.71 seconds, Fetched: 1 row(s)
hive (gmall)>
成功!
第二台机器测试
[root@slave2 lib]# hive
Status: Running (Executing on YARN cluster with App id application_1570612082129_0009)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 34.67 s
----------------------------------------------------------------------------------------------
OK
_c0
1978
Time taken: 120.195 seconds, Fetched: 1 row(s)
hive (gmall)>
成功!
博主机器内存20G,开了好多东西,运行起来巨慢,内存球基本到95%,要刚不住了…
好了,最后,CDH管理界面配置hive引擎为tez,并部署客户端配置
测试:
[root@master lib]# hive
hive (gmall)> select count(1) from ods_event_log;
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1570612082129_0010)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 20.21 s
----------------------------------------------------------------------------------------------
OK
_c0
1978
Time taken: 26.28 seconds, Fetched: 1 row(s)
hive (gmall)>
大功告成!!!本篇攻略到此结束,欢迎随时提供各种bug!
版权声明：本文为cmqzyd2222原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。