博客

在ubuntu上搭建hadoop，记录一下

环境：ubuntu 16.04

hadoop-3.1.0

1.安装JDK

已经使用apt安装了jdk环境

whereis java

ls -l /usr/bin/java

ls -l /etc/alternatives/java

默认jdk路径：/usr/lib/jvm/java-8-openjdk-amd64

2.创建hadoop组、hadoop用户

sudo addgroup hadoop

sudo adduser –ingroup hadoop hadoop

给hadoop用户添加权限

sudo vi /etc/sudoers

添加如下：hadoop ALL=(ALL:ALL) ALL

3.配置SSH无密码登陆

切换用户

su hadoop

安装ssh，openssh-server

sudo apt install ssh

sudo apt install openssh-server

启动服务

sudo /etc/init.d/ssh start

查看服务是否启动

ps -e | grep ssh

进入宿主目录

cd /home/hadoop/

设置免密码登陆

ssh-keygen -t rsa -P “”

/home/hadoop/.ssh下生成两个文件： id_rsa （私钥）  、id_rsa.pub  （

公钥

）

把公钥加到用于认证的公钥文件中

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 600 .ssh/authorized_keys

登入ssh，确认不需要密码

ssh localhost

登出

exit

4.下载解压hadoop

回到用户工作目录

cd ~

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz

tar -zxvf

hadoop-3.1.0.tar.gz

移动到/usr/local/hadoop目录

sudo mv hadoop-3.1.0  /usr/local/hadoop

赋予用户对该文件夹的读写权限

sudo chmod 777 /usr/local/hadoop

5.配置

sudo vi ~/.bashrc

添加如下：

#HADOOP VARIABLES START

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/

export HADOOP_INSTALL=/usr/local/hadoop

export PATH=$PATH:$HADOOP_INSTALL/bin

export PATH=$PATH:$HADOOP_INSTALL/sbin

export HADOOP_MAPRED_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_HOME=$HADOOP_INSTALL

export HADOOP_HDFS_HOME=$HADOOP_INSTALL

export YARN_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native

export HADOOP_OPTS=”-Djava.library.path=$HADOOP_INSTALL/lib”

#HADOOP VARIABLES END

使配置生效

source ~/.bashrc

sudo vi /usr/local/hadoop/etc/hadoop/hadoop-env.sh

添加如下：

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

6.WordCount测试

执行hadoop自带实例WordCount，验证是否安装成功

cd /usr/local/hadoop

创建输入文件

mkdir input

cp README.txt input

执行

bin/hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-3.1.0-sources.jar org.apache.hadoop.examples.WordCount input output

查看字符统计结果

cat output/*

参考：

https://blog.csdn.net/wenyun_kang/article/details/77413714