Original
Original
Original
3 on
Windows 10
Enable Windows Subsystem for Linux
• Open Microsoft Store and search for Linux
• Download and Install Ubuntu
• Launch Ubuntu and create a new account
• Congrats! Now, you have a Linux system on your Windows!
Install Java 8
• Run the fullowing command to install Java 8 (jdk1.8)
wget https://mirrors.ocf.berkeley.edu/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz
Unzip Hadoop Binary
• Run the following command to create a hadoop folder under user
home folder:
mkdir ~/hadoop
• And then run the following command to unzip the binary package:
tar -xvzf hadoop-3.3.0.tar.gz -C ~/hadoop
• Once it is unpacked, change the current directory to the Hadoop
folder:
cd ~/hadoop/hadoop-3.3.0/
Configure ssh
• Make sure you can SSH to localhost in Ubuntu:
ssh localhost
• If you cannot ssh to localhost without a passphrase, run the following command to initialize your private and
public keys:
• If you encounter errors like ‘ssh: connect to host localhost port 22: Connection refused’, run the following
commands:
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export HADOOP_HOME=~/hadoop/hadoop-3.3.0
export PATH=$PATH:$HADOOP_HOME/bin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
source ~/.bashrc
• Add the following to $HADOOP_HOME/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
• Add the following configuration to
$HADOOP_HOME/etc/hadoop/core-site.xml (i.e., use the following to
replace empty <configuration> </configuration>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
• Add the following configuration to
$HADOOP_HOME/etc/hadoop/hdfs-site.xml (i.e., use the following to
replace empty <configuration> </configuration>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
• Add the following configuration to $HADOOP_HOME/etc/hadoop/mapred-
site.xml (i.e., use the following to replace empty <configuration> </configuration>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/
share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
• Add the following configuration to $HADOOP_HOME/etc/hadoop/yarn-site.xml
(i.e., use the following to replace empty <configuration> </configuration>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_P
REPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
Format namenode
• Run the following command
cd $HADOOP_HOME
bin/hdfs namenode –format
jps
cd $HADOOP_HOME
sbin/start-dfs.sh
Stop the service
• Run the following command
cd $HADOOP_HOME
sbin/stop-dfs.sh