Tag: yarn

Spark Installation on Hadoop

In this tutorial I will show you how to use Kerberos/SSL with Spark integrated with Yarn. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Hadoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

  1. cd /etc/security/keytabs/
  2.  
  3. sudo kadmin.local
  4.  
  5. #You can list princepals
  6. listprincs
  7.  
  8. #Create the following principals
  9. addprinc -randkey spark/hadoop@REALM.CA
  10.  
  11. #Create the keytab files.
  12. #You will need these for Hadoop to be able to login
  13. xst -k spark.service.keytab spark/hadoop@REALM.CA

Set Keytab Permissions/Ownership

  1. sudo chown root:hadoopuser /etc/security/keytabs/*
  2. sudo chmod 750 /etc/security/keytabs/*

Download

Go to Apache Spark Download and get the link for Spark.

  1. wget http://apache.forsale.plus/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
  2. tar -xvf spark-2.4.4-bin-hadoop2.7.tgz
  3. mv spark-2.4.4-bin-hadoop2.7 /usr/local/spark/

Update .bashrc

  1. sudo nano ~/.bashrc
  2.  
  3. #Ensure we have the following in the Hadoop section
  4. export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
  5.  
  6. #Add the following
  7.  
  8. #SPARK VARIABLES START
  9. export SPARK_HOME=/usr/local/spark
  10. export PATH=$PATH:$SPARK_HOME/bin
  11. export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
  12. #SPARK VARIABLES STOP
  13.  
  14. source ~/.bashrc

Setup Configuration

  1. cd /usr/local/spark/conf
  2. mv spark-defaults.conf.template spark-defaults.conf
  3. nano spark-defaults.conf
  4.  
  5. #Add to the end
  6. spark.master                            yarn
  7. spark.yarn.historyServer.address        ${hadoopconf-yarn.resourcemanager.hostname}:18080
  8. spark.yarn.keytab                       /etc/security/keytabs/spark.service.keytab
  9. spark.yarn.principal                    spark/hadoop@REALM.CA
  10. spark.yarn.access.hadoopFileSystems     hdfs://NAMENODE:54310
  11. spark.authenticate                      true
  12. spark.driver.bindAddress                0.0.0.0
  13. spark.authenticate.enableSaslEncryption true
  14. spark.eventLog.enabled                  true
  15. spark.eventLog.dir                      hdfs://NAMENODE:54310/user/spark/applicationHistory
  16. spark.history.fs.logDirectory           hdfs://NAMENODE:54310/user/spark/applicationHistory
  17. spark.history.fs.update.interval        10s
  18. spark.history.ui.port                   18080
  19.  
  20. #SSL
  21. spark.ssl.enabled                       true
  22. spark.ssl.keyPassword                   PASSWORD
  23. spark.ssl.keyStore                      /etc/security/serverKeys/keystore.jks
  24. spark.ssl.keyStorePassword              PASSWORD
  25. spark.ssl.keyStoreType                  JKS
  26. spark.ssl.trustStore                    /etc/security/serverKeys/truststore.jks
  27. spark.ssl.trustStorePassword            PASSWORD
  28. spark.ssl.trustStoreType                JKS

Kinit

  1. kinit -kt /etc/security/keytabs/spark.service.keytab spark/hadoop@REALM.CA
  2. klist
  3. hdfs dfs -mkdir /user/spark/
  4. hdfs dfs -mkdir /user/spark/applicationHistory
  5. hdfs dfs -ls /user/spark

Start The Service

  1. $SPARK_HOME/sbin/start-history-server.sh

Stop The Service

  1. $SPARK_HOME/sbin/stop-history-server.sh

Spark History Server Web UI

References

I used a lot of different resources and reference material on this. Below are just a few I used.

https://spark.apache.org/docs/latest/running-on-yarn.html#configuration

https://spark.apache.org/docs/latest/security.html

https://www.linode.com/docs/databases/hadoop/install-configure-run-spark-on-top-of-hadoop-yarn-cluster/

 

 

 

 

Hadoop 3.2.0: Installation

I would like to share what I have learned and applied in the hopes that it will help someone else configure their system. The deployment I have done is to have a Name Node and 1-* DataNodes on Ubuntu 16.04 assuming 5 cpu and 13GB RAM. I will put all commands used in this tutorial right down to the very basics for those that are new to Ubuntu.

NOTE: Sometimes you may have to use “sudo” in front of the command. I also use nano for this article for beginners but you can use any editor you prefer (ie: vi). Also this article does not take into consideration any SSL, kerberos, etc. For all purposes here Hadoop will be open without having to login, etc.

Additional Setup/Configurations to Consider:

Zookeeper: It is also a good idea to use ZooKeeper to synchronize your configuration

Secondary NameNode: This should be done on a seperate server and it’s function is to take checkpoints of the namenodes file system.

Rack Awareness: Fault tolerance to ensure blocks are placed as evenly as possible on different racks if they are available.

Apply the following to all NameNode and DataNodes unless otherwise directed:

Hadoop User:
For this example we will just use hduser as our group and user for simplicity sake.
The “-a” on usermod is for appending to a group used with –G for which groups

  1. addgroup hduser
  2. sudo gpasswd -a $USER sudo
  3. usermod –a –G sudo hduser

Install JDK:

  1. apt-get update
  2. apt-get upgrade
  3. apt-get install default-jdk

Install SSH:

  1. apt-get install ssh
  2. which ssh
  3. which sshd

These two commands will check that ssh installed correctly and will return “/usr/bin/ssh” and “/usr/bin/sshd”

  1. java -version

You use this to verify that java installed correctly and will return something like the following.

openjdk version “1.8.0_171”
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

System Configuration

  1. nano ~/.bashrc

The .bashrc is a script that is executed when a terminal session is started.
Add the following line to the end and save because Hadoop uses IPv4.

export _JAVA_OPTIONS=’-XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true’

  1. source ~/.bashrc

sysctl.conf

Disable ipv6 as it causes issues in getting your server up and running.

  1. nano /etc/sysctl.conf

Add the following to the end and save

  1. net.ipv6.conf.all.disable_ipv6 = 1
  2. net.ipv6.conf.default.disable_ipv6 = 1
  3. net.ipv6.conf.lo.disable_ipv6 = 1
  4. #Change eth0 to what ifconfig has
  5. net.ipv6.conf.eth0.disable_ipv6 = 1

Close sysctl

  1. sysctl -p
  2. cat /proc/sys/net/ipv6/conf/all/disable_ipv6
  3. reboot

If all the above disabling IPv6 configuration was successful you should get “1” returned.
Sometimes you can reach open file descriptor limit and open file limit. If you do encounter this issue you might have to set the ulimit and descriptor limit. For this example I have set some values but you will have to figure out the best numbers for your specific case.

If you get “cannot stat /proc/sys/-p: No such file or directory”. Then you need to add /sbin/ to PATH.

  1. sudo nano ~/.bashrc
  2. export PATH=$PATH:/sbin/
  1. nano /etc/sysctl.conf

fs.file-max = 500000

  1. sysctl –p

limits.conf

  1. nano /etc/security/limits.conf

* soft nofile 60000
* hard nofile 60000

  1.  reboot

Test Limits

You can now test the limits you applied to make sure they took.

  1. ulimit -a
  2. more /proc/sys/fs/file-max
  3. more /proc/sys/fs/file-nr
  4. lsof | wc -l

file-max: Current open file descriptor limit
file-nr: How many file descriptors are currently being used
lsof wc: How many files are currently open

You might be wondering why we installed ssh at the beginning. That is because Hadoop uses ssh to access its nodes. We need to eliminate the password requirement by setting up ssh certificates. If asked for a filename just leave it blank and confirm with enter.

  1. su hduser

If not already logged in as the user we created in the Hadoop user section.

  1. ssh-keygen –t rsa –P ""

You will get the below example as well as the fingerprint and randomart image.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory ‘/home/hduser/.ssh’.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.

  1. cat $HOME/.ssh/id-rsa.pub >> $HOME/.ssh/authorized_keys

You may get “No such file or directory”. It is most likely just the id-rsa.pub filename. Look in the .ssh directory for the name it most likely will be “id_rsa.pub”.

This will add the newly created key to the list of authorized keys so that Hadoop can use SSH without prompting for a password.
Now we check that it worked by running “ssh localhost”. When prompted with if you should continue connecting type “yes” and enter. You will be permanently added to localhost
Once we have done this on all Name Node and Data Node you should run the following command from the Name Node to each Data Node.

  1. ssh-copy-id –i ~/.ssh/id_rsa.pub hduser@DATANODEHOSTNAME
  2. ssh DATANODEHOSTNAME

/etc/hosts Update

We need to update the hosts file.

  1. sudo nano /etc/hosts
  2.  
  3. #Comment out line "127.0.0.1 localhost"
  4.  
  5. 127.0.0.1 HOSTNAME localhost

Now we are getting to the part we have been waiting for.

Hadoop Installation:

NAMENODE: You will see this in the config files below and it can be the hostname, the static ip or it could be 0.0.0.0 so that all TCP ports will be bound to all IP’s of the server. You should also note that the masters and slaves file later on in this tutorial can still be the hostname.

Note: You could run rsync after setting up the Name Node Initial configuration to each Data Node if you want. This would save initial hadoop setup time. You do that by running the following command:

  1. rsync –a /usr/local/hadoop/ hduser@DATANODEHOSTNAME:/usr/local/hadoop/

Download & Extract:

  1. wget https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz 
  2. tar xvzf hadoop-3.2.0.tar.gz
  3. sudo mv hadoop-3.2.0/ /usr/local/hadoop
  4. chown –R hduser:hduser /usr/local/hadoop
  5. update-alternatives --config java

Basically the above downloads, extracts, moves the extracted hadoop directory to the /usr/local directory, if the hduser doesn’t own the newly created directory then switch ownership
and tells us the path where java was been installed to to set the JAVA_HOME environment variable. It should return something like the following:

There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

  1. nano ~/.bashrc

Add the following to the end of the file. Make sure to do this on Name Node and all Data Nodes:

#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_INSTALL/lib”
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_HOME=$HADOOP_INSTALL

export HDFS_NAMENODE_USER=hduser
export HDFS_DATANODE_USER=hduser
export HDFS_SECONDARYNAMENODE_USER=hduser

#HADOOP VARIABLES END

  1. source ~/.bashrc
  2. javac –version
  3. which javac
  4. readlink –f /usr/bin/javac

This basically validates that bashrc update worked!
javac should return “javac 1.8.0_171” or something similar
which javac should return “/usr/bin/javac”
readlink should return “/usr/lib/jvm/java-8-openjdk-amd64/bin/javac”

Memory Tools

There is an application from HortonWorks you can download which can help get you started on how you should setup memory utilization for yarn. I found it’s a great starting point but you need to tweak it to work for what you need on your specific case.

  1. wget http://public-repo-1.hortonworks.com/HDP/tools/2.6.0.3/hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
  2. tar zxvf hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
  3. cd hdp_manual_install_rpm_helper_files-2.6.0.3.8/
  4. sudo apt-get install python2.7
  5. python2.7 scripts/yarn-utils.py -c 5 -m 13 -d 1 -k False

-c is for how many cores you have
-m is for how much memory you have
-d is for how many disks you have
False is if you are running HBASE. True if you are.

After the script is ran it will give you guidelines on yarn/mapreduce settings. See below for example. Remember they are guidelines. Tweak as needed.
Now the real fun begins!!! Remember that these settings are what worked for me and you may need to adjust them.

 

hadoop-env.sh

  1. nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh

You will see JAVA_HOME near the beginning of the file you will need to change that to where java is installed on your system.

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HEAPSIZE=1000
export HADOOP_NAMENODE_OPTS=”-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,DRFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,RFAAUDIT} $HADOOP_NAMENODE_OPTS”
export HADOOP_SECONDARYNAMENODE_OPTS=$HADOOP_NAMENODE_OPTS
export HADOOP_CLIENT_OPTS=”-Xmx1024m $HADOOP_CLIENT_OPTS”

  1. mkdir –p /app/hadoop/tmp

This is the temp directory hadoop uses

  1. chown hduser:hduser /app/hadoop/tmp

core-site.xml

Click here to view the docs.

  1. nano /usr/local/hadoop/etc/hadoop/core-site.xml

This file contains configuration properties that Hadoop uses when starting up. By default it will look like . This will need to be changed.

  1. <configuration>
  2.       <property>
  3.             <name>fs.defaultFS</name>
  4.             <value>hdfs://NAMENODE:54310</value>
  5.             <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming
  6. the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description>
  7.       </property>
  8.       <property>
  9.             <name>hadoop.tmp.dir</name>
  10.             <value>/app/hadoop/tmp</value>
  11.       </property>
  12.       <property>
  13.             <name>hadoop.proxyuser.hduser.hosts</name>
  14.             <value>*</value>
  15.       </property>
  16.       <property>
  17.             <name>hadoop.proxyuser.hduser.groups</name>
  18.             <value>*</value>
  19.       </property>
  20. </configuration>

yarn-site.xml

Click here to view the docs.

  1. nano /usr/local/hadoop/etc/hadoop/yarn-site.xml
  1. <configuration>
  2.       <property>
  3.             <name>yarn.nodemanager.aux-services</name>
  4.             <value>mapreduce_shuffle</value>
  5.       </property>
  6.       <property>
  7.             <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
  8.       </property>
  9.       <property>
  10.             <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
  11.             <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  12.       </property>
  13.       <property>
  14.             <name>yarn.nodemanager.resource.memory-mb</name>
  15.             <value>12288</value>
  16.             <final>true</final>
  17.       </property>
  18.       <property>
  19.             <name>yarn.scheduler.minimum-allocation-mb</name>
  20.             <value>4096</value>
  21.             <final>true</final>
  22.       </property>
  23.       <property>
  24.             <name>yarn.scheduler.maximum-allocation-mb</name>
  25.             <value>12288</value>
  26.             <final>true</final>
  27.       </property>
  28.       <property>
  29.             <name>yarn.app.mapreduce.am.resource.mb</name>
  30.             <value>4096</value>
  31.       </property>
  32.       <property>
  33.             <name>yarn.app.mapreduce.am.command-opts</name>
  34.             <value>-Xmx3276m</value>
  35.       </property>
  36.       <property>
  37.             <name>yarn.nodemanager.local-dirs</name>
  38.             <value>/app/hadoop/tmp/nm-local-dir</value>
  39.       </property>
  40.       <!--LOG-->
  41.       <property>
  42.             <name>yarn.log-aggregation-enable</name>
  43.             <value>true</value>
  44.       </property>
  45.       <property>
  46.             <description>Where to aggregate logs to.</description>
  47.             <name>yarn.nodemanager.remote-app-log-dir</name>
  48.             <value>/tmp/yarn/logs</value>
  49.       </property>
  50.       <property>
  51.             <name>yarn.log-aggregation.retain-seconds</name>
  52.             <value>604800</value>
  53.       </property>
  54.       <property>
  55.             <name>yarn.log-aggregation.retain-check-interval-seconds</name>
  56.             <value>86400</value>
  57.       </property>
  58.       <property>
  59.             <name>yarn.log.server.url</name>
  60.             <value>http://NAMENODE:19888/jobhistory/logs/</value>
  61.       </property>
  62.       
  63.       <!--URLs-->
  64.       <property>
  65.             <name>yarn.resourcemanager.resource-tracker.address</name>
  66.             <value>${yarn.resourcemanager.hostname}:8025</value>
  67.       </property>
  68.       <property>
  69.             <name>yarn.resourcemanager.scheduler.address</name>
  70.             <value>${yarn.resourcemanager.hostname}:8030</value>
  71.       </property>
  72.       <property>
  73.             <name>yarn.resourcemanager.address</name>
  74.             <value>${yarn.resourcemanager.hostname}:8050</value>
  75.       </property>
  76.       <property>
  77.             <name>yarn.resourcemanager.admin.address</name>
  78.             <value>${yarn.resourcemanager.hostname}:8033</value>
  79.       </property>
  80.       <property>
  81.             <name>yarn.resourcemanager.webapp.address</name>
  82.             <value>${yarn.nodemanager.hostname}:8088</value>
  83.       </property>
  84.       <property>
  85.             <name>yarn.nodemanager.hostname</name>
  86.             <value>0.0.0.0</value>
  87.       </property>
  88.       <property>
  89.             <name>yarn.nodemanager.address</name>
  90.             <value>${yarn.nodemanager.hostname}:0</value>
  91.       </property>
  92.       <property>
  93.             <name>yarn.nodemanager.webapp.address</name>
  94.             <value>${yarn.nodemanager.hostname}:8042</value>
  95.       </property>
  96. </configuration>

By default it will look like . This will need to be changed.

mapred-site.xml

Click here to view the docs. By default, the /usr/local/hadoop/etc/hadoop/ folder contains /usr/local/hadoop/etc/hadoop/mapred-site.xml.template file which has to be renamed/copied with the name mapred-site.xml By default it will look like . This will need to be changed.

  1. cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
  2.  
  3. nano /usr/local/hadoop/etc/hadoop/mapred-site.xml
  1. <configuration>
  2.       <property>
  3.             <name>mapreduce.framework.name</name>
  4.             <value>yarn</value>
  5.       </property>
  6.       <property>
  7.             <name>mapreduce.jobhistory.address</name>
  8.             <value>0.0.0.0:10020</value>
  9.       </property>
  10.       <property>
  11.             <name>mapreduce.jobhistory.webapp.address</name>
  12.             <value>0.0.0.0:19888</value>
  13.       </property>
  14.       <property>
  15.             <name>mapreduce.jobtracker.address</name>
  16.             <value>0.0.0.0:54311</value>
  17.       </property>
  18.       <property>
  19.             <name>mapreduce.jobhistory.admin.address</name>
  20.             <value>0.0.0.0:10033</value>
  21.       </property>
  22.       <!-- Memory and concurrency tuning -->
  23.       <property>
  24.             <name>mapreduce.map.memory.mb</name>
  25.             <value>4096</value>
  26.       </property>
  27.       <property>
  28.             <name>mapreduce.map.java.opts</name>
  29.             <value>-server -Xmx3276m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value>
  30.       </property>
  31.       <property>
  32.             <name>mapreduce.reduce.memory.mb</name>
  33.             <value>4096</value>
  34.       </property>
  35.       <property>
  36.             <name>mapreduce.reduce.java.opts</name>
  37.             <value>-server -Xmx3276m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value>
  38.       </property>
  39.       <property>
  40.             <name>mapreduce.reduce.shuffle.input.buffer.percent</name>
  41.             <value>0.5</value>
  42.       </property>
  43.       <property>
  44.             <name>mapreduce.task.io.sort.mb</name>
  45.             <value>600</value>
  46.       </property>
  47.       <property>
  48.             <name>mapreduce.task.io.sort.factor</name>
  49.             <value>1638</value>
  50.       </property>
  51.       <property>
  52.             <name>mapreduce.map.sort.spill.percent</name>
  53.             <value>0.50</value>
  54.       </property>
  55.       <property>
  56.             <name>mapreduce.map.speculative</name>
  57.             <value>false</value>
  58.       </property>
  59.       <property>
  60.             <name>mapreduce.reduce.speculative</name>
  61.             <value>false</value>
  62.       </property>
  63.       <property>
  64.             <name>mapreduce.task.timeout</name>
  65.             <value>1800000</value>
  66.       </property>
  67. </configuration>

yarn-env.sh

  1. nano /usr/local/hadoop/etc/hadoop/yarn-env.sh

Change or uncomment or add the following:

JAVA_HEAP_MAX=Xmx2000m
HADOOP_OPTS=”$HADOOP_OPTS-server -Dhadoop.log.dir=$YARN_LOG_DIR”
HADOOP_OPTS=”$HADOOP_OPTS-Djava.net.preferIPv4Stack=true”

Master

Add the namenode hostname.

  1. nano /usr/local/hadoop/etc/hadoop/masters

APPLY THE FOLLOWING TO THE NAMENODE ONLY

Slaves

Add namenode hostname and all datanodes hostname.

  1. nano /usr/local/hadoop/etc/hadoop/slaves

hdfs-site.xml

Click here to view the docs. By default it will look like . This will need to be changed. The /usr/local/hadoop/etc/hadoop/hdfs-site.xml file needs to be configured for each host in the cluster that is being used. Before editing this file, we need to create the namenode directory.

  1. mkdir -p /usr/local/hadoop_store/data/namenode
  2. chown -R hduser:hduser /usr/local/hadoop_store
  3. nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  1. <configuration>
  2.       <property>
  3.             <name>dfs.replication</name>
  4.             <value>3</value>
  5.             <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.</description>
  6.       </property>
  7.       <property>
  8.             <name>dfs.permissions</name>
  9.             <value>false</value>
  10.       </property>
  11.       <property>
  12.             <name>dfs.namenode.name.dir</name>
  13.             <value>file:/usr/local/hadoop_store/data/namenode</value>
  14.       </property>
  15.       <property>
  16.             <name>dfs.datanode.use.datanode.hostname</name>
  17.             <value>false</value>
  18.       </property>
  19.       <property>
  20.             <name>dfs.blocksize</name>
  21.             <value>128m</value>
  22.       </property>
  23.       <property>
  24.             <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
  25.             <value>false</value>
  26.       </property>
  27.       
  28.       <!-- URL -->
  29.       <property>
  30.             <name>dfs.namenode.http-address</name>
  31.             <value>${dfs.namenode.http-bind-host}:50070</value>
  32.             <description>Your NameNode hostname for http access.</description>
  33.       </property>
  34.       <property>
  35.             <name>dfs.namenode.secondary.http-address</name>
  36.             <value>${dfs.namenode.http-bind-host}:50090</value>
  37.             <description>Your Secondary NameNode hostname for http access.</description>
  38.       </property>
  39.       <property>
  40.             <name>dfs.datanode.http.address</name>
  41.             <value>${dfs.namenode.http-bind-host}:50075</value>
  42.       </property>
  43.       <property>
  44.             <name>dfs.datanode.address</name>
  45.             <value>${dfs.namenode.http-bind-host}:50076</value>
  46.       </property>
  47.       <property>
  48.             <name>dfs.namenode.http-bind-host</name>
  49.             <value>0.0.0.0</value>
  50.       </property>
  51.       <property>
  52.             <name>dfs.namenode.rpc-bind-host</name>
  53.             <value>0.0.0.0</value>
  54.       </property>
  55.       <property>
  56.              <name>dfs.namenode.servicerpc-bind-host</name>
  57.              <value>0.0.0.0</value>
  58.       </property>
  59. &lt;/configuration>

APPLY THE FOLLOWING TO THE DATANODE(s) ONLY

Slaves

Add only that datanodes hostname.

  1. nano /usr/local/hadoop/etc/hadoop/slaves

hdfs-site.xml

The /usr/local/hadoop/etc/hadoop/hdfs-site.xml file needs to be configured for each host in the cluster that is being used. Before editing this file, we need to create the datanode directory.
By default it will look like . This will need to be changed.

  1. mkdir -p /usr/local/hadoop_store/data/datanode
  2. chown -R hduser:hduser /usr/local/hadoop_store
  3. nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  1. <configuration>
  2.       <property>
  3.             <name>dfs.replication</name>
  4.             <value>3</value>
  5.             <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.</description>
  6.       </property>
  7.       <property>
  8.             <name>dfs.permissions</name>
  9.             <value>false</value>
  10.       </property>
  11.       <property>
  12.             <name>dfs.blocksize</name>
  13.             <value>128m</value>
  14.       </property>
  15.       <property>
  16.             <name>dfs.datanode.data.dir</name>
  17.             <value>file:/usr/local/hadoop_store/data/datanode</value>
  18.       </property>
  19.       <property>
  20.             <name>dfs.datanode.use.datanode.hostname</name>
  21.             <value>false</value>
  22.       </property>
  23.       <property>
  24.             <name>dfs.namenode.http-address</name>
  25.             <value>${dfs.namenode.http-bind-host}:50070</value>
  26.             <description>Your NameNode hostname for http access.</description>
  27.       </property>
  28.       <property>
  29.             <name>dfs.namenode.secondary.http-address</name>
  30.             <value>${dfs.namenode.http-bind-host}:50090</value>
  31.             <description>Your Secondary NameNode hostname for http access.</description>
  32.       </property>
  33.       <property>
  34.             <name>dfs.datanode.http.address</name>
  35.             <value>${dfs.namenode.http-bind-host}:50075</value>
  36.       </property>
  37.       <property>
  38.             <name>dfs.datanode.address</name>
  39.             <value>${dfs.namenode.http-bind-host}:50076</value>
  40.       </property>
  41.       <property>
  42.             <name>dfs.namenode.http-bind-host</name>
  43.             <value>0.0.0.0</value>
  44.       </property>
  45.       <property>
  46.             <name>dfs.namenode.rpc-bind-host</name>
  47.             <value>0.0.0.0</value>
  48.       </property>
  49.       <property>
  50.              <name>dfs.namenode.servicerpc-bind-host</name>
  51.              <value>0.0.0.0</value>
  52.       </property>
  53. </configuration>

You need to allow the pass-through for all ports necessary. If you have the Ubuntu firewall on.

  1. sudo ufw allow 50070
  2. sudo ufw allow 8088

Format Cluster:
Only do this if NO data is present. All data will be destroyed when the following is done.
This is to be done on NAMENODE ONLY!

  1. hdfs namenode -format

Start The Cluster:
You can now start the cluster.
You do this from the NAMENODE ONLY.

  1. start-dfs.sh
  2. start-yarn.sh
  3. mapred --config $HADOOP_CONF_DIR --daemon start historyserver

If the above three commands didn’t work something went wrong. As it should have found the scripts located /usr/local/hadoop/sbin/ directory.

Cron Job:
You should probably setup a cron job to start the cluster when you reboot.

  1. crontab –e

@reboot /usr/local/hadoop/sbin/start-dfs.sh > /home/hduser/dfs-start.log 2>&1
@reboot /usr/local/hadoop/sbin/start-yarn.sh > /home/hduser/yarn-start.log 2>&1
@reboot /usr/local/hadoop/bin/mapred –config $HADOOP_CONF_DIR –daemon start historyserver > /home/hduser/history-stop.log 2>&1

Verification:
To check that everything is working as it should run “jps” on the NAMENODE. It should return something like the following where the pid will be different:

  1. jps

You could also run “netstat -plten | grep java” or “lsof –i :50070” and “lsof –i :8088”.

Picked up _JAVA_OPTIONS: -Xms3g -Xmx10g -Djava.net.preferIPv4Stack=true
12007 SecondaryNameNode
13090 Jps
12796 JobHistoryServer
12261 ResourceManager
11653 NameNode
12397 NodeManager
11792 DataNode

You can check the DATA NODES by ssh into each one and running “jps”. It should return something like the following where the pid will be different:

Picked up _JAVA_OPTIONS: -Xms3g -Xmx10g -Djava.net.preferIPv4Stack=true
3218 Jps
2215 NodeManager
2411 DataNode

If for any reason only of the services is not running you need to review the logs. They can be found at /usr/local/hadoop/logs/. If it’s ResourceManager that isn’t running then look at file that has “yarn” and “resourcemanager” in it.

WARNING:
Never reboot the system without first stopping the cluster. When the cluster shuts down it is safe to reboot it. Also if you configured a cronjob @reboot you should make sure the DATANODES are up and running first before starting the NAMENODE that way it automatically starts the DATANODES for you

Web Ports:

NameNode

  • 50070: HDFS Namenode
  • 50075: HDFS Datanode
  • 50090: HDFS Secondary Namenode
  • 8088: Resource Manager
  • 19888: Job History

DataNode

  • 50075: HDFS Datanode

NetStat

To check that all the Hadoop ports are available on which IP run the following.

  1. sudo netstat -ltnp

Port Check

If for some reason you are having issues connecting to a Hadoop port then run the following command as you try and connect via the port.

  1. sudo tcpdump -n -tttt -i eth1 port 50070

References

I used a lot of different resources and reference material on this. However I did not save all the relevant links I used. Below are just a few I used. There was various blog posts about memory utilization, etc.

HDFS/Yarn/MapRed: Kerberize/SSL

In this tutorial I will show you how to use Kerberos/SSL with HDFS/Yarn/MapRed. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Hadoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

  1. cd /etc/security/keytabs/
  2.  
  3. sudo kadmin.local
  4.  
  5. #You can list princepals
  6. listprincs
  7.  
  8. #Create the following principals
  9. addprinc -randkey nn/hadoop@REALM.CA
  10. addprinc -randkey jn/hadoop@REALM.CA
  11. addprinc -randkey dn/hadoop@REALM.CA
  12. addprinc -randkey sn/hadoop@REALM.CA
  13. addprinc -randkey nm/hadoop@REALM.CA
  14. addprinc -randkey rm/hadoop@REALM.CA
  15. addprinc -randkey jhs/hadoop@REALM.CA
  16. addprinc -randkey HTTP/hadoop@REALM.CA
  17.  
  18. #We are going to create a user to access with later
  19. addprinc -pw hadoop myuser/hadoop@REALM.CA
  20. xst -k myuser.keytab myuser/hadoop@REALM.CA
  21.  
  22. #Create the keytab files.
  23. #You will need these for Hadoop to be able to login
  24. xst -k nn.service.keytab nn/hadoop@REALM.CA
  25. xst -k jn.service.keytab jn/hadoop@REALM.CA
  26. xst -k dn.service.keytab dn/hadoop@REALM.CA
  27. xst -k sn.service.keytab sn/hadoop@REALM.CA
  28. xst -k nm.service.keytab nm/hadoop@REALM.CA
  29. xst -k rm.service.keytab rm/hadoop@REALM.CA
  30. xst -k jhs.service.keytab jhs/hadoop@REALM.CA
  31. xst -k spnego.service.keytab HTTP/hadoop@REALM.CA

Set Keytab Permissions/Ownership

  1. sudo chown root:hadoopuser /etc/security/keytabs/*
  2. sudo chmod 750 /etc/security/keytabs/*

Stop the Cluster

  1. stop-dfs.sh
  2. stop-yarn.sh
  3. mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

Hosts Update

  1. sudo nano /etc/hosts
  2.  
  3. #Remove 127.0.1.1 line
  4.  
  5. #Change 127.0.0.1 to the following
  6. #Notice how realm.ca is there its because we need to tell where that host resides
  7. 127.0.0.1 realm.ca hadoop localhost

hadoop-env.sh

We don’t set the HADOOP_SECURE_DN_USER because we are going to use Kerberos

  1. sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh
  2.  
  3. #Locate "export ${HADOOP_SECURE_DN_USER}=${HADOOP_SECURE_DN_USER}"
  4. #and change to
  5.  
  6. export HADOOP_SECURE_DN_USER=

core-site.xml

  1. nano /usr/local/hadoop/etc/hadoop/core-site.xml
  2.  
  3. <configuration>
  4. 	<property>
  5. 		<name>fs.defaultFS</name>
  6. 		<value>hdfs://NAMENODE:54310</value>
  7. 		<description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming
  8. 		the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description>
  9. 	</property>
  10. 	<property>
  11. 		<name>hadoop.tmp.dir</name>
  12. 		<value>/app/hadoop/tmp</value>
  13. 	</property>
  14. 	<property>
  15. 		<name>hadoop.proxyuser.hadoopuser.hosts</name>
  16. 		<value>*</value>
  17. 	</property>
  18. 	<property>
  19. 		<name>hadoop.proxyuser.hadoopuser.groups</name>
  20. 		<value>*</value>
  21. 	</property>
  22. 	<property>
  23. 		<name>hadoop.security.authentication</name>
  24. 		<value>kerberos</value> <!-- A value of "simple" would disable security. -->
  25. 	</property>
  26. 	<property>
  27. 		<name>hadoop.security.authorization</name>
  28. 		<value>true</value>
  29. 	</property>
  30. 	<property>
  31. 		<name>hadoop.security.auth_to_local</name>
  32. 		<value>
  33. 		RULE:[2:$1@$0](nn/.*@.*REALM.TLD)s/.*/hdfs/
  34. 		RULE:[2:$1@$0](jn/.*@.*REALM.TLD)s/.*/hdfs/
  35. 		RULE:[2:$1@$0](dn/.*@.*REALM.TLD)s/.*/hdfs/
  36. 		RULE:[2:$1@$0](sn/.*@.*REALM.TLD)s/.*/hdfs/
  37. 		RULE:[2:$1@$0](nm/.*@.*REALM.TLD)s/.*/yarn/
  38. 		RULE:[2:$1@$0](rm/.*@.*REALM.TLD)s/.*/yarn/
  39. 		RULE:[2:$1@$0](jhs/.*@.*REALM.TLD)s/.*/mapred/
  40. 		DEFAULT
  41. 		</value>
  42. 	</property>
  43. 	<property>
  44. 		<name>hadoop.rpc.protection</name>
  45. 		<value>integrity</value>
  46. 	</property>
  47. 	<property>
  48. 		<name>hadoop.ssl.require.client.cert</name>
  49. 		<value>false</value>
  50. 	</property>
  51. 	<property>
  52. 		<name>hadoop.ssl.hostname.verifier</name>
  53. 		<value>DEFAULT</value>
  54. 	</property>
  55. 	<property>
  56. 		<name>hadoop.ssl.keystores.factory.class</name>
  57. 		<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
  58. 	</property>
  59. 	<property>
  60. 		<name>hadoop.ssl.server.conf</name>
  61. 		<value>ssl-server.xml</value>
  62. 	</property>
  63. 	<property>
  64. 		<name>hadoop.ssl.client.conf</name>
  65. 		<value>ssl-client.xml</value>
  66. 	</property>
  67. 	<property>
  68. 		<name>hadoop.rpc.protection</name>
  69. 		<value>integrity</value>
  70. 	</property>
  71. </configuration>

ssl-server.xml

Change ssl-server.xml.example to ssl-server.xml

  1. cp /usr/local/hadoop/etc/hadoop/ssl-server.xml.example /usr/local/hadoop/etc/hadoop/ssl-server.xml
  2.  
  3. nano /usr/local/hadoop/etc/hadoop/ssl-server.xml

Update properties

  1. <configuration>
  2. 	<property>
  3. 		<name>ssl.server.truststore.location</name>
  4. 		<value>/etc/security/serverKeys/truststore.jks</value>
  5. 		<description>Truststore to be used by NN and DN. Must be specified.</description>
  6. 	</property>
  7. 	<property>
  8. 		<name>ssl.server.truststore.password</name>
  9. 		<value>PASSWORD</value>
  10. 		<description>Optional. Default value is "".</description>
  11. 	</property>
  12. 	<property>
  13. 		<name>ssl.server.truststore.type</name>
  14. 		<value>jks</value>
  15. 		<description>Optional. The keystore file format, default value is "jks".</description>
  16. 	</property>
  17. 	<property>
  18. 		<name>ssl.server.truststore.reload.interval</name>
  19. 		<value>10000</value>
  20. 		<description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds).</description>
  21. 	</property>
  22. 	<property>
  23. 		<name>ssl.server.keystore.location</name>
  24. 		<value>/etc/security/serverKeys/keystore.jks</value>
  25. 		<description>Keystore to be used by NN and DN. Must be specified.</description>
  26. 	</property>
  27. 	<property>
  28. 		<name>ssl.server.keystore.password</name>
  29. 		<value>PASSWORD</value>
  30. 		<description>Must be specified.</description>
  31. 	</property>
  32. 	<property>
  33. 		<name>ssl.server.keystore.keypassword</name>
  34. 		<value>PASSWORD</value>
  35. 		<description>Must be specified.</description>
  36. 	</property>
  37. 	<property>
  38. 		<name>ssl.server.keystore.type</name>
  39. 		<value>jks</value>
  40. 		<description>Optional. The keystore file format, default value is "jks".</description>
  41. 	</property>
  42. 	<property>
  43. 		<name>ssl.server.exclude.cipher.list</name>
  44. 		<value>TLS_ECDHE_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,
  45. 		SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,
  46. 		SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,
  47. 		SSL_RSA_WITH_RC4_128_MD5</value>
  48. 		<description>Optional. The weak security cipher suites that you want excluded from SSL communication.</description>
  49. 	</property>
  50. </configuration>

ssl-client.xml

Change ssl-client.xml.example to ssl-client.xml

  1. cp /usr/local/hadoop/etc/hadoop/ssl-client.xml.example /usr/local/hadoop/etc/hadoop/ssl-client.xml
  2.  
  3. nano /usr/local/hadoop/etc/hadoop/ssl-client.xml

Update properties

  1. <configuration>
  2. 	<property>
  3. 		<name>ssl.client.truststore.location</name>
  4. 		<value>/etc/security/serverKeys/truststore.jks</value>
  5. 		<description>Truststore to be used by clients like distcp. Must be specified.</description>
  6. 	</property>
  7. 	<property>
  8. 		<name>ssl.client.truststore.password</name>
  9. 		<value>PASSWORD</value>
  10. 		<description>Optional. Default value is "".</description>
  11. 	</property>
  12. 	<property>
  13. 		<name>ssl.client.truststore.type</name>
  14. 		<value>jks</value>
  15. 		<description>Optional. The keystore file format, default value is "jks".</description>
  16. 	</property>
  17. 	<property>
  18. 		<name>ssl.client.truststore.reload.interval</name>
  19. 		<value>10000</value>
  20. 		<description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds).</description>
  21. 	</property>
  22. 	<property>
  23. 		<name>ssl.client.keystore.location</name>
  24. 		<value></value>
  25. 		<description>Keystore to be used by clients like distcp. Must be specified.</description>
  26. 	</property>
  27. 	<property>
  28. 		<name>ssl.client.keystore.password</name>
  29. 		<value></value>
  30. 		<description>Optional. Default value is "".</description>
  31. 	</property>
  32. 	<property>
  33. 		<name>ssl.client.keystore.keypassword</name>
  34. 		<value></value>
  35. 		<description>Optional. Default value is "".</description>
  36. 	</property>
  37. 	<property>
  38. 		<name>ssl.client.keystore.type</name>
  39. 		<value>jks</value>
  40. 		<description>Optional. The keystore file format, default value is "jks".</description>
  41. 	</property>
  42. </configuration>

mapred-site.xml

Just add the following to the config to let it know the Kerberos keytabs to use.

  1. nano /usr/local/hadoop/etc/hadoop/mapred-site.xml
  2.  
  3. <property>
  4. 	<name>mapreduce.jobhistory.keytab</name>
  5. 	<value>/etc/security/keytabs/jhs.service.keytab</value>
  6. </property>
  7. <property>
  8. 	<name>mapreduce.jobhistory.principal</name>
  9. 	<value>jhs/_HOST@REALM.CA</value>
  10. </property>
  11. <property>
  12. 	<name>mapreduce.jobhistory.http.policy</name>
  13. 	<value>HTTPS_ONLY</value>
  14. </property>

hdfs-site.xml

Add the following properties

  1. nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  2.  
  3. <property>
  4. 	<name>dfs.http.policy</name>
  5. 	<value>HTTPS_ONLY</value>
  6. </property>
  7. <property>
  8. 	<name>hadoop.ssl.enabled</name>
  9. 	<value>true</value>
  10. </property>
  11. <property>
  12. 	<name>dfs.datanode.https.address</name>
  13. 	<value>NAMENODE:50475</value>
  14. </property>
  15. <property>
  16. 	<name>dfs.namenode.https-address</name>
  17. 	<value>NAMENODE:50470</value>
  18. 	<description>Your NameNode hostname for http access.</description>
  19. </property>
  20. <property>
  21. 	<name>dfs.namenode.secondary.https-address</name>
  22. 	<value>NAMENODE:50091</value>
  23. 	<description>Your Secondary NameNode hostname for http access.</description>
  24. </property>
  25. <property>
  26. 	<name>dfs.namenode.https-bind-host</name>
  27. 	<value>0.0.0.0</value>
  28. </property>
  29. <property>
  30. 	<name>dfs.block.access.token.enable</name>
  31. 	<value>true</value>
  32. 	<description> If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanod</description>
  33. </property>
  34. <property>
  35. 	<name>dfs.namenode.kerberos.principal</name>
  36. 	<value>nn/_HOST@REALM.CA</value>
  37. 	<description> Kerberos principal name for the NameNode</description>
  38. </property>
  39. <property>
  40. 	<name>dfs.secondary.namenode.kerberos.principal</name>
  41. 	<value>sn/_HOST@REALM.CA</value>
  42. 	<description>Kerberos principal name for the secondary NameNode.</description>
  43. </property>
  44. <property>
  45. 	<name>dfs.web.authentication.kerberos.keytab</name>
  46. 	<value>/etc/security/keytabs/spnego.service.keytab</value>
  47. 	<description>The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</description>
  48. </property>
  49. <property>
  50. 	<name>dfs.namenode.keytab.file</name>
  51. 	<value>/etc/security/keytabs/nn.service.keytab</value>
  52. 	<description>Combined keytab file containing the namenode service and host principals.</description>
  53. </property>
  54. <property>
  55. 	<name>dfs.datanode.keytab.file</name>
  56. 	<value>/etc/security/keytabs/dn.service.keytab</value>
  57. 	<description>The filename of the keytab file for the DataNode.</description>
  58. </property>
  59. <property>
  60. 	<name>dfs.datanode.kerberos.principal</name>
  61. 	<value>dn/_HOST@REALM.CA</value>
  62. 	<description>The Kerberos principal that the DataNode runs as. "_HOST" is replaced by the real host name.</description>
  63. </property>
  64. <property>
  65. 	<name>dfs.namenode.kerberos.internal.spnego.principal</name>
  66. 	<value>${dfs.web.authentication.kerberos.principal}</value>
  67. </property>
  68. <property>
  69. 	<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  70. 	<value>>${dfs.web.authentication.kerberos.principal}</value>
  71. </property>
  72. <property>
  73. 	<name>dfs.web.authentication.kerberos.principal</name>
  74. 	<value>HTTP/_HOST@REALM.CA</value>
  75. 	<description>The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</description>          
  76. </property>
  77. <property>
  78. 	<name>dfs.data.transfer.protection</name>
  79. 	<value>integrity</value>
  80. </property>
  81. <property>
  82. 	<name>dfs.datanode.address</name>
  83. 	<value>NAMENODE:50010</value>
  84. </property>
  85. <property>
  86. 	<name>dfs.secondary.namenode.keytab.file</name>
  87. 	<value>/etc/security/keytabs/sn.service.keytab</value>
  88. </property>
  89. <property>
  90. 	<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  91. 	<value>HTTP/_HOST@REALM.CA</value>
  92. </property>
  93. <property>
  94. 	<name>dfs.webhdfs.enabled</name>
  95. 	<value>true</value>
  96. </property>

Remove the following properties

  1. dfs.namenode.http-address
  2. dfs.namenode.secondary.http-address
  3. dfs.namenode.http-bind-host

yarn-site.xml

Add the following properties

  1. nano /usr/local/hadoop/etc/hadoop/yarn-site.xml
  2.  
  3. <property>
  4. 	<name>yarn.http.policy</name>
  5. 	<value>HTTPS_ONLY</value>
  6. </property>
  7. <property>
  8. 	<name>yarn.resourcemanager.webapp.https.address</name>
  9. 	<value>${yarn.resourcemanager.hostname}:8090</value>
  10. </property>
  11. <property>
  12. 	<name>yarn.resourcemanager.hostname</name>
  13. 	<value>NAMENODE</value>
  14. </property>
  15. <property>
  16. 	<name>yarn.nodemanager.bind-host</name>
  17. 	<value>0.0.0.0</value>
  18. </property>
  19. <property>
  20. 	<name>yarn.nodemanager.webapp.address</name>
  21. 	<value>${yarn.nodemanager.hostname}:8042</value>
  22. </property>
  23. <property>
  24. 	<name>yarn.resourcemanager.principal</name>
  25. 	<value>rm/_HOST@REALM.CA</value>
  26. </property>
  27. <property>
  28. 	<name>yarn.resourcemanager.keytab</name>
  29. 	<value>/etc/security/keytabs/rm.service.keytab</value>
  30. </property>
  31. <property>
  32. 	<name>yarn.nodemanager.principal</name>
  33. 	<value>nm/_HOST@REALM.CA</value>
  34. </property>
  35. <property>
  36. 	<name>yarn.nodemanager.keytab</name>
  37. 	<value>/etc/security/keytabs/nm.service.keytab</value>
  38. </property>
  39. <property>
  40. 	<name>yarn.nodemanager.hostname</name>
  41. 	<value>NAMENODE</value>
  42. </property>
  43. <property>
  44. 	<name>yarn.resourcemanager.bind-host</name>
  45. 	<value>0.0.0.0</value>
  46. </property>
  47. <property>
  48. 	<name>yarn.timeline-service.bind-host</name>
  49. 	<value>0.0.0.0</value>
  50. </property>

Remove the following properties

  1. yarn.resourcemanager.webapp.address

SSL

Setup SSL Directories

  1. sudo mkdir -p /etc/security/serverKeys
  2. sudo chown -R root:hadoopuser /etc/security/serverKeys/
  3. sudo chmod 755 /etc/security/serverKeys/
  4.  
  5. cd /etc/security/serverKeys

Setup Keystore

  1. sudo keytool -genkey -alias NAMENODE -keyalg RSA -keysize 1024 -dname "CN=NAMENODE,OU=ORGANIZATION_UNIT,C=canada" -keypass PASSWORD -keystore /etc/security/serverKeys/keystore.jks -storepass PASSWORD
  2. sudo keytool -export -alias NAMENODE -keystore /etc/security/serverKeys/keystore.jks -rfc -file /etc/security/serverKeys/NAMENODE.csr -storepass PASSWORD

Setup Truststore

  1. sudo keytool -import -noprompt -alias NAMENODE -file /etc/security/serverKeys/NAMENODE.csr -keystore /etc/security/serverKeys/truststore.jks -storepass PASSWORD

Generate Self Signed Certifcate

  1. sudo openssl genrsa -out /etc/security/serverKeys/NAMENODE.key 2048
  2.  
  3. sudo openssl req -x509 -new -key /etc/security/serverKeys/NAMENODE.key -days 300 -out /etc/security/serverKeys/NAMENODE.pem
  4.  
  5. sudo keytool -keystore /etc/security/serverKeys/keystore.jks -alias NAMENODE -certreq -file /etc/security/serverKeys/NAMENODE.cert -storepass PASSWORD -keypass PASSWORD
  6.  
  7. sudo openssl x509 -req -CA /etc/security/serverKeys/NAMENODE.pem -CAkey /etc/security/serverKeys/NAMENODE.key -in /etc/security/serverKeys/NAMENODE.cert -out /etc/security/serverKeys/NAMENODE.signed -days 300 -CAcreateserial

Setup File Permissions

  1. sudo chmod 440 /etc/security/serverKeys/*
  2. sudo chown root:hadoopuser /etc/security/serverKeys/*

Start the Cluster

  1. start-dfs.sh
  2. start-yarn.sh
  3. mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

Create User Directory

  1. kinit -kt /etc/security/keytabs/myuser.keytab myuser/hadoop@REALM.CA
  2. #ensure the login worked
  3. klist
  4.  
  5. #Create hdfs directory now
  6. hdfs dfs -mkdir /user
  7. hdfs dfs -mkdir /user/myuser
  8.  
  9. #remove kerberos ticket
  10. kdestroy

URL

https://NAMENODE:50470
https://NAMENODE:50475
https://NAMENODE:8090

References

https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/admin_ssl_hbase_mr_yarn_hdfs_web.html

HortonWorks: Install YARN/MR

This entry is part 6 of 7 in the series HortonWorks

This tutorial guides you through installing YARN/MapReduce on Hortonworks using a multi node cluster setup with Ubuntu OS.

Step 1: Go to “Stack and Version”. Then click “Add Service” on YARN. You will notice that “MapReduce2” comes with it.

Step 2: Assign Masters I usually put the ResourceManager, History Server and App Timeline Server all on the secondary namenode. But it is totally up to you how you setup your environment.

Step 3: Assign Slaves and Clients I put NodeManagers on all the datanodes and Client’s on all servers. Up to you though. This is what worked for me and my requirements.

Step 4: During Customize Services you may get the warning that Ambari Metrics “hbase_master_heapsize” needs to be increased. I recommend doing this change but it’s up to you and what makes sense in your environment.

Step 5: Follow the remaining steps and installation should complete with no issues. Should an issue arise review the error and if it was just a turning on connection error then you may not have any issues and it just needs all services to be stopped and started again. Please not Ambari Metrics may report errors but they should clear in around 15 minutes.

 

Hadoop: Commands

Below is a list of all the commands I have had to use while working with Hadoop. If you have any other ones that are not listed here please feel free to add them in or if you have updates to ones below.

Move Files:

  1.  hadoop fs -mv /OLD_DIR/* /NEW_DIR/

Sort Files By Size. Note this is for viewing information only on terminal. It has no affect on the files or the way they are displayed via web ui:

  1.  hdfs fsck /logs/ -files | grep "/FILE_DIR/" | grep -v "<dir>" | gawk '{print $2, $1;}' | sort –n

Display system information:

  1.  hdfs fsck /FILE_dir/ -files

Remove folder with all files in it:

  1.  hadoop fs -rm -R hdfs:///DIR_TO_REMOVE

Make folder:

  1.  hadoop fs -mkdir hdfs:///NEW_DIR

Remove one file:

  1.  hadoop fs -rm hdfs:///DIR/FILENAME.EXTENSION

Copy all file from directory outside of HDFS to HDFS:

  1.  hadoop fs -copyFromLocal LOCAL_DIR hdfs:///DIR

Copy files from HDFS to local directory:

  1.  hadoop dfs -copyToLocal hdfs:///DIR/REGPATTERN LOCAL_DIR

Kill a running MR job:

  1.  hadoop job -kill job_1461090210469_0003

You could also do that via the 8088 web ui interface

Kill yarn application:

  1.  yarn application -kill application_1461778722971_0001

Check status of DATANODES. Check “Under Replicated blocks” field. If you have any you should probably rebalance:

  1.  hadoop dfsadmin –report

Number of files in HDFS directory:

  1.  hadoop fs -count -q hdfs:///DIR

-q is optional – Gives columns QUOTA, REMAINING_QUATA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILE_NAME

Rename directory:

  1.  hadoop fs -mv hdfs:///OLD_NAME hdfs:///NEW_NAME

Change replication factor on files:

  1.  hadoop fs -setrep -R 3 hdfs:///DIR

3 is the replication number.
You can choose a file if you want

Get yarn log. You can also view via web ui 8088:

  1.  yarn logs -applicationId application_1462141864581_0016

Refresh Nodes:

  1.  hadoop dfsadmin –refreshNodes

Report of blocks and their locations:

  1.  hadoop fsck / -files -blocks –locations

Find out where a particular file is located with blocks:

  1.  hadoop fsck /DIR/FILENAME -files -locations –blocks

Fix under replicated blocks. First command gets the blocks that are under replicated. The second sets replication to 2 for those files. You might have to restart the dfs to see a change from dfsadmin –report:

  1.  hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
  2.  
  3. for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 2 $hdfsfile; done

Show all the classpaths associated to hadoop:

  1.  hadoop classpath

Hadoop 2.9.1: Installation

I have been working with Hadoop 2.9.1 for over a year and have learned much on the installation of Hadoop in a multi node cluster environment. I would like to share what I have learned and applied in the hopes that it will help someone else configure their system. The deployment I have done is to have a Name Node and 1-* DataNodes on Ubuntu 16.04 assuming 5 cpu and 13GB RAM. I will put all commands used in this tutorial right down to the very basics for those that are new to Ubuntu.

NOTE: Sometimes you may have to use “sudo” in front of the command. I also use nano for this article for beginners but you can use any editor you prefer (ie: vi). Also this article does not take into consideration any SSL, kerberos, etc. For all purposes here Hadoop will be open without having to login, etc.

Additional Setup/Configurations to Consider:

Zookeeper: It is also a good idea to use ZooKeeper to synchronize your configuration

Secondary NameNode: This should be done on a seperate server and it’s function is to take checkpoints of the namenodes file system.

Rack Awareness: Fault tolerance to ensure blocks are placed as evenly as possible on different racks if they are available.

Apply the following to all NameNode and DataNodes unless otherwise directed:

Hadoop User:
For this example we will just use hduser as our group and user for simplicity sake.
The “-a” on usermod is for appending to a group used with –G for which groups

  1. addgroup hduser
  2. sudo gpasswd -a $USER sudo
  3. usermod –a –G sudo hduser

Install JDK:

  1. apt-get update
  2. apt-get upgrade
  3. apt-get install default-jdk

Install SSH:

  1. apt-get install ssh
  2. which ssh
  3. which sshd

These two commands will check that ssh installed correctly and will return “/usr/bin/ssh” and “/usr/bin/sshd”

  1. java -version

You use this to verify that java installed correctly and will return something like the following.

openjdk version “1.8.0_171”
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

System Configuration

  1. nano ~/.bashrc

The .bashrc is a script that is executed when a terminal session is started.
Add the following line to the end and save because Hadoop uses IPv4.

export _JAVA_OPTIONS=’-XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true’

  1. source ~/.bashrc

sysctl.conf

Disable ipv6 as it causes issues in getting your server up and running.

  1. nano /etc/sysctl.conf

Add the following to the end and save

  1. net.ipv6.conf.all.disable_ipv6 = 1
  2. net.ipv6.conf.default.disable_ipv6 = 1
  3. net.ipv6.conf.lo.disable_ipv6 = 1
  4. #Change eth0 to what ifconfig has
  5. net.ipv6.conf.eth0.disable_ipv6 = 1

Close sysctl

  1. sysctl -p
  2. cat /proc/sys/net/ipv6/conf/all/disable_ipv6
  3. reboot

If all the above disabling IPv6 configuration was successful you should get “1” returned.
Sometimes you can reach open file descriptor limit and open file limit. If you do encounter this issue you might have to set the ulimit and descriptor limit. For this example I have set some values but you will have to figure out the best numbers for your specific case.

If you get “cannot stat /proc/sys/-p: No such file or directory”. Then you need to add /sbin/ to PATH.

  1. sudo nano ~/.bashrc
  2. export PATH=$PATH:/sbin/
  1. nano /etc/sysctl.conf

fs.file-max = 500000

  1. sysctl –p

limits.conf

  1. nano /etc/security/limits.conf

* soft nofile 60000
* hard nofile 60000

  1.  reboot

Test Limits

You can now test the limits you applied to make sure they took.

  1. ulimit -a
  2. more /proc/sys/fs/file-max
  3. more /proc/sys/fs/file-nr
  4. lsof | wc -l

file-max: Current open file descriptor limit
file-nr: How many file descriptors are currently being used
lsof wc: How many files are currently open

You might be wondering why we installed ssh at the beginning. That is because Hadoop uses ssh to access its nodes. We need to eliminate the password requirement by setting up ssh certificates. If asked for a filename just leave it blank and confirm with enter.

  1. su hduser

If not already logged in as the user we created in the Hadoop user section.

  1. ssh-keygen –t rsa –P ""

You will get the below example as well as the fingerprint and randomart image.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory ‘/home/hduser/.ssh’.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.

  1. cat $HOME/.ssh/id-rsa.pub >> $HOME/.ssh/authorized_keys

You may get “No such file or directory”. It is most likely just the id-rsa.pub filename. Look in the .ssh directory for the name it most likely will be “id_rsa.pub”.

This will add the newly created key to the list of authorized keys so that Hadoop can use SSH without prompting for a password.
Now we check that it worked by running “ssh localhost”. When prompted with if you should continue connecting type “yes” and enter. You will be permanently added to localhost
Once we have done this on all Name Node and Data Node you should run the following command from the Name Node to each Data Node.

  1. ssh-copy-id –i ~/.ssh/id_rsa.pub hduser@DATANODEHOSTNAME
  2. ssh DATANODEHOSTNAME

/etc/hosts Update

We need to update the hosts file.

  1. sudo nano /etc/hosts
  2.  
  3. #Comment out line "127.0.0.1 localhost"
  4.  
  5. 127.0.0.1 HOSTNAME localhost

Now we are getting to the part we have been waiting for.

Hadoop Installation:

NAMENODE: You will see this in the config files below and it can be the hostname, the static ip or it could be 0.0.0.0 so that all TCP ports will be bound to all IP’s of the server. You should also note that the masters and slaves file later on in this tutorial can still be the hostname.

Note: You could run rsync after setting up the Name Node Initial configuration to each Data Node if you want. This would save initial hadoop setup time. You do that by running the following command:

  1. rsync –a /usr/local/hadoop/ hduser@DATANODEHOSTNAME:/usr/local/hadoop/

Download & Extract:

  1. wget http://mirrors.sonic.net/apache/hadoop/common/hadoop-2.9.1/hadoop-2.9.1.tar.gz
  2. tar xvzf hadoop-2.9.1.tar.gz
  3. sudo mv hadoop-2.9.1/ /usr/local/hadoop
  4. chown –R hduser:hduser /usr/local/hadoop
  5. update-alternatives --config java

Basically the above downloads, extracts, moves the extracted hadoop directory to the /usr/local directory, if the hduser doesn’t own the newly created directory then switch ownership
and tells us the path where java was been installed to to set the JAVA_HOME environment variable. It should return something like the following:

There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

  1. nano ~/.bashrc

Add the following to the end of the file. Make sure to do this on Name Node and all Data Nodes:

#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_INSTALL/lib”
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_HOME=$HADOOP_INSTALL
#HADOOP VARIABLES END

  1. source ~/.bashrc
  2. javac –version
  3. which javac
  4. readlink –f /usr/bin/javac

This basically validates that bashrc update worked!
javac should return “javac 1.8.0_171” or something similar
which javac should return “/usr/bin/javac”
readlink should return “/usr/lib/jvm/java-8-openjdk-amd64/bin/javac”

Memory Tools

There is an application from HortonWorks you can download which can help get you started on how you should setup memory utilization for yarn. I found it’s a great starting point but you need to tweak it to work for what you need on your specific case.

  1. wget http://public-repo-1.hortonworks.com/HDP/tools/2.6.0.3/hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
  2. tar zxvf hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
  3. cd hdp_manual_install_rpm_helper_files-2.6.0.3.8/
  4. sudo apt-get install python2.7
  5. python2.7 scripts/yarn-utils.py -c 5 -m 13 -d 1 -k False

-c is for how many cores you have
-m is for how much memory you have
-d is for how many disks you have
False is if you are running HBASE. True if you are.

After the script is ran it will give you guidelines on yarn/mapreduce settings. See below for example. Remember they are guidelines. Tweak as needed.
Now the real fun begins!!! Remember that these settings are what worked for me and you may need to adjust them.

 

hadoop-env.sh

  1. nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh

You will see JAVA_HOME near the beginning of the file you will need to change that to where java is installed on your system.

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HEAPSIZE=1000
export HADOOP_NAMENODE_OPTS=”-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,DRFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,RFAAUDIT} $HADOOP_NAMENODE_OPTS”
export HADOOP_SECONDARYNAMENODE_OPTS=$HADOOP_NAMENODE_OPTS
export HADOOP_CLIENT_OPTS=”-Xmx1024m $HADOOP_CLIENT_OPTS”

  1. mkdir –p /app/hadoop/tmp

This is the temp directory hadoop uses

  1. chown hduser:hduser /app/hadoop/tmp

core-site.xml

Click here to view the docs.

  1. nano /usr/local/hadoop/etc/hadoop/core-site.xml

This file contains configuration properties that Hadoop uses when starting up. By default it will look like . This will need to be changed.

  1. <configuration>
  2.       <property>
  3.             <name>fs.defaultFS</name>
  4.             <value>hdfs://NAMENODE:54310</value>
  5.             <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming
  6. the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description>
  7.       </property>
  8.       <property>
  9.             <name>hadoop.tmp.dir</name>
  10.             <value>/app/hadoop/tmp</value>
  11.       </property>
  12.       <property>
  13.             <name>hadoop.proxyuser.hduser.hosts</name>
  14.             <value>*</value>
  15.       </property>
  16.       <property>
  17.             <name>hadoop.proxyuser.hduser.groups</name>
  18.             <value>*</value>
  19.       </property>
  20. </configuration>

yarn-site.xml

Click here to view the docs.

  1. nano /usr/local/hadoop/etc/hadoop/yarn-site.xml
  1. <configuration>
  2.       <property>
  3.             <name>yarn.nodemanager.aux-services</name>
  4.             <value>mapreduce_shuffle</value>
  5.       </property>
  6.       <property>
  7.             <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
  8.       </property>
  9.       <property>
  10.             <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
  11.             <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  12.       </property>
  13.       <property>
  14.             <name>yarn.nodemanager.resource.memory-mb</name>
  15.             <value>12288</value>
  16.             <final>true</final>
  17.       </property>
  18.       <property>
  19.             <name>yarn.scheduler.minimum-allocation-mb</name>
  20.             <value>4096</value>
  21.             <final>true</final>
  22.       </property>
  23.       <property>
  24.             <name>yarn.scheduler.maximum-allocation-mb</name>
  25.             <value>12288</value>
  26.             <final>true</final>
  27.       </property>
  28.       <property>
  29.             <name>yarn.app.mapreduce.am.resource.mb</name>
  30.             <value>4096</value>
  31.       </property>
  32.       <property>
  33.             <name>yarn.app.mapreduce.am.command-opts</name>
  34.             <value>-Xmx3276m</value>
  35.       </property>
  36.       <property>
  37.             <name>yarn.nodemanager.local-dirs</name>
  38.             <value>/app/hadoop/tmp/nm-local-dir</value>
  39.       </property>
  40.       <!--LOG-->
  41.       <property>
  42.             <name>yarn.log-aggregation-enable</name>
  43.             <value>true</value>
  44.       </property>
  45.       <property>
  46.             <description>Where to aggregate logs to.</description>
  47.             <name>yarn.nodemanager.remote-app-log-dir</name>
  48.             <value>/tmp/yarn/logs</value>
  49.       </property>
  50.       <property>
  51.             <name>yarn.log-aggregation.retain-seconds</name>
  52.             <value>604800</value>
  53.       </property>
  54.       <property>
  55.             <name>yarn.log-aggregation.retain-check-interval-seconds</name>
  56.             <value>86400</value>
  57.       </property>
  58.       <property>
  59.             <name>yarn.log.server.url</name>
  60.             <value>http://NAMENODE:19888/jobhistory/logs/</value>
  61.       </property>
  62.       
  63.       <!--URLs-->
  64.       <property>
  65.             <name>yarn.resourcemanager.resource-tracker.address</name>
  66.             <value>NAMENODE:8025</value>
  67.       </property>
  68.       <property>
  69.             <name>yarn.resourcemanager.scheduler.address</name>
  70.             <value>NAMENODE:8030</value>
  71.       </property>
  72.       <property>
  73.             <name>yarn.resourcemanager.address</name>
  74.             <value>NAMENODE:8050</value>
  75.       </property>
  76.       <property>
  77.             <name>yarn.resourcemanager.admin.address</name>
  78.             <value>NAMENODE:8033</value>
  79.       </property>
  80.       <property>
  81.             <name>yarn.resourcemanager.webapp.address</name>
  82.             <value>NAMENODE:8088</value>
  83.       </property>
  84. </configuration>

By default it will look like . This will need to be changed.

mapred-site.xml

Click here to view the docs. By default, the /usr/local/hadoop/etc/hadoop/ folder contains /usr/local/hadoop/etc/hadoop/mapred-site.xml.template file which has to be renamed/copied with the name mapred-site.xml By default it will look like . This will need to be changed.

  1. cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
  2.  
  3. nano /usr/local/hadoop/etc/hadoop/mapred-site.xml
  1. <configuration>
  2.       <property>
  3.             <name>mapreduce.framework.name</name>
  4.             <value>yarn</value>
  5.       </property>
  6.       <property>
  7.             <name>mapreduce.jobhistory.address</name>
  8.             <value>NAMENODE:10020</value>
  9.       </property>
  10.       <property>
  11.             <name>mapreduce.jobhistory.webapp.address</name>
  12.             <value>NAMENODE:19888</value>
  13.       </property>
  14.       <property>
  15.             <name>mapreduce.jobtracker.address</name>
  16.             <value>NAMENODE:54311</value>
  17.       </property>
  18.       <!-- Memory and concurrency tuning -->
  19.       <property>
  20.             <name>mapreduce.map.memory.mb</name>
  21.             <value>4096</value>
  22.       </property>
  23.       <property>
  24.             <name>mapreduce.map.java.opts</name>
  25.             <value>-server -Xmx3276m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value>
  26.       </property>
  27.       <property>
  28.             <name>mapreduce.reduce.memory.mb</name>
  29.             <value>4096</value>
  30.       </property>
  31.       <property>
  32.             <name>mapreduce.reduce.java.opts</name>
  33.             <value>-server -Xmx3276m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value>
  34.       </property>
  35.       <property>
  36.             <name>mapreduce.reduce.shuffle.input.buffer.percent</name>
  37.             <value>0.5</value>
  38.       </property>
  39.       <property>
  40.             <name>mapreduce.task.io.sort.mb</name>
  41.             <value>600</value>
  42.       </property>
  43.       <property>
  44.             <name>mapreduce.task.io.sort.factor</name>
  45.             <value>1638</value>
  46.       </property>
  47.       <property>
  48.             <name>mapreduce.map.sort.spill.percent</name>
  49.             <value>0.50</value>
  50.       </property>
  51.       <property>
  52.             <name>mapreduce.map.speculative</name>
  53.             <value>false</value>
  54.       </property>
  55.       <property>
  56.             <name>mapreduce.reduce.speculative</name>
  57.             <value>false</value>
  58.       </property>
  59.       <property>
  60.             <name>mapreduce.task.timeout</name>
  61.             <value>1800000</value>
  62.       </property>
  63. </configuration>

yarn-env.sh

  1. nano /usr/local/hadoop/etc/hadoop/yarn-env.sh

Change or uncomment or add the following:

JAVA_HEAP_MAX=Xmx2000m
YARN_OPTS=”$YARN_OPTS -server -Dhadoop.log.dir=$YARN_LOG_DIR”
YARN_OPTS=”$YARN_OPTS -Djava.net.preferIPv4Stack=true”

Master

Add the namenode hostname.

  1. nano /usr/local/hadoop/etc/hadoop/masters

APPLY THE FOLLOWING TO THE NAMENODE ONLY

Slaves

Add namenode hostname and all datanodes hostname.

  1. nano /usr/local/hadoop/etc/hadoop/slaves

hdfs-site.xml

Click here to view the docs. By default it will look like . This will need to be changed. The /usr/local/hadoop/etc/hadoop/hdfs-site.xml file needs to be configured for each host in the cluster that is being used. Before editing this file, we need to create the namenode directory.

  1. mkdir -p /usr/local/hadoop_store/data/namenode
  2. chown -R hduser:hduser /usr/local/hadoop_store
  3. nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  1. <configuration>
  2.       <property>
  3.             <name>dfs.replication</name>
  4.             <value>3</value>
  5.             <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.</description>
  6.       </property>
  7.       <property>
  8.             <name>dfs.permissions</name>
  9.             <value>false</value>
  10.       </property>
  11.       <property>
  12.             <name>dfs.namenode.name.dir</name>
  13.             <value>file:/usr/local/hadoop_store/data/namenode</value>
  14.       </property>
  15.       <property>
  16.             <name>dfs.datanode.use.datanode.hostname</name>
  17.             <value>false</value>
  18.       </property>
  19.       <property>
  20.             <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
  21.             <value>false</value>
  22.       </property>
  23.       <property>
  24.             <name>dfs.namenode.http-address</name>
  25.             <value>NAMENODE:50070</value>
  26.             <description>Your NameNode hostname for http access.</description>
  27.       </property>
  28.       <property>
  29.             <name>dfs.namenode.secondary.http-address</name>
  30.             <value>SECONDARYNAMENODE:50090</value>
  31.             <description>Your Secondary NameNode hostname for http access.</description>
  32.       </property>
  33.       <property>
  34.             <name>dfs.blocksize</name>
  35.             <value>128m</value>
  36.       </property>
  37.       <property>
  38.             <name>dfs.namenode.http-bind-host</name>
  39.             <value>0.0.0.0</value>
  40.       </property>
  41.       <property>
  42.             <name>dfs.namenode.rpc-bind-host</name>
  43.             <value>0.0.0.0</value>
  44.       </property>
  45.       <property>
  46.              <name>dfs.namenode.servicerpc-bind-host</name>
  47.              <value>0.0.0.0</value>
  48.       </property>
  49. </configuration>

APPLY THE FOLLOWING TO THE DATANODE(s) ONLY

Slaves

Add only that datanodes hostname.

  1. nano /usr/local/hadoop/etc/hadoop/slaves

hdfs-site.xml

The /usr/local/hadoop/etc/hadoop/hdfs-site.xml file needs to be configured for each host in the cluster that is being used. Before editing this file, we need to create the datanode directory.
By default it will look like . This will need to be changed.

  1. mkdir -p /usr/local/hadoop_store/data/datanode
  2. chown -R hduser:hduser /usr/local/hadoop_store
  3. nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  1. <configuration>
  2.       <property>
  3.             <name>dfs.replication</name>
  4.             <value>3</value>
  5.             <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.</description>
  6.       </property>
  7.       <property>
  8.             <name>dfs.permissions</name>
  9.             <value>false</value>
  10.       </property>
  11.       <property>
  12.             <name>dfs.datanode.data.dir</name>
  13.             <value>file:/usr/local/hadoop_store/data/datanode</value>
  14.       </property>
  15.       <property>
  16.             <name>dfs.datanode.use.datanode.hostname</name>
  17.             <value>false</value>
  18.       </property>
  19.       <property>
  20.             <name>dfs.namenode.http-address</name>
  21.             <value>NAMENODE:50070</value>
  22.             <description>Your NameNode hostname for http access.</description>
  23.       </property>
  24.       <property>
  25.             <name>dfs.namenode.secondary.http-address</name>
  26.             <value>SECONDARYNAMENODE:50090</value>
  27.             <description>Your Secondary NameNode hostname for http access.</description>
  28.       </property>
  29.       <property>
  30.             <name>dfs.datanode.http.address</name>
  31.             <value>DATANODE:50075</value>
  32.       </property>
  33.       <property>
  34.             <name>dfs.blocksize</name>
  35.             <value>128m</value>
  36.       </property>
  37. </configuration>

You need to allow the pass-through for all ports necessary. If you have the Ubuntu firewall on.

  1. sudo ufw allow 50070
  2. sudo ufw allow 8088

Format Cluster:
Only do this if NO data is present. All data will be destroyed when the following is done.
This is to be done on NAMENODE ONLY!

  1. hdfs namenode –format

Start The Cluster:
You can now start the cluster.
You do this from the NAMENODE ONLY.

  1. start-dfs.sh
  2. start-yarn.sh
  3. mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

If the above three commands didn’t work something went wrong. As it should have found the scripts located /usr/local/hadoop/sbin/ directory.

Cron Job:
You should probably setup a cron job to start the cluster when you reboot.

  1. crontab –e

@reboot /usr/local/hadoop/sbin/start-dfs.sh > /home/hduser/dfs-start.log 2>&1
@reboot /usr/local/hadoop/sbin/start-yarn.sh > /home/hduser/yarn-start.log 2>&1
@reboot /usr/local/hadoop/sbin/mr-jobhistory-daemon.sh –config $HADOOP_CONF_DIR stop historyserver > /home/hduser/history-stop.log 2>&1

Verification:
To check that everything is working as it should run “jps” on the NAMENODE. It should return something like the following where the pid will be different:

  1. jps

You could also run “netstat -plten | grep java” or “lsof –i :50070” and “lsof –i :8088”.

Picked up _JAVA_OPTIONS: -Xms3g -Xmx10g -Djava.net.preferIPv4Stack=true
2596 SecondaryNameNode
3693 Jps
1293 JobHistoryServer
1317 ResourceManager
1840 NameNode
1743 NodeManager
2351 DataNode

You can check the DATA NODES by ssh into each one and running “jps”. It should return something like the following where the pid will be different:

Picked up _JAVA_OPTIONS: -Xms3g -Xmx10g -Djava.net.preferIPv4Stack=true
3218 Jps
2215 NodeManager
2411 DataNode

If for any reason only of the services is not running you need to review the logs. They can be found at /usr/local/hadoop/logs/. If it’s ResourceManager that isn’t running then look at file that has “yarn” and “resourcemanager” in it.

WARNING:
Never reboot the system without first stopping the cluster. When the cluster shuts down it is safe to reboot it. Also if you configured a cronjob @reboot you should make sure the DATANODES are up and running first before starting the NAMENODE that way it automatically starts the DATANODES for you

Web Ports:

NameNode

  • 50070: HDFS Namenode
  • 50075: HDFS Datanode
  • 50090: HDFS Secondary Namenode
  • 8088: Resource Manager
  • 19888: Job History

DataNode

  • 50075: HDFS Datanode

NetStat

To check that all the Hadoop ports are available on which IP run the following.

  1. sudo netstat -ltnp

Port Check

If for some reason you are having issues connecting to a Hadoop port then run the following command as you try and connect via the port.

  1. sudo tcpdump -n -tttt -i eth1 port 50070

References

I used a lot of different resources and reference material on this. However I did not save all the relevant links I used. Below are just a few I used. There was various blog posts about memory utilization, etc.