Category: Kerberos

Spark Installation on Hadoop

In this tutorial I will show you how to use Kerberos/SSL with Spark integrated with Yarn. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Hadoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

cd /etc/security/keytabs/

sudo kadmin.local

#You can list princepals
listprincs

#Create the following principals
addprinc -randkey spark/hadoop@REALM.CA

#Create the keytab files.
#You will need these for Hadoop to be able to login
xst -k spark.service.keytab spark/hadoop@REALM.CA

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Download

Go to Apache Spark Download and get the link for Spark.

wget http://apache.forsale.plus/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
tar -xvf spark-2.4.4-bin-hadoop2.7.tgz
mv spark-2.4.4-bin-hadoop2.7 /usr/local/spark/

Update .bashrc

sudo nano ~/.bashrc

#Ensure we have the following in the Hadoop section
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

#Add the following

#SPARK VARIABLES START
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
#SPARK VARIABLES STOP

source ~/.bashrc

Setup Configuration

cd /usr/local/spark/conf
mv spark-defaults.conf.template spark-defaults.conf
nano spark-defaults.conf

#Add to the end
spark.master                            yarn
spark.yarn.historyServer.address        ${hadoopconf-yarn.resourcemanager.hostname}:18080
spark.yarn.keytab                       /etc/security/keytabs/spark.service.keytab
spark.yarn.principal                    spark/hadoop@REALM.CA
spark.yarn.access.hadoopFileSystems     hdfs://NAMENODE:54310
spark.authenticate                      true
spark.driver.bindAddress                0.0.0.0
spark.authenticate.enableSaslEncryption true
spark.eventLog.enabled                  true
spark.eventLog.dir                      hdfs://NAMENODE:54310/user/spark/applicationHistory
spark.history.fs.logDirectory           hdfs://NAMENODE:54310/user/spark/applicationHistory
spark.history.fs.update.interval        10s
spark.history.ui.port                   18080

#SSL
spark.ssl.enabled                       true
spark.ssl.keyPassword                   PASSWORD
spark.ssl.keyStore                      /etc/security/serverKeys/keystore.jks
spark.ssl.keyStorePassword              PASSWORD
spark.ssl.keyStoreType                  JKS
spark.ssl.trustStore                    /etc/security/serverKeys/truststore.jks
spark.ssl.trustStorePassword            PASSWORD
spark.ssl.trustStoreType                JKS

Kinit

kinit -kt /etc/security/keytabs/spark.service.keytab spark/hadoop@REALM.CA
klist
hdfs dfs -mkdir /user/spark/
hdfs dfs -mkdir /user/spark/applicationHistory
hdfs dfs -ls /user/spark

Start The Service

$SPARK_HOME/sbin/start-history-server.sh

Stop The Service

$SPARK_HOME/sbin/stop-history-server.sh

Spark History Server Web UI

References

I used a lot of different resources and reference material on this. Below are just a few I used.

https://spark.apache.org/docs/latest/running-on-yarn.html#configuration

https://spark.apache.org/docs/latest/security.html

https://www.linode.com/docs/databases/hadoop/install-configure-run-spark-on-top-of-hadoop-yarn-cluster/

 

 

 

 

Sqoop2: Kerberize Installation

In this tutorial I will show you how to kerberize Sqoop installation. Before you begin ensure you have installed Sqoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

cd /etc/security/keytabs
sudo kadmin.local
addprinc -randkey sqoop/hadoop@REALM.CA
xst -kt sqoop.service.keytab sqoop/hadoop@REALM.CA
addprinc -randkey sqoopHTTP/hadoop@REALM.CA
xst -kt sqoopHTTP.service.keytab sqoopHTTP/hadoop@REALM.CA
q

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Configuration

Configure Kerberos with Sqoop

cd /usr/local/sqoop/conf/
nano sqoop.properties

#uncomment the following
org.apache.sqoop.security.authentication.type=KERBEROS
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.KerberosAuthenticationHandler

#update to the following
org.apache.sqoop.security.authentication.kerberos.principal=sqoop/hadoop@GAUDREAULT_KDC.CA
org.apache.sqoop.security.authentication.kerberos.keytab=/etc/security/keytabs/sqoop.service.keytab

 

 

 

 

 

 

 

 

 

 

Kerberos: Commands

In this tutorial I will give you a few useful commands when using Kerberos. If you haven’t installed Kerberos yet go here. I will keep this updated as time goes on. Also note that the commands below have a variety of options. Please go check.

Admin

This will open Kerberos V5 administration system.

kadmin.local
Add Principal

This will add a new principal. -randkey is optional. When specified the encrypted key will be chosen at random instead of derived from a password. Be sure to change USER to whatever your user is.

addprinc -randkey USER/_HOST@REALM.CA
Create KeyTab

This will create a keytab in the directory where you generated it. You should put it in /etc/security/keytabs/ folder. You can also specify the full path (IE: /etc/security/keytabs/USER.keytab). Be sure to change USER to whatever your user is.

xst -k USER.keytab USER/_HOST@REALM.CA
Kinit

When using the -kt uses the keytab to grant a ticket

kinit -kt /etc/security/keytabs/USER.keytab USER/_HOST@REALM.CA
Klist

If you want to see what tickets have been granted. You can issue the below command.

klist
Inline Commands

You can do inline Kerberos commands without first opening kadmin.local. To do so you must specify the “-q” option then in quotes the command to issue. See below.

kadmin.local -q "addprinc -randkey USER/_HOST@REALM.CA"

 

 

 

 

 

 

 

 

Hadoop & Java: Connect to Remote Kerberos HDFS using KeyTab

In this tutorial I will show you how to connect to remote Kerberos HDFS cluster using Java.  If you haven’t install hdfs with kerberos yet follow the tutorial.

Import SSL Cert to Java:

Follow this tutorial to “Installing unlimited strength encryption Java libraries”

If on Windows do the following

#Import it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -import -file hadoop.csr -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts" -alias "hadoop"

#Check it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -list -v -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

#If you want to delete it
"C:\Program Files\Java\jdk1.8.0_171\bin\keytool" -delete -alias hadoop -keystore "C:\Program Files\Java\jdk1.8.0_171\jre\lib\security\cacerts"

POM.xml:

<dependency>
	<groupId>org.apache.hadoop</groupId>
	<artifactId>hadoop-client</artifactId>
	<version>2.9.1</version>
</dependency>

Imports:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.UserGroupInformation;

Connect:

// Setup the configuration object.
final Configuration config = new Configuration();

config.set("fs.defaultFS", "swebhdfs://hadoop:50470");
config.set("hadoop.security.authentication", "kerberos");
config.set("hadoop.rpc.protection", "integrity");

System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
System.setProperty("java.security.krb5.conf", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\krb5.conf");
System.setProperty("java.security.krb5.realm", "REALM.CA");
System.setProperty("java.security.krb5.kdc", "REALM.CA");
System.setProperty("sun.security.krb5.debug", "true");
System.setProperty("javax.net.debug", "all");
System.setProperty("javax.net.ssl.keyStorePassword","YOURPASSWORD");
System.setProperty("javax.net.ssl.keyStore","C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_171\\jre\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword","YOURPASSWORD");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");

UserGroupInformation.setConfiguration(config);
UserGroupInformation.setLoginUser(UserGroupInformation.loginUserFromKeytabAndReturnUGI("myuser/hadoop@REALM.CA", "c:\\data\\myuser.keytab"));

System.out.println(UserGroupInformation.getLoginUser());
System.out.println(UserGroupInformation.getCurrentUser());

HDFS/Yarn/MapRed: Kerberize/SSL

In this tutorial I will show you how to use Kerberos/SSL with HDFS/Yarn/MapRed. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Hadoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

cd /etc/security/keytabs/

sudo kadmin.local

#You can list princepals
listprincs

#Create the following principals
addprinc -randkey nn/hadoop@REALM.CA
addprinc -randkey jn/hadoop@REALM.CA
addprinc -randkey dn/hadoop@REALM.CA
addprinc -randkey sn/hadoop@REALM.CA
addprinc -randkey nm/hadoop@REALM.CA
addprinc -randkey rm/hadoop@REALM.CA
addprinc -randkey jhs/hadoop@REALM.CA
addprinc -randkey HTTP/hadoop@REALM.CA

#We are going to create a user to access with later
addprinc -pw hadoop myuser/hadoop@REALM.CA
xst -k myuser.keytab myuser/hadoop@REALM.CA

#Create the keytab files.
#You will need these for Hadoop to be able to login
xst -k nn.service.keytab nn/hadoop@REALM.CA
xst -k jn.service.keytab jn/hadoop@REALM.CA
xst -k dn.service.keytab dn/hadoop@REALM.CA
xst -k sn.service.keytab sn/hadoop@REALM.CA
xst -k nm.service.keytab nm/hadoop@REALM.CA
xst -k rm.service.keytab rm/hadoop@REALM.CA
xst -k jhs.service.keytab jhs/hadoop@REALM.CA
xst -k spnego.service.keytab HTTP/hadoop@REALM.CA

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Stop the Cluster

stop-dfs.sh
stop-yarn.sh
mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

Hosts Update

sudo nano /etc/hosts

#Remove 127.0.1.1 line

#Change 127.0.0.1 to the following
#Notice how realm.ca is there its because we need to tell where that host resides
127.0.0.1 realm.ca hadoop localhost

hadoop-env.sh

We don’t set the HADOOP_SECURE_DN_USER because we are going to use Kerberos

sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh

#Locate "export ${HADOOP_SECURE_DN_USER}=${HADOOP_SECURE_DN_USER}"
#and change to

export HADOOP_SECURE_DN_USER=

core-site.xml

nano /usr/local/hadoop/etc/hadoop/core-site.xml

<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://NAMENODE:54310</value>
		<description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming
		the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/app/hadoop/tmp</value>
	</property>
	<property>
		<name>hadoop.proxyuser.hadoopuser.hosts</name>
		<value>*</value>
	</property>
	<property>
		<name>hadoop.proxyuser.hadoopuser.groups</name>
		<value>*</value>
	</property>
	<property>
		<name>hadoop.security.authentication</name>
		<value>kerberos</value> <!-- A value of "simple" would disable security. -->
	</property>
	<property>
		<name>hadoop.security.authorization</name>
		<value>true</value>
	</property>
	<property>
		<name>hadoop.security.auth_to_local</name>
		<value>
		RULE:[2:$1@$0](nn/.*@.*REALM.TLD)s/.*/hdfs/
		RULE:[2:$1@$0](jn/.*@.*REALM.TLD)s/.*/hdfs/
		RULE:[2:$1@$0](dn/.*@.*REALM.TLD)s/.*/hdfs/
		RULE:[2:$1@$0](sn/.*@.*REALM.TLD)s/.*/hdfs/
		RULE:[2:$1@$0](nm/.*@.*REALM.TLD)s/.*/yarn/
		RULE:[2:$1@$0](rm/.*@.*REALM.TLD)s/.*/yarn/
		RULE:[2:$1@$0](jhs/.*@.*REALM.TLD)s/.*/mapred/
		DEFAULT
		</value>
	</property>
	<property>
		<name>hadoop.rpc.protection</name>
		<value>integrity</value>
	</property>
	<property>
		<name>hadoop.ssl.require.client.cert</name>
		<value>false</value>
	</property>
	<property>
		<name>hadoop.ssl.hostname.verifier</name>
		<value>DEFAULT</value>
	</property>
	<property>
		<name>hadoop.ssl.keystores.factory.class</name>
		<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
	</property>
	<property>
		<name>hadoop.ssl.server.conf</name>
		<value>ssl-server.xml</value>
	</property>
	<property>
		<name>hadoop.ssl.client.conf</name>
		<value>ssl-client.xml</value>
	</property>
	<property>
		<name>hadoop.rpc.protection</name>
		<value>integrity</value>
	</property>
</configuration>

ssl-server.xml

Change ssl-server.xml.example to ssl-server.xml

cp /usr/local/hadoop/etc/hadoop/ssl-server.xml.example /usr/local/hadoop/etc/hadoop/ssl-server.xml

nano /usr/local/hadoop/etc/hadoop/ssl-server.xml

Update properties

<configuration>
	<property>
		<name>ssl.server.truststore.location</name>
		<value>/etc/security/serverKeys/truststore.jks</value>
		<description>Truststore to be used by NN and DN. Must be specified.</description>
	</property>
	<property>
		<name>ssl.server.truststore.password</name>
		<value>PASSWORD</value>
		<description>Optional. Default value is "".</description>
	</property>
	<property>
		<name>ssl.server.truststore.type</name>
		<value>jks</value>
		<description>Optional. The keystore file format, default value is "jks".</description>
	</property>
	<property>
		<name>ssl.server.truststore.reload.interval</name>
		<value>10000</value>
		<description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds).</description>
	</property>
	<property>
		<name>ssl.server.keystore.location</name>
		<value>/etc/security/serverKeys/keystore.jks</value>
		<description>Keystore to be used by NN and DN. Must be specified.</description>
	</property>
	<property>
		<name>ssl.server.keystore.password</name>
		<value>PASSWORD</value>
		<description>Must be specified.</description>
	</property>
	<property>
		<name>ssl.server.keystore.keypassword</name>
		<value>PASSWORD</value>
		<description>Must be specified.</description>
	</property>
	<property>
		<name>ssl.server.keystore.type</name>
		<value>jks</value>
		<description>Optional. The keystore file format, default value is "jks".</description>
	</property>
	<property>
		<name>ssl.server.exclude.cipher.list</name>
		<value>TLS_ECDHE_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,
		SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,
		SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,
		SSL_RSA_WITH_RC4_128_MD5</value>
		<description>Optional. The weak security cipher suites that you want excluded from SSL communication.</description>
	</property>
</configuration>

ssl-client.xml

Change ssl-client.xml.example to ssl-client.xml

cp /usr/local/hadoop/etc/hadoop/ssl-client.xml.example /usr/local/hadoop/etc/hadoop/ssl-client.xml

nano /usr/local/hadoop/etc/hadoop/ssl-client.xml

Update properties

<configuration>
	<property>
		<name>ssl.client.truststore.location</name>
		<value>/etc/security/serverKeys/truststore.jks</value>
		<description>Truststore to be used by clients like distcp. Must be specified.</description>
	</property>
	<property>
		<name>ssl.client.truststore.password</name>
		<value>PASSWORD</value>
		<description>Optional. Default value is "".</description>
	</property>
	<property>
		<name>ssl.client.truststore.type</name>
		<value>jks</value>
		<description>Optional. The keystore file format, default value is "jks".</description>
	</property>
	<property>
		<name>ssl.client.truststore.reload.interval</name>
		<value>10000</value>
		<description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds).</description>
	</property>
	<property>
		<name>ssl.client.keystore.location</name>
		<value></value>
		<description>Keystore to be used by clients like distcp. Must be specified.</description>
	</property>
	<property>
		<name>ssl.client.keystore.password</name>
		<value></value>
		<description>Optional. Default value is "".</description>
	</property>
	<property>
		<name>ssl.client.keystore.keypassword</name>
		<value></value>
		<description>Optional. Default value is "".</description>
	</property>
	<property>
		<name>ssl.client.keystore.type</name>
		<value>jks</value>
		<description>Optional. The keystore file format, default value is "jks".</description>
	</property>
</configuration>

mapred-site.xml

Just add the following to the config to let it know the Kerberos keytabs to use.

nano /usr/local/hadoop/etc/hadoop/mapred-site.xml

<property>
	<name>mapreduce.jobhistory.keytab</name>
	<value>/etc/security/keytabs/jhs.service.keytab</value>
</property>
<property>
	<name>mapreduce.jobhistory.principal</name>
	<value>jhs/_HOST@REALM.CA</value>
</property>
<property>
	<name>mapreduce.jobhistory.http.policy</name>
	<value>HTTPS_ONLY</value>
</property>

hdfs-site.xml

Add the following properties

nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<property>
	<name>dfs.http.policy</name>
	<value>HTTPS_ONLY</value>
</property>
<property>
	<name>hadoop.ssl.enabled</name>
	<value>true</value>
</property>
<property>
	<name>dfs.datanode.https.address</name>
	<value>NAMENODE:50475</value>
</property>
<property>
	<name>dfs.namenode.https-address</name>
	<value>NAMENODE:50470</value>
	<description>Your NameNode hostname for http access.</description>
</property>
<property>
	<name>dfs.namenode.secondary.https-address</name>
	<value>NAMENODE:50091</value>
	<description>Your Secondary NameNode hostname for http access.</description>
</property>
<property>
	<name>dfs.namenode.https-bind-host</name>
	<value>0.0.0.0</value>
</property>
<property>
	<name>dfs.block.access.token.enable</name>
	<value>true</value>
	<description> If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanod</description>
</property>
<property>
	<name>dfs.namenode.kerberos.principal</name>
	<value>nn/_HOST@REALM.CA</value>
	<description> Kerberos principal name for the NameNode</description>
</property>
<property>
	<name>dfs.secondary.namenode.kerberos.principal</name>
	<value>sn/_HOST@REALM.CA</value>
	<description>Kerberos principal name for the secondary NameNode.</description>
</property>
<property>
	<name>dfs.web.authentication.kerberos.keytab</name>
	<value>/etc/security/keytabs/spnego.service.keytab</value>
	<description>The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</description>
</property>
<property>
	<name>dfs.namenode.keytab.file</name>
	<value>/etc/security/keytabs/nn.service.keytab</value>
	<description>Combined keytab file containing the namenode service and host principals.</description>
</property>
<property>
	<name>dfs.datanode.keytab.file</name>
	<value>/etc/security/keytabs/dn.service.keytab</value>
	<description>The filename of the keytab file for the DataNode.</description>
</property>
<property>
	<name>dfs.datanode.kerberos.principal</name>
	<value>dn/_HOST@REALM.CA</value>
	<description>The Kerberos principal that the DataNode runs as. "_HOST" is replaced by the real host name.</description>
</property>
<property>
	<name>dfs.namenode.kerberos.internal.spnego.principal</name>
	<value>${dfs.web.authentication.kerberos.principal}</value>
</property>
<property>
	<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
	<value>>${dfs.web.authentication.kerberos.principal}</value>
</property>
<property>
	<name>dfs.web.authentication.kerberos.principal</name>
	<value>HTTP/_HOST@REALM.CA</value>
	<description>The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</description>          
</property>
<property>
	<name>dfs.data.transfer.protection</name>
	<value>integrity</value>
</property>
<property>
	<name>dfs.datanode.address</name>
	<value>NAMENODE:50010</value>
</property>
<property>
	<name>dfs.secondary.namenode.keytab.file</name>
	<value>/etc/security/keytabs/sn.service.keytab</value>
</property>
<property>
	<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
	<value>HTTP/_HOST@REALM.CA</value>
</property>
<property>
	<name>dfs.webhdfs.enabled</name>
	<value>true</value>
</property>

Remove the following properties

dfs.namenode.http-address
dfs.namenode.secondary.http-address
dfs.namenode.http-bind-host

yarn-site.xml

Add the following properties

nano /usr/local/hadoop/etc/hadoop/yarn-site.xml

<property>
	<name>yarn.http.policy</name>
	<value>HTTPS_ONLY</value>
</property>
<property>
	<name>yarn.resourcemanager.webapp.https.address</name>
	<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
	<name>yarn.resourcemanager.hostname</name>
	<value>NAMENODE</value>
</property>
<property>
	<name>yarn.nodemanager.bind-host</name>
	<value>0.0.0.0</value>
</property>
<property>
	<name>yarn.nodemanager.webapp.address</name>
	<value>${yarn.nodemanager.hostname}:8042</value>
</property>
<property>
	<name>yarn.resourcemanager.principal</name>
	<value>rm/_HOST@REALM.CA</value>
</property>
<property>
	<name>yarn.resourcemanager.keytab</name>
	<value>/etc/security/keytabs/rm.service.keytab</value>
</property>
<property>
	<name>yarn.nodemanager.principal</name>
	<value>nm/_HOST@REALM.CA</value>
</property>
<property>
	<name>yarn.nodemanager.keytab</name>
	<value>/etc/security/keytabs/nm.service.keytab</value>
</property>
<property>
	<name>yarn.nodemanager.hostname</name>
	<value>NAMENODE</value>
</property>
<property>
	<name>yarn.resourcemanager.bind-host</name>
	<value>0.0.0.0</value>
</property>
<property>
	<name>yarn.timeline-service.bind-host</name>
	<value>0.0.0.0</value>
</property>

Remove the following properties

yarn.resourcemanager.webapp.address

SSL

Setup SSL Directories

sudo mkdir -p /etc/security/serverKeys
sudo chown -R root:hadoopuser /etc/security/serverKeys/
sudo chmod 755 /etc/security/serverKeys/

cd /etc/security/serverKeys

Setup Keystore

sudo keytool -genkey -alias NAMENODE -keyalg RSA -keysize 1024 -dname "CN=NAMENODE,OU=ORGANIZATION_UNIT,C=canada" -keypass PASSWORD -keystore /etc/security/serverKeys/keystore.jks -storepass PASSWORD
sudo keytool -export -alias NAMENODE -keystore /etc/security/serverKeys/keystore.jks -rfc -file /etc/security/serverKeys/NAMENODE.csr -storepass PASSWORD

Setup Truststore

sudo keytool -import -noprompt -alias NAMENODE -file /etc/security/serverKeys/NAMENODE.csr -keystore /etc/security/serverKeys/truststore.jks -storepass PASSWORD

Generate Self Signed Certifcate

sudo openssl genrsa -out /etc/security/serverKeys/NAMENODE.key 2048

sudo openssl req -x509 -new -key /etc/security/serverKeys/NAMENODE.key -days 300 -out /etc/security/serverKeys/NAMENODE.pem

sudo keytool -keystore /etc/security/serverKeys/keystore.jks -alias NAMENODE -certreq -file /etc/security/serverKeys/NAMENODE.cert -storepass PASSWORD -keypass PASSWORD

sudo openssl x509 -req -CA /etc/security/serverKeys/NAMENODE.pem -CAkey /etc/security/serverKeys/NAMENODE.key -in /etc/security/serverKeys/NAMENODE.cert -out /etc/security/serverKeys/NAMENODE.signed -days 300 -CAcreateserial

Setup File Permissions

sudo chmod 440 /etc/security/serverKeys/*
sudo chown root:hadoopuser /etc/security/serverKeys/*

Start the Cluster

start-dfs.sh
start-yarn.sh
mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

Create User Directory

kinit -kt /etc/security/keytabs/myuser.keytab myuser/hadoop@REALM.CA
#ensure the login worked
klist

#Create hdfs directory now
hdfs dfs -mkdir /user
hdfs dfs -mkdir /user/myuser

#remove kerberos ticket
kdestroy

URL

https://NAMENODE:50470
https://NAMENODE:50475
https://NAMENODE:8090

References

https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/admin_ssl_hbase_mr_yarn_hdfs_web.html

Kerberos Server Installation

In this tutorial I will show you how to install Kerberos server on Ubuntu 16.04.

sudo apt install krb5-kdc krb5-admin-server krb5-config -y

Enter your realm. I will use REALM.CA

Enter your servers. I will use localhost

Enter your administrative server. I will use localhost

Now you can click Ok and installation will continue.

Next we can create our new realm

sudo krb5_newrealm

Enter your password then confirm it.

Now we can edit our kadm5.acl to have admin. Uncomment “*/admin *”

sudo nano /etc/krb5kdc/kadm5.acl

Now we make our keytabs directory and grant the necessary permissions.

sudo mkdir -p /etc/security/keytabs/
sudo chown root:hduser /etc/security/keytabs
sudo chmod 750 /etc/security/keytabs

Now we edit our krb5.conf file

sudo nano /etc/krb5.conf

Ensure it looks like the below

[libdefaults]
        default_realm = REALM.CA


[realms]
        REALM.CA = {
                kdc = localhost
                admin_server = localhost
        }


[domain_realm]
        .realm.ca = REALM.CA
        realm.ca = REALM.CA

Now we can restart the kerberos services

sudo service krb5-kdc restart; service krb5-admin-server restart

Once you create a principal if when you attempt to use kadmin you get the error “GSS-API (or Kerberos) error while initializing kadmin interface”. Then do the following.

sudo RUNLEVEL=1 apt-get install rng-tools
cat /dev/random | rngtest -c 1000
sudo apt-get install haveged
cat /proc/sys/kernel/random/entropy_avail
cat /dev/random | rngtest -c 1000
haveged -n 2g -f - | dd of=/dev/null

Uninstallation

sudo apt remove --purge krb5-kdc krb5-admin-server krb5-config -y
sudo rm -rf /var/lib/krb5kdc

References
I used the following references as a guide.

http://blog.ruanbekker.com/blog/2017/10/18/setup-kerberos-server-and-client-on-ubuntu/ 
http://csetutorials.com/setup-kerberos-ubuntu.html