HDFS/Yarn/MapRed: Kerberize/SSL

(Last Updated On: )

In this tutorial I will show you how to use Kerberos/SSL with HDFS/Yarn/MapRed. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Hadoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

  1. cd /etc/security/keytabs/
  2.  
  3. sudo kadmin.local
  4.  
  5. #You can list princepals
  6. listprincs
  7.  
  8. #Create the following principals
  9. addprinc -randkey nn/hadoop@REALM.CA
  10. addprinc -randkey jn/hadoop@REALM.CA
  11. addprinc -randkey dn/hadoop@REALM.CA
  12. addprinc -randkey sn/hadoop@REALM.CA
  13. addprinc -randkey nm/hadoop@REALM.CA
  14. addprinc -randkey rm/hadoop@REALM.CA
  15. addprinc -randkey jhs/hadoop@REALM.CA
  16. addprinc -randkey HTTP/hadoop@REALM.CA
  17.  
  18. #We are going to create a user to access with later
  19. addprinc -pw hadoop myuser/hadoop@REALM.CA
  20. xst -k myuser.keytab myuser/hadoop@REALM.CA
  21.  
  22. #Create the keytab files.
  23. #You will need these for Hadoop to be able to login
  24. xst -k nn.service.keytab nn/hadoop@REALM.CA
  25. xst -k jn.service.keytab jn/hadoop@REALM.CA
  26. xst -k dn.service.keytab dn/hadoop@REALM.CA
  27. xst -k sn.service.keytab sn/hadoop@REALM.CA
  28. xst -k nm.service.keytab nm/hadoop@REALM.CA
  29. xst -k rm.service.keytab rm/hadoop@REALM.CA
  30. xst -k jhs.service.keytab jhs/hadoop@REALM.CA
  31. xst -k spnego.service.keytab HTTP/hadoop@REALM.CA

Set Keytab Permissions/Ownership

  1. sudo chown root:hadoopuser /etc/security/keytabs/*
  2. sudo chmod 750 /etc/security/keytabs/*

Stop the Cluster

  1. stop-dfs.sh
  2. stop-yarn.sh
  3. mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

Hosts Update

  1. sudo nano /etc/hosts
  2.  
  3. #Remove 127.0.1.1 line
  4.  
  5. #Change 127.0.0.1 to the following
  6. #Notice how realm.ca is there its because we need to tell where that host resides
  7. 127.0.0.1 realm.ca hadoop localhost

hadoop-env.sh

We don’t set the HADOOP_SECURE_DN_USER because we are going to use Kerberos

  1. sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh
  2.  
  3. #Locate "export ${HADOOP_SECURE_DN_USER}=${HADOOP_SECURE_DN_USER}"
  4. #and change to
  5.  
  6. export HADOOP_SECURE_DN_USER=

core-site.xml

  1. nano /usr/local/hadoop/etc/hadoop/core-site.xml
  2.  
  3. <configuration>
  4. 	<property>
  5. 		<name>fs.defaultFS</name>
  6. 		<value>hdfs://NAMENODE:54310</value>
  7. 		<description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming
  8. 		the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description>
  9. 	</property>
  10. 	<property>
  11. 		<name>hadoop.tmp.dir</name>
  12. 		<value>/app/hadoop/tmp</value>
  13. 	</property>
  14. 	<property>
  15. 		<name>hadoop.proxyuser.hadoopuser.hosts</name>
  16. 		<value>*</value>
  17. 	</property>
  18. 	<property>
  19. 		<name>hadoop.proxyuser.hadoopuser.groups</name>
  20. 		<value>*</value>
  21. 	</property>
  22. 	<property>
  23. 		<name>hadoop.security.authentication</name>
  24. 		<value>kerberos</value> <!-- A value of "simple" would disable security. -->
  25. 	</property>
  26. 	<property>
  27. 		<name>hadoop.security.authorization</name>
  28. 		<value>true</value>
  29. 	</property>
  30. 	<property>
  31. 		<name>hadoop.security.auth_to_local</name>
  32. 		<value>
  33. 		RULE:[2:$1@$0](nn/.*@.*REALM.TLD)s/.*/hdfs/
  34. 		RULE:[2:$1@$0](jn/.*@.*REALM.TLD)s/.*/hdfs/
  35. 		RULE:[2:$1@$0](dn/.*@.*REALM.TLD)s/.*/hdfs/
  36. 		RULE:[2:$1@$0](sn/.*@.*REALM.TLD)s/.*/hdfs/
  37. 		RULE:[2:$1@$0](nm/.*@.*REALM.TLD)s/.*/yarn/
  38. 		RULE:[2:$1@$0](rm/.*@.*REALM.TLD)s/.*/yarn/
  39. 		RULE:[2:$1@$0](jhs/.*@.*REALM.TLD)s/.*/mapred/
  40. 		DEFAULT
  41. 		</value>
  42. 	</property>
  43. 	<property>
  44. 		<name>hadoop.rpc.protection</name>
  45. 		<value>integrity</value>
  46. 	</property>
  47. 	<property>
  48. 		<name>hadoop.ssl.require.client.cert</name>
  49. 		<value>false</value>
  50. 	</property>
  51. 	<property>
  52. 		<name>hadoop.ssl.hostname.verifier</name>
  53. 		<value>DEFAULT</value>
  54. 	</property>
  55. 	<property>
  56. 		<name>hadoop.ssl.keystores.factory.class</name>
  57. 		<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
  58. 	</property>
  59. 	<property>
  60. 		<name>hadoop.ssl.server.conf</name>
  61. 		<value>ssl-server.xml</value>
  62. 	</property>
  63. 	<property>
  64. 		<name>hadoop.ssl.client.conf</name>
  65. 		<value>ssl-client.xml</value>
  66. 	</property>
  67. 	<property>
  68. 		<name>hadoop.rpc.protection</name>
  69. 		<value>integrity</value>
  70. 	</property>
  71. </configuration>

ssl-server.xml

Change ssl-server.xml.example to ssl-server.xml

  1. cp /usr/local/hadoop/etc/hadoop/ssl-server.xml.example /usr/local/hadoop/etc/hadoop/ssl-server.xml
  2.  
  3. nano /usr/local/hadoop/etc/hadoop/ssl-server.xml

Update properties

  1. <configuration>
  2. 	<property>
  3. 		<name>ssl.server.truststore.location</name>
  4. 		<value>/etc/security/serverKeys/truststore.jks</value>
  5. 		<description>Truststore to be used by NN and DN. Must be specified.</description>
  6. 	</property>
  7. 	<property>
  8. 		<name>ssl.server.truststore.password</name>
  9. 		<value>PASSWORD</value>
  10. 		<description>Optional. Default value is "".</description>
  11. 	</property>
  12. 	<property>
  13. 		<name>ssl.server.truststore.type</name>
  14. 		<value>jks</value>
  15. 		<description>Optional. The keystore file format, default value is "jks".</description>
  16. 	</property>
  17. 	<property>
  18. 		<name>ssl.server.truststore.reload.interval</name>
  19. 		<value>10000</value>
  20. 		<description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds).</description>
  21. 	</property>
  22. 	<property>
  23. 		<name>ssl.server.keystore.location</name>
  24. 		<value>/etc/security/serverKeys/keystore.jks</value>
  25. 		<description>Keystore to be used by NN and DN. Must be specified.</description>
  26. 	</property>
  27. 	<property>
  28. 		<name>ssl.server.keystore.password</name>
  29. 		<value>PASSWORD</value>
  30. 		<description>Must be specified.</description>
  31. 	</property>
  32. 	<property>
  33. 		<name>ssl.server.keystore.keypassword</name>
  34. 		<value>PASSWORD</value>
  35. 		<description>Must be specified.</description>
  36. 	</property>
  37. 	<property>
  38. 		<name>ssl.server.keystore.type</name>
  39. 		<value>jks</value>
  40. 		<description>Optional. The keystore file format, default value is "jks".</description>
  41. 	</property>
  42. 	<property>
  43. 		<name>ssl.server.exclude.cipher.list</name>
  44. 		<value>TLS_ECDHE_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,
  45. 		SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,
  46. 		SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,
  47. 		SSL_RSA_WITH_RC4_128_MD5</value>
  48. 		<description>Optional. The weak security cipher suites that you want excluded from SSL communication.</description>
  49. 	</property>
  50. </configuration>

ssl-client.xml

Change ssl-client.xml.example to ssl-client.xml

  1. cp /usr/local/hadoop/etc/hadoop/ssl-client.xml.example /usr/local/hadoop/etc/hadoop/ssl-client.xml
  2.  
  3. nano /usr/local/hadoop/etc/hadoop/ssl-client.xml

Update properties

  1. <configuration>
  2. 	<property>
  3. 		<name>ssl.client.truststore.location</name>
  4. 		<value>/etc/security/serverKeys/truststore.jks</value>
  5. 		<description>Truststore to be used by clients like distcp. Must be specified.</description>
  6. 	</property>
  7. 	<property>
  8. 		<name>ssl.client.truststore.password</name>
  9. 		<value>PASSWORD</value>
  10. 		<description>Optional. Default value is "".</description>
  11. 	</property>
  12. 	<property>
  13. 		<name>ssl.client.truststore.type</name>
  14. 		<value>jks</value>
  15. 		<description>Optional. The keystore file format, default value is "jks".</description>
  16. 	</property>
  17. 	<property>
  18. 		<name>ssl.client.truststore.reload.interval</name>
  19. 		<value>10000</value>
  20. 		<description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds).</description>
  21. 	</property>
  22. 	<property>
  23. 		<name>ssl.client.keystore.location</name>
  24. 		<value></value>
  25. 		<description>Keystore to be used by clients like distcp. Must be specified.</description>
  26. 	</property>
  27. 	<property>
  28. 		<name>ssl.client.keystore.password</name>
  29. 		<value></value>
  30. 		<description>Optional. Default value is "".</description>
  31. 	</property>
  32. 	<property>
  33. 		<name>ssl.client.keystore.keypassword</name>
  34. 		<value></value>
  35. 		<description>Optional. Default value is "".</description>
  36. 	</property>
  37. 	<property>
  38. 		<name>ssl.client.keystore.type</name>
  39. 		<value>jks</value>
  40. 		<description>Optional. The keystore file format, default value is "jks".</description>
  41. 	</property>
  42. </configuration>

mapred-site.xml

Just add the following to the config to let it know the Kerberos keytabs to use.

  1. nano /usr/local/hadoop/etc/hadoop/mapred-site.xml
  2.  
  3. <property>
  4. 	<name>mapreduce.jobhistory.keytab</name>
  5. 	<value>/etc/security/keytabs/jhs.service.keytab</value>
  6. </property>
  7. <property>
  8. 	<name>mapreduce.jobhistory.principal</name>
  9. 	<value>jhs/_HOST@REALM.CA</value>
  10. </property>
  11. <property>
  12. 	<name>mapreduce.jobhistory.http.policy</name>
  13. 	<value>HTTPS_ONLY</value>
  14. </property>

hdfs-site.xml

Add the following properties

  1. nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  2.  
  3. <property>
  4. 	<name>dfs.http.policy</name>
  5. 	<value>HTTPS_ONLY</value>
  6. </property>
  7. <property>
  8. 	<name>hadoop.ssl.enabled</name>
  9. 	<value>true</value>
  10. </property>
  11. <property>
  12. 	<name>dfs.datanode.https.address</name>
  13. 	<value>NAMENODE:50475</value>
  14. </property>
  15. <property>
  16. 	<name>dfs.namenode.https-address</name>
  17. 	<value>NAMENODE:50470</value>
  18. 	<description>Your NameNode hostname for http access.</description>
  19. </property>
  20. <property>
  21. 	<name>dfs.namenode.secondary.https-address</name>
  22. 	<value>NAMENODE:50091</value>
  23. 	<description>Your Secondary NameNode hostname for http access.</description>
  24. </property>
  25. <property>
  26. 	<name>dfs.namenode.https-bind-host</name>
  27. 	<value>0.0.0.0</value>
  28. </property>
  29. <property>
  30. 	<name>dfs.block.access.token.enable</name>
  31. 	<value>true</value>
  32. 	<description> If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanod</description>
  33. </property>
  34. <property>
  35. 	<name>dfs.namenode.kerberos.principal</name>
  36. 	<value>nn/_HOST@REALM.CA</value>
  37. 	<description> Kerberos principal name for the NameNode</description>
  38. </property>
  39. <property>
  40. 	<name>dfs.secondary.namenode.kerberos.principal</name>
  41. 	<value>sn/_HOST@REALM.CA</value>
  42. 	<description>Kerberos principal name for the secondary NameNode.</description>
  43. </property>
  44. <property>
  45. 	<name>dfs.web.authentication.kerberos.keytab</name>
  46. 	<value>/etc/security/keytabs/spnego.service.keytab</value>
  47. 	<description>The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</description>
  48. </property>
  49. <property>
  50. 	<name>dfs.namenode.keytab.file</name>
  51. 	<value>/etc/security/keytabs/nn.service.keytab</value>
  52. 	<description>Combined keytab file containing the namenode service and host principals.</description>
  53. </property>
  54. <property>
  55. 	<name>dfs.datanode.keytab.file</name>
  56. 	<value>/etc/security/keytabs/dn.service.keytab</value>
  57. 	<description>The filename of the keytab file for the DataNode.</description>
  58. </property>
  59. <property>
  60. 	<name>dfs.datanode.kerberos.principal</name>
  61. 	<value>dn/_HOST@REALM.CA</value>
  62. 	<description>The Kerberos principal that the DataNode runs as. "_HOST" is replaced by the real host name.</description>
  63. </property>
  64. <property>
  65. 	<name>dfs.namenode.kerberos.internal.spnego.principal</name>
  66. 	<value>${dfs.web.authentication.kerberos.principal}</value>
  67. </property>
  68. <property>
  69. 	<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  70. 	<value>>${dfs.web.authentication.kerberos.principal}</value>
  71. </property>
  72. <property>
  73. 	<name>dfs.web.authentication.kerberos.principal</name>
  74. 	<value>HTTP/_HOST@REALM.CA</value>
  75. 	<description>The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</description>          
  76. </property>
  77. <property>
  78. 	<name>dfs.data.transfer.protection</name>
  79. 	<value>integrity</value>
  80. </property>
  81. <property>
  82. 	<name>dfs.datanode.address</name>
  83. 	<value>NAMENODE:50010</value>
  84. </property>
  85. <property>
  86. 	<name>dfs.secondary.namenode.keytab.file</name>
  87. 	<value>/etc/security/keytabs/sn.service.keytab</value>
  88. </property>
  89. <property>
  90. 	<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  91. 	<value>HTTP/_HOST@REALM.CA</value>
  92. </property>
  93. <property>
  94. 	<name>dfs.webhdfs.enabled</name>
  95. 	<value>true</value>
  96. </property>

Remove the following properties

  1. dfs.namenode.http-address
  2. dfs.namenode.secondary.http-address
  3. dfs.namenode.http-bind-host

yarn-site.xml

Add the following properties

  1. nano /usr/local/hadoop/etc/hadoop/yarn-site.xml
  2.  
  3. <property>
  4. 	<name>yarn.http.policy</name>
  5. 	<value>HTTPS_ONLY</value>
  6. </property>
  7. <property>
  8. 	<name>yarn.resourcemanager.webapp.https.address</name>
  9. 	<value>${yarn.resourcemanager.hostname}:8090</value>
  10. </property>
  11. <property>
  12. 	<name>yarn.resourcemanager.hostname</name>
  13. 	<value>NAMENODE</value>
  14. </property>
  15. <property>
  16. 	<name>yarn.nodemanager.bind-host</name>
  17. 	<value>0.0.0.0</value>
  18. </property>
  19. <property>
  20. 	<name>yarn.nodemanager.webapp.address</name>
  21. 	<value>${yarn.nodemanager.hostname}:8042</value>
  22. </property>
  23. <property>
  24. 	<name>yarn.resourcemanager.principal</name>
  25. 	<value>rm/_HOST@REALM.CA</value>
  26. </property>
  27. <property>
  28. 	<name>yarn.resourcemanager.keytab</name>
  29. 	<value>/etc/security/keytabs/rm.service.keytab</value>
  30. </property>
  31. <property>
  32. 	<name>yarn.nodemanager.principal</name>
  33. 	<value>nm/_HOST@REALM.CA</value>
  34. </property>
  35. <property>
  36. 	<name>yarn.nodemanager.keytab</name>
  37. 	<value>/etc/security/keytabs/nm.service.keytab</value>
  38. </property>
  39. <property>
  40. 	<name>yarn.nodemanager.hostname</name>
  41. 	<value>NAMENODE</value>
  42. </property>
  43. <property>
  44. 	<name>yarn.resourcemanager.bind-host</name>
  45. 	<value>0.0.0.0</value>
  46. </property>
  47. <property>
  48. 	<name>yarn.timeline-service.bind-host</name>
  49. 	<value>0.0.0.0</value>
  50. </property>

Remove the following properties

  1. yarn.resourcemanager.webapp.address

SSL

Setup SSL Directories

  1. sudo mkdir -p /etc/security/serverKeys
  2. sudo chown -R root:hadoopuser /etc/security/serverKeys/
  3. sudo chmod 755 /etc/security/serverKeys/
  4.  
  5. cd /etc/security/serverKeys

Setup Keystore

  1. sudo keytool -genkey -alias NAMENODE -keyalg RSA -keysize 1024 -dname "CN=NAMENODE,OU=ORGANIZATION_UNIT,C=canada" -keypass PASSWORD -keystore /etc/security/serverKeys/keystore.jks -storepass PASSWORD
  2. sudo keytool -export -alias NAMENODE -keystore /etc/security/serverKeys/keystore.jks -rfc -file /etc/security/serverKeys/NAMENODE.csr -storepass PASSWORD

Setup Truststore

  1. sudo keytool -import -noprompt -alias NAMENODE -file /etc/security/serverKeys/NAMENODE.csr -keystore /etc/security/serverKeys/truststore.jks -storepass PASSWORD

Generate Self Signed Certifcate

  1. sudo openssl genrsa -out /etc/security/serverKeys/NAMENODE.key 2048
  2.  
  3. sudo openssl req -x509 -new -key /etc/security/serverKeys/NAMENODE.key -days 300 -out /etc/security/serverKeys/NAMENODE.pem
  4.  
  5. sudo keytool -keystore /etc/security/serverKeys/keystore.jks -alias NAMENODE -certreq -file /etc/security/serverKeys/NAMENODE.cert -storepass PASSWORD -keypass PASSWORD
  6.  
  7. sudo openssl x509 -req -CA /etc/security/serverKeys/NAMENODE.pem -CAkey /etc/security/serverKeys/NAMENODE.key -in /etc/security/serverKeys/NAMENODE.cert -out /etc/security/serverKeys/NAMENODE.signed -days 300 -CAcreateserial

Setup File Permissions

  1. sudo chmod 440 /etc/security/serverKeys/*
  2. sudo chown root:hadoopuser /etc/security/serverKeys/*

Start the Cluster

  1. start-dfs.sh
  2. start-yarn.sh
  3. mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

Create User Directory

  1. kinit -kt /etc/security/keytabs/myuser.keytab myuser/hadoop@REALM.CA
  2. #ensure the login worked
  3. klist
  4.  
  5. #Create hdfs directory now
  6. hdfs dfs -mkdir /user
  7. hdfs dfs -mkdir /user/myuser
  8.  
  9. #remove kerberos ticket
  10. kdestroy

URL

https://NAMENODE:50470
https://NAMENODE:50475
https://NAMENODE:8090

References

https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/admin_ssl_hbase_mr_yarn_hdfs_web.html