Spark Installation on Hadoop

(Last Updated On: )

In this tutorial I will show you how to use Kerberos/SSL with Spark integrated with Yarn. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Hadoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

  1. cd /etc/security/keytabs/
  2.  
  3. sudo kadmin.local
  4.  
  5. #You can list princepals
  6. listprincs
  7.  
  8. #Create the following principals
  9. addprinc -randkey spark/hadoop@REALM.CA
  10.  
  11. #Create the keytab files.
  12. #You will need these for Hadoop to be able to login
  13. xst -k spark.service.keytab spark/hadoop@REALM.CA

Set Keytab Permissions/Ownership

  1. sudo chown root:hadoopuser /etc/security/keytabs/*
  2. sudo chmod 750 /etc/security/keytabs/*

Download

Go to Apache Spark Download and get the link for Spark.

  1. wget http://apache.forsale.plus/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
  2. tar -xvf spark-2.4.4-bin-hadoop2.7.tgz
  3. mv spark-2.4.4-bin-hadoop2.7 /usr/local/spark/

Update .bashrc

  1. sudo nano ~/.bashrc
  2.  
  3. #Ensure we have the following in the Hadoop section
  4. export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
  5.  
  6. #Add the following
  7.  
  8. #SPARK VARIABLES START
  9. export SPARK_HOME=/usr/local/spark
  10. export PATH=$PATH:$SPARK_HOME/bin
  11. export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
  12. #SPARK VARIABLES STOP
  13.  
  14. source ~/.bashrc

Setup Configuration

  1. cd /usr/local/spark/conf
  2. mv spark-defaults.conf.template spark-defaults.conf
  3. nano spark-defaults.conf
  4.  
  5. #Add to the end
  6. spark.master                            yarn
  7. spark.yarn.historyServer.address        ${hadoopconf-yarn.resourcemanager.hostname}:18080
  8. spark.yarn.keytab                       /etc/security/keytabs/spark.service.keytab
  9. spark.yarn.principal                    spark/hadoop@REALM.CA
  10. spark.yarn.access.hadoopFileSystems     hdfs://NAMENODE:54310
  11. spark.authenticate                      true
  12. spark.driver.bindAddress                0.0.0.0
  13. spark.authenticate.enableSaslEncryption true
  14. spark.eventLog.enabled                  true
  15. spark.eventLog.dir                      hdfs://NAMENODE:54310/user/spark/applicationHistory
  16. spark.history.fs.logDirectory           hdfs://NAMENODE:54310/user/spark/applicationHistory
  17. spark.history.fs.update.interval        10s
  18. spark.history.ui.port                   18080
  19.  
  20. #SSL
  21. spark.ssl.enabled                       true
  22. spark.ssl.keyPassword                   PASSWORD
  23. spark.ssl.keyStore                      /etc/security/serverKeys/keystore.jks
  24. spark.ssl.keyStorePassword              PASSWORD
  25. spark.ssl.keyStoreType                  JKS
  26. spark.ssl.trustStore                    /etc/security/serverKeys/truststore.jks
  27. spark.ssl.trustStorePassword            PASSWORD
  28. spark.ssl.trustStoreType                JKS

Kinit

  1. kinit -kt /etc/security/keytabs/spark.service.keytab spark/hadoop@REALM.CA
  2. klist
  3. hdfs dfs -mkdir /user/spark/
  4. hdfs dfs -mkdir /user/spark/applicationHistory
  5. hdfs dfs -ls /user/spark

Start The Service

  1. $SPARK_HOME/sbin/start-history-server.sh

Stop The Service

  1. $SPARK_HOME/sbin/stop-history-server.sh

Spark History Server Web UI

References

I used a lot of different resources and reference material on this. Below are just a few I used.

https://spark.apache.org/docs/latest/running-on-yarn.html#configuration

https://spark.apache.org/docs/latest/security.html

https://www.linode.com/docs/databases/hadoop/install-configure-run-spark-on-top-of-hadoop-yarn-cluster/