Tag: ambari

HortonWorks: Kerberize Ambari Server

This entry is part 7 of 7 in the series HortonWorks

You may want to integrate Kerberos authentication into your Ambari Server implementation. If you do follow the next few steps. It’s that easy.

Step 1: Stop Ambari Server

sudo ambari-server stop

Step 2: Create keytab file

ktutil
 
addent -password -p ##USER##@##DOMAIN##.COM -k 1 -e RC4-HMAC
 
# Enter password
 
wkt ##USER##.keytab
q
 
$ sudo mkdir /etc/security/keytabs
$ mv ##USER##.keytab /etc/security/keytabs

Step 3: Test Keytab. You should see the ticket once you klist.

kinit -kt /etc/security/keytabs/ambarisa.keytab -a ambarisa@AERYON.COM
klist

Step 4: Run Ambari Server Kerberos Setup

sudo ambari-server setup-kerberos

Follow the prompts. Say true to enabling kerberos. The keytab file will be the /etc/security/##USER##.keytab file. You should be able to leave the rest defaults. Save the settings and you are done.

Step 5: Remove the kinit ticket you created that way you can make sure you kerberos authentication is working correctly.

kdestroy

Step 6: Start Ambari Server

sudo ambari-server start

Step 7: Validate Kerberos. You should see your ticket get created and you should now be able to login with no issues.

klist

HortonWorks: Install YARN/MR

This entry is part 6 of 7 in the series HortonWorks

This tutorial guides you through installing YARN/MapReduce on Hortonworks using a multi node cluster setup with Ubuntu OS.

Step 1: Go to “Stack and Version”. Then click “Add Service” on YARN. You will notice that “MapReduce2” comes with it.

Step 2: Assign Masters I usually put the ResourceManager, History Server and App Timeline Server all on the secondary namenode. But it is totally up to you how you setup your environment.

Step 3: Assign Slaves and Clients I put NodeManagers on all the datanodes and Client’s on all servers. Up to you though. This is what worked for me and my requirements.

Step 4: During Customize Services you may get the warning that Ambari Metrics “hbase_master_heapsize” needs to be increased. I recommend doing this change but it’s up to you and what makes sense in your environment.

Step 5: Follow the remaining steps and installation should complete with no issues. Should an issue arise review the error and if it was just a turning on connection error then you may not have any issues and it just needs all services to be stopped and started again. Please not Ambari Metrics may report errors but they should clear in around 15 minutes.

 

HortonWorks: Install Hadoop

This entry is part 4 of 7 in the series HortonWorks

This tutorial guides you through installing Hadoop on hortonworks using a multi node cluster setup with Ubuntu OS.

Hosts File:

Ensure every server has the FQDN of all the servers to be in the cluster.

sudo nano /etc/hosts

SSH (Ambari Server)

You will do the following to all the servers in the Hadoop cluster.

ssh-keygen -t rsa -P ""
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
ssh-copy-id -i ~/.ssh/id_rsa.pub ##USER##@##FQDN##
ssh ##USER##@##FQDN##

Pre-Requisites: (not Ambari Server)

Java 8:

sudo apt-get install openjdk-8-jdk

Chrony:

sudo apt-get install chrony

Disable HugePage:

sudo su
echo never > /sys/kernel/mm/transparent_hugepage/enabled
exit

Install HDFS Service:

You will need to login to Ambari Server and click “Launch Install Wizard”. For the most part you will just follow the prompts. The major hurdles is that in the “Install Options” section make sure you put the FQDN (IE: host@domain.com). You will also need to get the SSH Private Key from the Ambari Server you just did during pre requisites from this location /home/##USER##/.ssh/id_rsa. Make sure you also set the SSH User Account to what you used during SSH creation. If for any reason it fails you can click the status to find out what failed and rectify the problem. As long as you did the pre-requisites you should be fine.

ZooKeeper / Ambari Metrics

As you install HDFS you will notice that Ambari Metrics and ZooKeeper get installed automatically. This is a good thing and you want it. ZooKeeper keeps all configs in sync and Ambari Metrics lets you easily monitor the system.

Assign Masters

You will need to setup how you want your masters to look. I usually have three zookeepers. Your secondary name node should go on a separate server. But it is totally up to you how you design your cluster. Have fun!

Assign Slaves / Clients

Your slaves (aka DataNodes) I don’t put any on my namenode or secondary namenode or my zookeeper servers. I leave my datanodes to perform that action alone. I also install clients on namenode, secondary namenode and all datanodes. Up to you how you configure it just have fun while doing it!

Key Config Optional Changes

Once you get to the customize services section. You can for the most part leave this as is and just do the password areas. But I do recommend reviewing the following and update as needed.

  1. HDFS: NameNode/DataNode/Secondary NameNode directories
  2. ZooKeeper: ZooKeeper directory

Deploy

Deploy should work with no issues. If there is issues sometimes you don’t need to worry about it. Such as connection issue. As long as it installed if it didn’t start right away and that was the connection issue then it may start once completed. You should also note that Ambari Metrics shows errors directly after starting. That is expected and no need to worry it will clear itself.

HortonWorks: SSL Setup

This entry is part 3 of 7 in the series HortonWorks

If you want to use SSL with Ambari Server (note this is not with Hadoop yet) then follow the below steps. Please note this does not cover the creation of a SSL Cert as there are many tutorials on how to create self signed certs, etc available.

Step 1: Stop the Ambari Server

sudo ambari-server stop

Step 2: Run Ambari Server Security Setup Command

sudo ambari-server setup-security

Select option 1 during the prompts and note that you cannot use port 443 for https as that is reserved in Ambari. The default is 8443 and that is what they recommend. Enter path to your cert /etc/ssl/certs/hostname.cer file. Enter path to your encrypted key /etc/ssl/private/hostname.key file. Follow the rest of the prompts.

Step 3: Start Ambari Server

sudo ambari-server start

Step 4: Login to Ambari Server now available at https://hostname:8443

HortonWorks: Ambari LDAP Integration

This entry is part 2 of 7 in the series HortonWorks

If you want to use LDAP with your Ambari Server then follow the below steps.

Step 1: Stop the Ambari Server to setup LDAP integration

sudo ambari-server stop

Step 2: Run Ambari Server LDAP Setup Command. This will require a bunch of settings to be set. Consult your IT department for your specific settings.

sudo ambari-server setup-ldap

Step 3: Create the groups and users text files and add the users you want to add comma separated to users and groups comma separated to the groups file.

nano ~/groups.txt
nano ~/users.txt

Step 4 (Optional): You may need to adjust the “SERVER_API_HOST” value to your ambari server hostname. Default is 127.0.0.1 which is technically your host but sometimes it complains and you need to make this modification.

sudo nano /usr/lib/python2.6/site-packages/ambari_server/serverUtils.py

Step 5: Import Groups/Users from the text files created in step 3. You will need to start the ambari server first.

sudo ambari-server start
#Import groups
sudo ambari-server sync-ldap --groups groups.txt
#Import users
sudo ambari-server sync-ldap --users users.txt

Step 6: Login to Ambari and got to manage ambari and you will see your new users and groups.

HortonWorks: Ambari Server Installation

This entry is part 1 of 7 in the series HortonWorks

I have been playing around with HortonWorks sandbox and thought it about time I attempt installation on a multi node cluster. Feel free to reach out to me for further support or information. I will be documenting more in the coming weeks.

Add Repo:
wget -O /etc/apt/sources.list.d/ambari.list http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.6.0.0/ambari.list
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
sudo apt-get update
Java 8
sudo apt-get install openjdk-8-jdk
Ambari Install
sudo apt-get install ambari-server
Configure Ambari

I recommend installed as non root. In fact do not do any of the defaults for users and passwords. But it is totally up to you. I will document more as I learn more.

sudo ambari-server setup
Ambari Start / Stop / Restart
sudo ambari-server restart
sudo ambari-server start
sudo ambari-server stop
Ambari Server Log Directory

I suggest changing the ambari log directory as well.

sudo vi /etc/ambari-server/conf/log4j.properties

#Look for the property "ambari.log.dir" and change it.
#Don't forget to create the folder you point to and ensure that chown to the user that is running ambari.
Ambari Agent Log Directory

As well change the ambari agent log directory.

sudo vi /etc/ambari-agent/conf/ambari-agent.ini

#Look for "logdir" and change to a directory that exists and has permissions.
Non-Root Install

Running as non-root user you will have to change the run directory because otherwise during a system reboot you may get folders not being able to be created. Because /var/run/ gets deleted on each restart and it get’s rebuilt. Do the following:

sudo mkdir -p /home/##NONROOTUSER##/run/ambari-server
sudo chown ##NONROOTUSER##:root /home/##NONROOTUSER##/run/ambari-server

Then you have to edit the ambari.properties file.

sudo vi /etc/ambari-server/conf/ambari.properties

#Change the following to the folder you created above.
bootstrap.dir
pid.dir
recommendations.dir