Oliver

PySpark: Save a DataFrame To ADLS

This how-to is how to save a DataFrame to ADLS

First we need a spark Session. See PySpark: Create a Spark Session for my details on that.

Then we need to create a DataFrame. See PySpark: Create a DataFrame.

Then we do the following:

You should note you don’t need all the options below. I just gave an example.

path = 'abfss://my_container@my_storage_account.dfs.core.windows.net/my_folder/'
mode = 'overwrite'
format = 'delta'
partitions = []
 
df.write.mode(mode).format(format).option('mergeSchema', False).partitionBy(*partitions).save(path)

PySpark: Create a Spark Session

This post is how to create a Spark Session

Imports

from pyspark.sql import SparkSession

Create the Spark Session

spark = SparkSession.builder.appName('pyspark_app_name').getOrCreate()

You can add any configs you wish during creation. You would add this before the “.getOrCreate()”.

You can see a list here

.config(“spark.sql.jsonGenerator.ignoreNullFields”, “false”)
- When reading JSON you will not ignore NULL fields
.config(“spark.sql.parquet.int96RebaseModeInWrite”, “CORRECTED”)
- Fixes issues in timestamps in write operations
.config(“spark.sql.parquet.int96RebaseModeInRead”, “CORRECTED”)
- Fixes issues in timestamps in read operations

PySpark: Create a DataFrame

This post is how to create a DataFrame in pyspark.

First we need a spark Session. See PySpark: Create a Spark Session for my details on that.

Next we need to import

from pyspark.sql import Row
from pyspark.sql.types import StringType, DecimalType, TimestampType, FloatType, IntegerType, LongType, StructField, StructType

Then you create the schema

schema = StructType([
    StructField('id', IntegerType()),
    .....
])
 
data = [Row(id=1)]

Create the DataFrame

df = spark.createDataFrame(data, schema=schema)

If you want to use a JSON file to build your schema do the following

import json
from pyspark.sql.types import StructType
 
data = {
    "fields": [
        {
            "metadata": {},
            "name": "column_a",
            "nullable": false,
            "type": "string"
        }
    ],
    "type": "struct"
}
 
json_schema = json.loads(data)
table_schema = StructType.fromJson(dict(json_schema))
 
df = spark.createDataFrame(data, schema=table_schema)

Python: Unit Testing

This post focus’ on common hurdles when trying to do unit testing.

Testing Values During Run

You add the following line to anywhere you want to pause the unit test to check values.

import pdb
pdb.set_trace()

How to Patch a Function

from unittest.mock import path
 
@patch('src.path.to.file.my_function')
@path('src.path.to.file.my_function_add')
def test_some_function(mock_my_function_add, mock_my_function):
    mock_function_add.return_value = <something>
    .......

How to Patch a Function With No Return Value

from unittest.mock import patch
 
def test_some_function():
    with(patch('src.path.to.file.my_function'):
        ...

How to Patch a Function With 1 Return Value

from unittest.mock import patch
 
def test_some_function():
    with(patch('src.path.to.file.my_function', MagicMock(return_value=[<MY_VALUES>])):
        ...

How to Patch a Function With Multiple Return Value

from unittest.mock import patch
 
def test_some_function():
    with(patch('src.path.to.file.my_function', MagicMock(side-effect=[[<MY_VALUES>], [<OTHER_VALUES>]])):
        ...

How to Create a Test Module

from unittest import TestCase
 
class MyModule(TestCase):
    def setUp(self):
        some_class.my_variable = <something>
        ... DO OTHER STUFF
    def test_my_function(self):
        ... DO Function Test Stuff

How to Patch a Method

patch_methods = [
    "pyodbc.connect"
]
 
for method in patch_methods:
    patch(method).start()

How to create a PySpark Session

Now once you do this you can just call spark and it will set it.

import pytest
from pyspark.sql import SparkSession
 
@pytest.fixture(scope='module')
def spark():
    return (SparkSession.builder.appName('pyspark_test').getOrCreate())

How to Create a Spark SQL Example

import pytest
from pyspark.sql import SparkSession, Row
from pyspark.sql.types import StructType, StructField, StringType
 
@pytest.fixture(scope='module')
def spark():
    return (SparkSession.builder.appName('pyspark_test').getOrCreate())
 
def test_function(spark):
    query = 'SELECT * FROM SOMETHING'
    schema = StructType([
        StructField('column_a', StringType()),
        StructField('column_b', StringType()),
        StructField('column_c', StringType()),
    ])
 
data = [Row(column_a='a', column_b='b', column_c='c')]
table = spark.createDataFrame(data, schema=schema)
table.createOrReplaceTempView('<table_name>')
df = spark.sql(query).toPandas()
 
assert not df.empty
assert df.shape[0] == 1
assert df.shape(1) == 5
 
spark.catalog.dropTempView('<table_name>')

How to Mock a Database Call

First let’s assume you have an exeucte sql function

def execute_sql(cursor, sql, params):
    result = cursor.execute(sql, params).fetchone()
    connection.commit()
    return result

Next in your unit tests you want to test that funciton

def test_execute_sql():
    val = <YOUR_RETURN_VALUE>
    with patch('path.to.code.execute_sql', MagicMock(return_value=val)) as mock_execute:
        return_val = some_other_function_that_calls_execute_sql(....)
        assert return_val == val

If you need to close a cursor or DB connection

def test_execute_sql():
    val = <YOUR_RETURN_VALUE>
    mock_cursor = MagicMock()
    mock_cursor.configure_mock(
        **{
              "close": MagicMock()
         }
    )
    mock_connection = MagicMock()
    mock_connection.configure_mock(
        **{
            "close": MagicMock()
        }
    )
 
    with patch('path.to.code.cursor', MagicMock(return_value=mock_cursor)) as mock_cursor_close:
        with patch('path.to.code.connection', MagicMock(return_value=mock_connection)) as mock_connection_close:
            return_val = some_other_function_that_calls_execute_sql(....)
            assert return_val == val

How to Mock Open a File Example 1

@patch('builtins.open", new_callable=mock_open, read_data='my_data')
def test_file_open(mock_file):
    assert open("my/file/path/filename.extension").read() == 'my_data'
    mock_file.assert_called_with("my/file/path/filename.extension")
 
    val = function_to_test(....)
    assert 'my_data' == val

How to Mock Open a File Example 2

def test_file_open():
    fake_file_path = 'file/path/to/mock'
    file_content_mock = 'test'
    with patch('path.to.code.function'.format(__name__), new=mock_open(read_data=file_content_mock)) as mock_file:
        with patch(os.utime') as mock_utime:
            actual = function_to_test(fake_file_path)
            mock_file.assert_called_once_with(fake_file_path)
            assertIsNotNone(actual)

Compare DataFrames

def as_dicts(df):
    df = [row.asDict() for row in df.collect()]
    return sorted(df, key=lambda row: str(row))
 
assert as_dicts(df1) == as_dicts(df2)

Python: Create a WHL File

This post will just be a how-to on creating a whl file.

You need the following files:

Manifest.in:

recursive-include <directory> *
recursive-exclude tests *.py

Requirements.txt:

This file just holds your packages and the version.

Setup.py

You remove pytest and coverage from your whl file because you don’t want those applications being required when you deploy your code.

from setuptools import find_packages
from distutils.core import setup
import os
import json
 
if os.path.exists('requirements.txt'):
    req = [line.strip('\n') for line in open('requirements.txt') if 'pytest' not in line and 'coverage' not in line]
 
setup(
    include_package_data=True,
    name=<app_name>,
    version=<app-version>,
    description=<app_desc>,
    install_requires=req,
    packages=find_packages(excude=["*tests.*","*tests"]),
    classifiers=[
        "Programming Language :: Python || <python_Version>",
        "License || OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=<python_version>',
    package_dir={<directory>: <directory>},
)

To Check Your Whl File

Install package

pip install check-wheel-contents

Check WHL

check-wheel-contents <PATH_TO_WHL>\<filename>.whl

Install WHL

This will deploy to <PATH_TO_PYTHON>\Lib\site-packages\<directory>

<PATH_TO_PYTHON>\Scripts\pip3.7.exe install <PATH_TO_WHL>\<filename>.whl

This will publish events to event hub. The important part here is the “EndPoint”. You need to login to Azure Portal and get the get the endpoint from the “Shared Access Policies” from the event hub namespace.

from azure.eventhub import EventHubProducerClient, EventData, EventHubConsumerClient
 
connection_str = 'Endpoint=sb://testeventhubnamespace.servicebus.windows.net/;SharedAccessKeyName=<<THE_ACCESS_KEY_NAME>>;SharedAccessKey=<<THE_ACCESS_KEY>>'
eventhub_name = '<<THE_EVENT_HUB_NAME>>'
producer = EventHubProducerClient.from_connection_string(connection_str, eventhub_name=eventhub_name)
 
event_data_batch = producer.create_batch()
 
event_data_batch.add(EventData('My Test Data'))
 
with producer:
    producer.send_batch(event_data_batch)

Create a Consumer

This will monitor the event hub for new messages.

from azure.eventhub import EventHubProducerClient, EventData, EventHubConsumerClient
 
connection_str = 'Endpoint=sb://testeventhubnamespace.servicebus.windows.net/;SharedAccessKeyName=<<THE_ACCESS_KEY_NAME>>;SharedAccessKey=<<THE_ACCESS_KEY>>'
eventhub_name = '<<THE_EVENT_HUB_NAME>>'
consumer_group = '<<THE_EVENT_HUB_CONSUMER_GROUP>>'
client = EventHubConsumerClient.from_connection_string(connection_str, consumer_group, eventhub_name=eventhub_name)
 
def on_event(partition_context, event):
    print("Received event from partition {} - {}".format(partition_context.partition_id, event))
    partition_context.update_checkpoint(event)
 
with client:
    #client.receive(
    #    on_event=on_event, 
    #    starting_position="-1",  # "-1" is from the beginning of the partition.
    #)
    client.receive(
        on_event=on_event
    )

Azure Toolkit Installation for Eclipse

In this tutorial I will show you how to install the Azure Toolkit for Eclipse. Ensure you have already installed Eclipse.

Step 1: Click Eclipse Marketplace.

Step 2: Search for “Azure Toolkit”.

Step 3: Review License.

Step 4: Click Finish. It will now install Azure Toolkit.

Hive: Database

This tutorial will show you some common usage for working with tables. If you have no installed Hive yet please follow this tutorial.

Create Database

CREATE DATABASE test;

Use Database

use test;

Jupyter Installation

In this tutorial I will show you how to install Jupyter. I will use self signed certs for this example.

This assumes your hostname is “hadoop”

Prerequisites

Python3.5 Installation

sudo apt install python3-pip

Update .bashrc

sudo nano ~/.bashrc
 
#Add the following
alias python=python3.5
 
source ~/.bashrc

Install

pip3 install jupyter
 
jupyter notebook --generate-config
 
jupyter notebook password
#ENTER PASSWORD
 
cat  ~/.jupyter/jupyter_notebook_config.json
#Get the SHA1 value

Setup Configuration

nano ~/.jupyter/jupyter_notebook_config.py
 
#Find and change the values for the following
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.port = 8888
c.NotebookApp.password = u'sha1:1234567fbbd5:dfgy8e0a3l12fehh46ea89f23jjjkae54a2kk54g'
c.NotebookApp.open_browser = False
c.NotebookApp.certfile = '/etc/security/serverKeys/hadoop.pem'
c.NotebookApp.keyfile = '/etc/security/serverKeys/hadoop.key'

Run Jupyter

jupyter notebook

https://NAMENODE:8888

References

https://jupyter.readthedocs.io/en/latest/index.html

Spark Installation on Hadoop

In this tutorial I will show you how to use Kerberos/SSL with Spark integrated with Yarn. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and Hadoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

cd /etc/security/keytabs/
 
sudo kadmin.local
 
#You can list princepals
listprincs
 
#Create the following principals
addprinc -randkey spark/hadoop@REALM.CA
 
#Create the keytab files.
#You will need these for Hadoop to be able to login
xst -k spark.service.keytab spark/hadoop@REALM.CA

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Download

Go to Apache Spark Download and get the link for Spark.

wget http://apache.forsale.plus/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
tar -xvf spark-2.4.4-bin-hadoop2.7.tgz
mv spark-2.4.4-bin-hadoop2.7 /usr/local/spark/

Update .bashrc

sudo nano ~/.bashrc
 
#Ensure we have the following in the Hadoop section
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
 
#Add the following
 
#SPARK VARIABLES START
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
#SPARK VARIABLES STOP
 
source ~/.bashrc

Setup Configuration

cd /usr/local/spark/conf
mv spark-defaults.conf.template spark-defaults.conf
nano spark-defaults.conf
 
#Add to the end
spark.master                            yarn
spark.yarn.historyServer.address        ${hadoopconf-yarn.resourcemanager.hostname}:18080
spark.yarn.keytab                       /etc/security/keytabs/spark.service.keytab
spark.yarn.principal                    spark/hadoop@REALM.CA
spark.yarn.access.hadoopFileSystems     hdfs://NAMENODE:54310
spark.authenticate                      true
spark.driver.bindAddress                0.0.0.0
spark.authenticate.enableSaslEncryption true
spark.eventLog.enabled                  true
spark.eventLog.dir                      hdfs://NAMENODE:54310/user/spark/applicationHistory
spark.history.fs.logDirectory           hdfs://NAMENODE:54310/user/spark/applicationHistory
spark.history.fs.update.interval        10s
spark.history.ui.port                   18080
 
#SSL
spark.ssl.enabled                       true
spark.ssl.keyPassword                   PASSWORD
spark.ssl.keyStore                      /etc/security/serverKeys/keystore.jks
spark.ssl.keyStorePassword              PASSWORD
spark.ssl.keyStoreType                  JKS
spark.ssl.trustStore                    /etc/security/serverKeys/truststore.jks
spark.ssl.trustStorePassword            PASSWORD
spark.ssl.trustStoreType                JKS

Kinit

kinit -kt /etc/security/keytabs/spark.service.keytab spark/hadoop@REALM.CA
klist
hdfs dfs -mkdir /user/spark/
hdfs dfs -mkdir /user/spark/applicationHistory
hdfs dfs -ls /user/spark

Start The Service

$SPARK_HOME/sbin/start-history-server.sh

Stop The Service

$SPARK_HOME/sbin/stop-history-server.sh

Spark History Server Web UI

References

I used a lot of different resources and reference material on this. Below are just a few I used.

https://spark.apache.org/docs/latest/running-on-yarn.html#configuration

https://spark.apache.org/docs/latest/security.html

https://www.linode.com/docs/databases/hadoop/install-configure-run-spark-on-top-of-hadoop-yarn-cluster/

Python: xlrd (Read Excel File)

In this tutorial I will show you how to read an excel file in Python.

Installation

pip install xlrd

Open The Workbook

import xlrd
 
my_excel = (r'C:\path\to\file')
wb = xlrd.open_workbook(my_excel)

Select Sheet

# Select the first sheet. If you want to select the third just change to (3)
sheet = wb.sheet_by_index(0)

Get Data In Column

#This loops through all the rows in that sheet
for i in range(sheet.nrows):
        # if the value isn't empty then print it out.
        if sheet.cell_value(i, 0) != '':
            print(sheet.cell_value(i, 0))

Get all the Column Header

#This loops through all the rows in that sheet
for i in range(sheet.ncols):
        # if the value isn't empty then print it out.
        if sheet.cell_value(0, i) != '':
            print(sheet.cell_value(0, i))

Sqoop2: Kerberize Installation

In this tutorial I will show you how to kerberize Sqoop installation. Before you begin ensure you have installed Sqoop.

This assumes your hostname is “hadoop”

Create Kerberos Principals

cd /etc/security/keytabs
sudo kadmin.local
addprinc -randkey sqoop/hadoop@REALM.CA
xst -kt sqoop.service.keytab sqoop/hadoop@REALM.CA
addprinc -randkey sqoopHTTP/hadoop@REALM.CA
xst -kt sqoopHTTP.service.keytab sqoopHTTP/hadoop@REALM.CA
q

Set Keytab Permissions/Ownership

sudo chown root:hadoopuser /etc/security/keytabs/*
sudo chmod 750 /etc/security/keytabs/*

Configuration

Configure Kerberos with Sqoop

cd /usr/local/sqoop/conf/
nano sqoop.properties
 
#uncomment the following
org.apache.sqoop.security.authentication.type=KERBEROS
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.KerberosAuthenticationHandler
 
#update to the following
org.apache.sqoop.security.authentication.kerberos.principal=sqoop/hadoop@GAUDREAULT_KDC.CA
org.apache.sqoop.security.authentication.kerberos.keytab=/etc/security/keytabs/sqoop.service.keytab

Sqoop2: Installation

We are going to install Sqoop. Ensure you have Hadoop installed already.

This assumes your hostname is “hadoop”

Install Java JDK

apt-get update
apt-get upgrade
apt-get install default-jdk

Download Sqoop:

wget https://archive.apache.org/dist/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz
tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz
sudo mv sqoop-1.99.7-bin-hadoop200 /usr/local/sqoop/
sudo chown -R root:hadoopuser /usr/local/sqoop/

Setup .bashrc:

 sudo nano ~/.bashrc

Add the following to the end of the file.

#SQOOP VARIABLES START
export SQOOP_HOME=/usr/local/sqoop
export PATH=$PATH:$SQOOP_HOME/bin
export SQOOP_CONF_DIR=$SQOOP_HOME/conf
export SQOOP_CLASS_PATH=$SQOOP_CONF_DIR
#SQOOP VARIABLES STOP

 source ~/.bashrc

Initialise Repository

./bin/sqoop2-tool upgrade

Modify sqoop2-server

If you are running Hadoop on the same server as Sqoop Server you will need to modify this file. The reason is because Sqoop needs you to point to the lib directory for common, hdfs, mapreduce and yarn.

nano /usr/loca/sqoop/bin/sqoop.sh
 
#Modify these lines
  HADOOP_COMMON_HOME=${HADOOP_COMMON_HOME:-${HADOOP_HOME}/share/hadoop/common}
  HADOOP_HDFS_HOME=${HADOOP_HDFS_HOME:-${HADOOP_HOME}/share/hadoop/hdfs}
  HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-${HADOOP_HOME}/share/hadoop/mapreduce}
  HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-${HADOOP_HOME}/share/hadoop/yarn}
 
#TO
 
  HADOOP_COMMON_HOME=${HADOOP_HOME}/share/hadoop/common
  HADOOP_HDFS_HOME=${HADOOP_HOME}/share/hadoop/hdfs
  HADOOP_MAPRED_HOME=${HADOOP_HOME}/share/hadoop/mapreduce
  HADOOP_YARN_HOME=${HADOOP_HOME}/share/hadoop/yarn

Configuration

nano /usr/local/sqoop/conf/sqoop.properties
#Update the following line
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/etc/hadoop/

Start Sqoop Server

./bin/sqoop2-server start

References

https://linoxide.com/tools/install-apache-sqoop-ubuntu-16-04/

PySpark: Eclipse Integration

This tutorial will guide you through configuring PySpark on Eclipse.

First you need to install Eclipse.

You need to add “pyspark.zip” and “py4j-0.10.7-src.zip” to “Libraries” for the Python Interpreter.

Next you need to configure the Environment variables for PySpark.

Test that it works!

from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession
 
def init_spark():
    spark = SparkSession.builder.appName("HelloWorld").getOrCreate()
    sc = spark.sparkContext
    return spark,sc
 
if __name__ == '__main__':
    spark,sc = init_spark()
    nums = sc.parallelize([1,2,3,4])
    print(nums.map(lambda x: x*x).collect())

Shell: Functions

In this tutorial I will show you how to work with functions.

Define Function

You return 0 for success and 1 for failure

function test()
{
    return 0
}

Call function with no arguments

test

Call function with arguments

val1='test'
val2='test2'
test $val1 $val2

Call function with arguments that have spaces in the value

val1='test'
val2='test2 test23'
test "${val1}" "${val2}"

PySpark: StandAlone Installation on Windows

This tutorial will guide you through installing PySpark StandAlone on Windows for development.

Install Python 3

You need to have python 3 installed. You can get it here.

Install Spark

Go to Apache Spark and download the latest version and package “Pre-built for Apache Hadoop 2.7 and later”. Download spark-2.4.1-bin-hadoop2.7.tgz.

Install 7 Zip

You will need 7 Zip to open spark-2.4.1-bin-hadoop2.7.tgz.

Install Java

You need to ensure you have Java 8 install.

Extract Spark

Once you have installed 7 Zip you can extract spark into C:\spark\ directory. The directory structure will look like this c:\spark\spark-2.4.1-bin-hadoop2.7\

Download WinUtils.exe

Download the winutils.exe and put to C:\spark\spark-2.4.1-bin-hadoop2.7\bin\

Environment Variables

You the following commands to set your Spark specific ENV variables. The “-m” option means all users. You can either use that or not. If you don’t then it adds for current user only.

setx -m SPARK_HOME C:\spark\spark-2.4.1-bin-hadoop2.7
setx -m HADOOP_HOME C:\spark\spark-2.4.1-bin-hadoop2.7

You also want to add the following to the “Path” env variable “;C:\spark\spark-2.4.1-bin-hadoop2.7\bin”

Run PySpark

Open a command prompt and type the following. The –master parameter is used for setting the master node address. local[2] is to tell Spark to run locally on 2 cores.

pyspark --master local[2]

Test

You can then test that it is working by running the following code.

words = sc.parallelize (
   ["scala", 
   "java", 
   "hadoop", 
   "spark", 
   "hbase",
   "spark vs hadoop", 
   "pyspark",
   "pyspark and spark"]
)
counts = words.count()
print("Number of elements in RDD -> %i" % (counts))

References

I used this as a guide. https://medium.com/@GalarnykMichael/install-spark-on-windows-pyspark-4498a5d8d66c

Hadoop 3.2.0: Installation

I would like to share what I have learned and applied in the hopes that it will help someone else configure their system. The deployment I have done is to have a Name Node and 1-* DataNodes on Ubuntu 16.04 assuming 5 cpu and 13GB RAM. I will put all commands used in this tutorial right down to the very basics for those that are new to Ubuntu.

NOTE: Sometimes you may have to use “sudo” in front of the command. I also use nano for this article for beginners but you can use any editor you prefer (ie: vi). Also this article does not take into consideration any SSL, kerberos, etc. For all purposes here Hadoop will be open without having to login, etc.

Additional Setup/Configurations to Consider:

Zookeeper: It is also a good idea to use ZooKeeper to synchronize your configuration

Secondary NameNode: This should be done on a seperate server and it’s function is to take checkpoints of the namenodes file system.

Rack Awareness: Fault tolerance to ensure blocks are placed as evenly as possible on different racks if they are available.

Apply the following to all NameNode and DataNodes unless otherwise directed:

Hadoop User:
For this example we will just use hduser as our group and user for simplicity sake.
The “-a” on usermod is for appending to a group used with –G for which groups

addgroup hduser
sudo gpasswd -a $USER sudo
usermod –a –G sudo hduser

Install JDK:

apt-get update
apt-get upgrade
apt-get install default-jdk

Install SSH:

apt-get install ssh
which ssh
which sshd

These two commands will check that ssh installed correctly and will return “/usr/bin/ssh” and “/usr/bin/sshd”

java -version

You use this to verify that java installed correctly and will return something like the following.

openjdk version “1.8.0_171”
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

System Configuration

nano ~/.bashrc

The .bashrc is a script that is executed when a terminal session is started.
Add the following line to the end and save because Hadoop uses IPv4.

export _JAVA_OPTIONS=’-XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true’

source ~/.bashrc

sysctl.conf

Disable ipv6 as it causes issues in getting your server up and running.

nano /etc/sysctl.conf

Add the following to the end and save

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
#Change eth0 to what ifconfig has
net.ipv6.conf.eth0.disable_ipv6 = 1

Close sysctl

sysctl -p
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
reboot

If all the above disabling IPv6 configuration was successful you should get “1” returned.
Sometimes you can reach open file descriptor limit and open file limit. If you do encounter this issue you might have to set the ulimit and descriptor limit. For this example I have set some values but you will have to figure out the best numbers for your specific case.

If you get “cannot stat /proc/sys/-p: No such file or directory”. Then you need to add /sbin/ to PATH.

sudo nano ~/.bashrc
export PATH=$PATH:/sbin/

nano /etc/sysctl.conf

fs.file-max = 500000

sysctl –p

limits.conf

nano /etc/security/limits.conf

* soft nofile 60000
* hard nofile 60000

 reboot

Test Limits

You can now test the limits you applied to make sure they took.

ulimit -a
more /proc/sys/fs/file-max
more /proc/sys/fs/file-nr
lsof | wc -l

file-max: Current open file descriptor limit
file-nr: How many file descriptors are currently being used
lsof wc: How many files are currently open

You might be wondering why we installed ssh at the beginning. That is because Hadoop uses ssh to access its nodes. We need to eliminate the password requirement by setting up ssh certificates. If asked for a filename just leave it blank and confirm with enter.

su hduser

If not already logged in as the user we created in the Hadoop user section.

ssh-keygen –t rsa –P ""

You will get the below example as well as the fingerprint and randomart image.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory ‘/home/hduser/.ssh’.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.

cat $HOME/.ssh/id-rsa.pub >> $HOME/.ssh/authorized_keys

You may get “No such file or directory”. It is most likely just the id-rsa.pub filename. Look in the .ssh directory for the name it most likely will be “id_rsa.pub”.

This will add the newly created key to the list of authorized keys so that Hadoop can use SSH without prompting for a password.
Now we check that it worked by running “ssh localhost”. When prompted with if you should continue connecting type “yes” and enter. You will be permanently added to localhost
Once we have done this on all Name Node and Data Node you should run the following command from the Name Node to each Data Node.

ssh-copy-id –i ~/.ssh/id_rsa.pub hduser@DATANODEHOSTNAME
ssh DATANODEHOSTNAME

/etc/hosts Update

We need to update the hosts file.

sudo nano /etc/hosts
 
#Comment out line "127.0.0.1 localhost"
 
127.0.0.1 HOSTNAME localhost

Now we are getting to the part we have been waiting for.

Hadoop Installation:

NAMENODE: You will see this in the config files below and it can be the hostname, the static ip or it could be 0.0.0.0 so that all TCP ports will be bound to all IP’s of the server. You should also note that the masters and slaves file later on in this tutorial can still be the hostname.

Note: You could run rsync after setting up the Name Node Initial configuration to each Data Node if you want. This would save initial hadoop setup time. You do that by running the following command:

rsync –a /usr/local/hadoop/ hduser@DATANODEHOSTNAME:/usr/local/hadoop/

Download & Extract:

wget https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz 
tar xvzf hadoop-3.2.0.tar.gz
sudo mv hadoop-3.2.0/ /usr/local/hadoop
chown –R hduser:hduser /usr/local/hadoop
update-alternatives --config java

Basically the above downloads, extracts, moves the extracted hadoop directory to the /usr/local directory, if the hduser doesn’t own the newly created directory then switch ownership
and tells us the path where java was been installed to to set the JAVA_HOME environment variable. It should return something like the following:

There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

nano ~/.bashrc

Add the following to the end of the file. Make sure to do this on Name Node and all Data Nodes:

#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_INSTALL/lib”
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_HOME=$HADOOP_INSTALL

export HDFS_NAMENODE_USER=hduser
export HDFS_DATANODE_USER=hduser
export HDFS_SECONDARYNAMENODE_USER=hduser

#HADOOP VARIABLES END

source ~/.bashrc
javac –version
which javac
readlink –f /usr/bin/javac

This basically validates that bashrc update worked!
javac should return “javac 1.8.0_171” or something similar
which javac should return “/usr/bin/javac”
readlink should return “/usr/lib/jvm/java-8-openjdk-amd64/bin/javac”

Memory Tools

There is an application from HortonWorks you can download which can help get you started on how you should setup memory utilization for yarn. I found it’s a great starting point but you need to tweak it to work for what you need on your specific case.

wget http://public-repo-1.hortonworks.com/HDP/tools/2.6.0.3/hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
tar zxvf hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
cd hdp_manual_install_rpm_helper_files-2.6.0.3.8/
sudo apt-get install python2.7
python2.7 scripts/yarn-utils.py -c 5 -m 13 -d 1 -k False

-c is for how many cores you have
-m is for how much memory you have
-d is for how many disks you have
False is if you are running HBASE. True if you are.

After the script is ran it will give you guidelines on yarn/mapreduce settings. See below for example. Remember they are guidelines. Tweak as needed.
Now the real fun begins!!! Remember that these settings are what worked for me and you may need to adjust them.

hadoop-env.sh

nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh

You will see JAVA_HOME near the beginning of the file you will need to change that to where java is installed on your system.

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HEAPSIZE=1000
export HADOOP_NAMENODE_OPTS=”-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,DRFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,RFAAUDIT} $HADOOP_NAMENODE_OPTS”
export HADOOP_SECONDARYNAMENODE_OPTS=$HADOOP_NAMENODE_OPTS
export HADOOP_CLIENT_OPTS=”-Xmx1024m $HADOOP_CLIENT_OPTS”

mkdir –p /app/hadoop/tmp

This is the temp directory hadoop uses

chown hduser:hduser /app/hadoop/tmp

core-site.xml

Click here to view the docs.

nano /usr/local/hadoop/etc/hadoop/core-site.xml

This file contains configuration properties that Hadoop uses when starting up. By default it will look like . This will need to be changed.

<configuration>
      <property>
            <name>fs.defaultFS</name>
            <value>hdfs://NAMENODE:54310</value>
            <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description>
      </property>
      <property>
            <name>hadoop.tmp.dir</name>
            <value>/app/hadoop/tmp</value>
      </property>
      <property>
            <name>hadoop.proxyuser.hduser.hosts</name>
            <value>*</value>
      </property>
      <property>
            <name>hadoop.proxyuser.hduser.groups</name>
            <value>*</value>
      </property>
</configuration>

yarn-site.xml

Click here to view the docs.

nano /usr/local/hadoop/etc/hadoop/yarn-site.xml

<configuration>
      <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
      </property>
      <property>
            <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
      </property>
      <property>
            <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
            <value>org.apache.hadoop.mapred.ShuffleHandler</value>
      </property>
      <property>
            <name>yarn.nodemanager.resource.memory-mb</name>
            <value>12288</value>
            <final>true</final>
      </property>
      <property>
            <name>yarn.scheduler.minimum-allocation-mb</name>
            <value>4096</value>
            <final>true</final>
      </property>
      <property>
            <name>yarn.scheduler.maximum-allocation-mb</name>
            <value>12288</value>
            <final>true</final>
      </property>
      <property>
            <name>yarn.app.mapreduce.am.resource.mb</name>
            <value>4096</value>
      </property>
      <property>
            <name>yarn.app.mapreduce.am.command-opts</name>
            <value>-Xmx3276m</value>
      </property>
      <property>
            <name>yarn.nodemanager.local-dirs</name>
            <value>/app/hadoop/tmp/nm-local-dir</value>
      </property>
      <!--LOG-->
      <property>
            <name>yarn.log-aggregation-enable</name>
            <value>true</value>
      </property>
      <property>
            <description>Where to aggregate logs to.</description>
            <name>yarn.nodemanager.remote-app-log-dir</name>
            <value>/tmp/yarn/logs</value>
      </property>
      <property>
            <name>yarn.log-aggregation.retain-seconds</name>
            <value>604800</value>
      </property>
      <property>
            <name>yarn.log-aggregation.retain-check-interval-seconds</name>
            <value>86400</value>
      </property>
      <property>
            <name>yarn.log.server.url</name>
            <value>http://NAMENODE:19888/jobhistory/logs/</value>
      </property>
      
      <!--URLs-->
      <property>
            <name>yarn.resourcemanager.resource-tracker.address</name>
            <value>${yarn.resourcemanager.hostname}:8025</value>
      </property>
      <property>
            <name>yarn.resourcemanager.scheduler.address</name>
            <value>${yarn.resourcemanager.hostname}:8030</value>
      </property>
      <property>
            <name>yarn.resourcemanager.address</name>
            <value>${yarn.resourcemanager.hostname}:8050</value>
      </property>
      <property>
            <name>yarn.resourcemanager.admin.address</name>
            <value>${yarn.resourcemanager.hostname}:8033</value>
      </property>
      <property>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>${yarn.nodemanager.hostname}:8088</value>
      </property>
      <property>
            <name>yarn.nodemanager.hostname</name>
            <value>0.0.0.0</value>
      </property>
      <property>
            <name>yarn.nodemanager.address</name>
            <value>${yarn.nodemanager.hostname}:0</value>
      </property>
      <property>
            <name>yarn.nodemanager.webapp.address</name>
            <value>${yarn.nodemanager.hostname}:8042</value>
      </property>
</configuration>

By default it will look like . This will need to be changed.

mapred-site.xml

Click here to view the docs. By default, the /usr/local/hadoop/etc/hadoop/ folder contains /usr/local/hadoop/etc/hadoop/mapred-site.xml.template file which has to be renamed/copied with the name mapred-site.xml By default it will look like . This will need to be changed.

cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
 
nano /usr/local/hadoop/etc/hadoop/mapred-site.xml

<configuration>
      <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
      </property>
      <property>
            <name>mapreduce.jobhistory.address</name>
            <value>0.0.0.0:10020</value>
      </property>
      <property>
            <name>mapreduce.jobhistory.webapp.address</name>
            <value>0.0.0.0:19888</value>
      </property>
      <property>
            <name>mapreduce.jobtracker.address</name>
            <value>0.0.0.0:54311</value>
      </property>
      <property>
            <name>mapreduce.jobhistory.admin.address</name>
            <value>0.0.0.0:10033</value>
      </property>
      <!-- Memory and concurrency tuning -->
      <property>
            <name>mapreduce.map.memory.mb</name>
            <value>4096</value>
      </property>
      <property>
            <name>mapreduce.map.java.opts</name>
            <value>-server -Xmx3276m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value>
      </property>
      <property>
            <name>mapreduce.reduce.memory.mb</name>
            <value>4096</value>
      </property>
      <property>
            <name>mapreduce.reduce.java.opts</name>
            <value>-server -Xmx3276m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value>
      </property>
      <property>
            <name>mapreduce.reduce.shuffle.input.buffer.percent</name>
            <value>0.5</value>
      </property>
      <property>
            <name>mapreduce.task.io.sort.mb</name>
            <value>600</value>
      </property>
      <property>
            <name>mapreduce.task.io.sort.factor</name>
            <value>1638</value>
      </property>
      <property>
            <name>mapreduce.map.sort.spill.percent</name>
            <value>0.50</value>
      </property>
      <property>
            <name>mapreduce.map.speculative</name>
            <value>false</value>
      </property>
      <property>
            <name>mapreduce.reduce.speculative</name>
            <value>false</value>
      </property>
      <property>
            <name>mapreduce.task.timeout</name>
            <value>1800000</value>
      </property>
</configuration>

yarn-env.sh

nano /usr/local/hadoop/etc/hadoop/yarn-env.sh

Change or uncomment or add the following:

JAVA_HEAP_MAX=Xmx2000m
HADOOP_OPTS=”$HADOOP_OPTS-server -Dhadoop.log.dir=$YARN_LOG_DIR”
HADOOP_OPTS=”$HADOOP_OPTS-Djava.net.preferIPv4Stack=true”

Master

Add the namenode hostname.

nano /usr/local/hadoop/etc/hadoop/masters

APPLY THE FOLLOWING TO THE NAMENODE ONLY

Slaves

Add namenode hostname and all datanodes hostname.

nano /usr/local/hadoop/etc/hadoop/slaves

hdfs-site.xml

Click here to view the docs. By default it will look like . This will need to be changed. The /usr/local/hadoop/etc/hadoop/hdfs-site.xml file needs to be configured for each host in the cluster that is being used. Before editing this file, we need to create the namenode directory.

mkdir -p /usr/local/hadoop_store/data/namenode
chown -R hduser:hduser /usr/local/hadoop_store
nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
      <property>
            <name>dfs.replication</name>
            <value>3</value>
            <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.</description>
      </property>
      <property>
            <name>dfs.permissions</name>
            <value>false</value>
      </property>
      <property>
            <name>dfs.namenode.name.dir</name>
            <value>file:/usr/local/hadoop_store/data/namenode</value>
      </property>
      <property>
            <name>dfs.datanode.use.datanode.hostname</name>
            <value>false</value>
      </property>
      <property>
            <name>dfs.blocksize</name>
            <value>128m</value>
      </property>
      <property>
            <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
            <value>false</value>
      </property>
      
      <!-- URL -->
      <property>
            <name>dfs.namenode.http-address</name>
            <value>${dfs.namenode.http-bind-host}:50070</value>
            <description>Your NameNode hostname for http access.</description>
      </property>
      <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>${dfs.namenode.http-bind-host}:50090</value>
            <description>Your Secondary NameNode hostname for http access.</description>
      </property>
      <property>
            <name>dfs.datanode.http.address</name>
            <value>${dfs.namenode.http-bind-host}:50075</value>
      </property>
      <property>
            <name>dfs.datanode.address</name>
            <value>${dfs.namenode.http-bind-host}:50076</value>
      </property>
      <property>
            <name>dfs.namenode.http-bind-host</name>
            <value>0.0.0.0</value>
      </property>
      <property>
            <name>dfs.namenode.rpc-bind-host</name>
            <value>0.0.0.0</value>
      </property>
      <property>
             <name>dfs.namenode.servicerpc-bind-host</name>
             <value>0.0.0.0</value>
      </property>
&lt;/configuration>

APPLY THE FOLLOWING TO THE DATANODE(s) ONLY

Slaves

Add only that datanodes hostname.

nano /usr/local/hadoop/etc/hadoop/slaves

hdfs-site.xml

The /usr/local/hadoop/etc/hadoop/hdfs-site.xml file needs to be configured for each host in the cluster that is being used. Before editing this file, we need to create the datanode directory.
By default it will look like . This will need to be changed.

mkdir -p /usr/local/hadoop_store/data/datanode
chown -R hduser:hduser /usr/local/hadoop_store
nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
      <property>
            <name>dfs.replication</name>
            <value>3</value>
            <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.</description>
      </property>
      <property>
            <name>dfs.permissions</name>
            <value>false</value>
      </property>
      <property>
            <name>dfs.blocksize</name>
            <value>128m</value>
      </property>
      <property>
            <name>dfs.datanode.data.dir</name>
            <value>file:/usr/local/hadoop_store/data/datanode</value>
      </property>
      <property>
            <name>dfs.datanode.use.datanode.hostname</name>
            <value>false</value>
      </property>
      <property>
            <name>dfs.namenode.http-address</name>
            <value>${dfs.namenode.http-bind-host}:50070</value>
            <description>Your NameNode hostname for http access.</description>
      </property>
      <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>${dfs.namenode.http-bind-host}:50090</value>
            <description>Your Secondary NameNode hostname for http access.</description>
      </property>
      <property>
            <name>dfs.datanode.http.address</name>
            <value>${dfs.namenode.http-bind-host}:50075</value>
      </property>
      <property>
            <name>dfs.datanode.address</name>
            <value>${dfs.namenode.http-bind-host}:50076</value>
      </property>
      <property>
            <name>dfs.namenode.http-bind-host</name>
            <value>0.0.0.0</value>
      </property>
      <property>
            <name>dfs.namenode.rpc-bind-host</name>
            <value>0.0.0.0</value>
      </property>
      <property>
             <name>dfs.namenode.servicerpc-bind-host</name>
             <value>0.0.0.0</value>
      </property>
</configuration>

You need to allow the pass-through for all ports necessary. If you have the Ubuntu firewall on.

sudo ufw allow 50070
sudo ufw allow 8088

Format Cluster:
Only do this if NO data is present. All data will be destroyed when the following is done.
This is to be done on NAMENODE ONLY!

hdfs namenode -format

Start The Cluster:
You can now start the cluster.
You do this from the NAMENODE ONLY.

start-dfs.sh
start-yarn.sh
mapred --config $HADOOP_CONF_DIR --daemon start historyserver

If the above three commands didn’t work something went wrong. As it should have found the scripts located /usr/local/hadoop/sbin/ directory.

Cron Job:
You should probably setup a cron job to start the cluster when you reboot.

crontab –e

@reboot /usr/local/hadoop/sbin/start-dfs.sh > /home/hduser/dfs-start.log 2>&1
@reboot /usr/local/hadoop/sbin/start-yarn.sh > /home/hduser/yarn-start.log 2>&1
@reboot /usr/local/hadoop/bin/mapred –config $HADOOP_CONF_DIR –daemon start historyserver > /home/hduser/history-stop.log 2>&1

Verification:
To check that everything is working as it should run “jps” on the NAMENODE. It should return something like the following where the pid will be different:

jps

You could also run “netstat -plten | grep java” or “lsof –i :50070” and “lsof –i :8088”.

Picked up _JAVA_OPTIONS: -Xms3g -Xmx10g -Djava.net.preferIPv4Stack=true
12007 SecondaryNameNode
13090 Jps
12796 JobHistoryServer
12261 ResourceManager
11653 NameNode
12397 NodeManager
11792 DataNode

You can check the DATA NODES by ssh into each one and running “jps”. It should return something like the following where the pid will be different:

Picked up _JAVA_OPTIONS: -Xms3g -Xmx10g -Djava.net.preferIPv4Stack=true
3218 Jps
2215 NodeManager
2411 DataNode

If for any reason only of the services is not running you need to review the logs. They can be found at /usr/local/hadoop/logs/. If it’s ResourceManager that isn’t running then look at file that has “yarn” and “resourcemanager” in it.

WARNING:
Never reboot the system without first stopping the cluster. When the cluster shuts down it is safe to reboot it. Also if you configured a cronjob @reboot you should make sure the DATANODES are up and running first before starting the NAMENODE that way it automatically starts the DATANODES for you

Web Ports:

NameNode

50070: HDFS Namenode
50075: HDFS Datanode
50090: HDFS Secondary Namenode
8088: Resource Manager
19888: Job History

DataNode

50075: HDFS Datanode

NetStat

To check that all the Hadoop ports are available on which IP run the following.

sudo netstat -ltnp

Port Check

If for some reason you are having issues connecting to a Hadoop port then run the following command as you try and connect via the port.

sudo tcpdump -n -tttt -i eth1 port 50070

References

I used a lot of different resources and reference material on this. However I did not save all the relevant links I used. Below are just a few I used. There was various blog posts about memory utilization, etc.

Shell: Misc Commands

In this tutorial I will show a few useful commands when working with linux shell.

Check Directory Exists:

if [ -d /opt/test/ ]; then
    echo 'Directory Exists'
fi

Check Directory Not Exists:

if [ ! -d /opt/test/ ]; then
    echo 'Directory Does Not Exist'
fi

Check File Exists:

if [ -f /opt/test/test.log ]; then
    echo 'File Exists'
fi

Check File Not Exists:

if [ ! -f /opt/test/test.log ]; then
    echo 'File Does Not Exist'
fi

Lowercase Variable:

val='TEXT'
echo "${val,,}"

Echo Variable:
This will print the value of “test”. Notice we use double quotes.

test='My Test Val'
echo "$test"

Echo String:

echo 'My test string'

Split:
This will split on the comma into an array list and then loop through it.

test='test 1,test 2'
split_test=(${test//,/ })
 
for val in "${split_test[@]}"
do
    echo $val
done

Date:
This will print the date in the format YYYY-MM-dd

my_date="$(date +Y-%m-%d)"
echo "$my_date"

Remove Space From Variable:

VAL='test string'
echo "${VAL//\ /}"

Increment Variable:

index=0
index=$((index+1))

Substring

VAL='test string'
echo "${VAL:4:4}"

If value is equal to

VAL='hello'
if [ "$VAL" == 'hello' ] ; then
    echo 'Is Equal'
fi

If with OR

VAL='hello'
if [ "$VAL" == 'hello' ] || [ "$VAL" != 'hi' ] ; then
    echo 'Is Hello'
fi

If Variable is Empty

VAL=''
if [ -z "$VAL" ] ; then
    echo 'Is Empty'
fi

Append to File

echo 'Hi' >> file_to_log_to.log

Write to File

echo 'Hi' > file_to_log_to.log

While Loop: Increment to 10

This will loop till the value is 9 then exit.

i=0
while [ $i -lt 10 ];
do
    echo "$i"
done

whoami

USER=$(whoami)

If Variable Contains Text

VAL='my Test String'
if [[ "${VAL,,}" == *"test"* ]] ; then
    echo "Found test"
fi

Color Coding

NoColor=$'\033[0m'
READ=$'\033[0;31m'
GREEN=$'\033[0;32m'
YELLOW=$'\033[1;33;40m'
 
printf "%s Variable Not Set %s\n" "${RED}" "${NoColor}"

Get Log to a logfile and console

SOME_COMMAND 2>&1 | tee -a "${LOG_FILE_PATH}"

Read a JSON config

JSON=$(cat "/path/to/json/file.json")
export MY_VAR=$(echo "${JSON}" | python -c 'import json,sys;obj=json.load(sys.stdin);print(obj["MyKey"])')

Extract tar to Folder

sudo tar -xvf /the/location/file.tar -C /to/location/ --force-local --no-same-owner

Update Certificates

This will update certificates. After you put a certificate in /usr/local/share/ca-certificates/

update-ca-certificates

PipeStatus

somecommand
RETURN_CODE=${PIPESTATUS[0]}

Hive: Struct

This tutorial will show you how to use struct. If you have no installed Hive yet please follow this tutorial.

Create Table with Struct:

CREATE TABLE test_struct (
    columnA STRING,
    columnB VARCHAR(15),
    columnC INT,
    columnD TIMESTAMP,
    columnE DATE,
    columnF STRUCT<key:STRING, value:INT>
)
STORED AS ORC;

Insert Data:

INSERT INTO test_struct 
SELECT '1', '2', 1, '2019-02-07 20:58:27', '2019-02-07', NAMED_STRUCT('key', 'val', 'value', 1);

Select Data:
This will give back the value of “key” and “value” in columnF.

SELECT columnF.key, columnF.value
FROM test_struct;

Hive: Map

This tutorial will show you how to use map. If you have no installed Hive yet please follow this tutorial.

Create Table with Map:

CREATE TABLE test_map (
    columnA STRING,
    columnB VARCHAR(15),
    columnC INT,
    columnD TIMESTAMP,
    columnE DATE,
    columnF MAP<STRING, INT>
)
STORED AS ORC;

Insert Data:

INSERT INTO test_map 
SELECT '1', '2', 1, '2019-02-07 20:58:27', '2019-02-07', MAP('Val', 1);

Select Data:
This will give back the value of “Val” in columnF.

SELECT columnF['Val']
FROM test_map;