CSS: FlexBox

I wanted to take a few minutes and just do a basic run through of the common uses of FlexBox. Just from the 30 minutes of doing a run through I found it to be extremely useful and easily configured.

For this example I will use the following html structure.

<div className="div-flex">
	<div>1</div>
	<div>2</div>
	<div>3</div>
	<div>4</div>
	<div>5</div>
	<div>6</div>
	<div>7</div>
	<div>8</div>
	<div>9</div>
	<div>10</div>
</div>

If you want them to display like the following “12345678910” you style like the following: They all become flex items.

display: inline-flex;
flex-direction:

To make it look like this “12345678910”.

flex-direction: row;

To make it look like this “10987654321”.

flex-direction: row-reverse;

To put it from 1 to 10 below each other.

flex-direction: column;

To put it from 10 to 1 below each other.

flex-direction: column-reverse;
flex-wrap:

If you set nowrap then the flex items will not wrap to the next line.

flex-wrap: nowrap;

If you want the divs to wrap to the next line use wrap. You will notice how it wraps to the next line when it runs out of space.

flex-wrap: wrap;

If you want the divs to wrap to the next line but in reverse. You will notice how it wraps to the next line when it runs out of space. But more than that you will notice that it started at 10 instead of 1.

flex-wrap: wrap-reverse;
flex-flow:

Is used to combine flex-direction and flex-wrap. See below an example. What you will see is that 1 will be on the right and as the text wraps it will wrap to the right side not left.

flex-flow: row-reverse wrap;
justify-content:

You can use this to center justify content. The text will always be center.

justify-content: center;

You can use this to left justify content. The text will always be to the left. Unless you use one of the reverse options in flex-direction or flex-flow.

justify-content: flex-start;

You can use this to right justify content. The text will always be to the right. Unless you use one of the reverse options in flex-direction or flex-flow.

justify-content: flex-end;

If you want to set space between each element to use the width of the page or container element. Notice you have to set the width otherwise it doesn’t seem to work.

width: 100%;
display: inline-flex;
flex-direction: row;
flex-wrap: wrap;
justify-content: space-between;

If you want to set space around each element to use the width/height of the page or container element. Notice you have to set the width/height otherwise it doesn’t seem to work.

width: 100%;
height: 100%;
display: inline-flex;
flex-direction: row;
flex-wrap: wrap;
justify-content: space-around;
Webkit:

If you want your flex code to work cross-browsers here are some of the commands using webkit and ms.

display: flex;
display: -ms-flexbox;
display: -webkit-flex;
-webkit-flex-direction: column;
flex-direction: column;
-webkit-flex-wrap: wrap;
flex-wrap: wrap;
-webkit-flex: 1;
-moz-flex: 1;
-ms-flex: 1;
flex: 1;

React Add CSS to Your Site

If you haven’t already done so please follow this tutorial in setting up your React/Python site.

Folder Structure:
  • You need to add file(s) to your existing structure.
  • Inside testApp create the following:
    • folder: app
      • folder: home
      • folder: css
        • file: home.css
NPM:

We need to add style-loader and css-loader to our application.

npm install style-loader --save
npm install css-loader --save
npm install create-react-class --save
Webpack.config.js Setup:

Now that we have installed our loaders we need to setup webpack.config.js to use our new loaders. The below will go in the “loaders” array.

	{
		test: /\.css$/,
		loader: 'style-loader!css-loader'
	}

Things to note:

  • The ! in the loader section just means it applies to both.
  • The .css in the test area says any css file.
Home.jsx Modification:

We need to pull in our css file for home.jsx. Notice the “require” now in our home.jsx file and that we added a class to our div.

var React = require("react");
var ReactDOM = require("react-dom");
var createReactClass = require("create-react-class");

require("../css/home.css");

var Home = createReactClass({
	render: function() {
		return (<div className="div-hi">Hi</div>);
	}
});

ReactDOM.render(<Home />, document.getElementById("app"));
Home.css Setup:

Put anything you want in the css file as long as it matches the classname we set in home.jsx.

.div-hi {
	color: red;
}
Run:

We can now run the following to build and deploy our site. It will be available on http://localhost:5000/. Once you login you will be able to see our red “hi”. As long as you following the building a react python site creation.

webpack
flask run

Python: Run Module as Script

We can run a Python module as a script directly. In order to do that it checks that “__name__” is set to “__main__”. If it is then it executes the code in the if block. See below for example.

#imports here

def run():
	#run code here
	return

if __name__ == "__main__":
    run()

Simple Node WebApp

This whole tutorial describes how to create and run a simple website and call a singular js file. In this tutorial I am just demoing how to get a page displaying on localhost. Nothing more.

FYI: I am using Windows at the moment for this tutorial but you can substitute Ubuntu in if you so chose.

We will require nodejs before you continue which you can get from here.

We use Express in this project demo. Express is a web framework.

Folder Structure:
  • You will need to create a folder called “testApp”.
  • Inside testApp create the following:
    • app.js
Node Init:

We need to go to our root directory testApp from command prompt and run the following order.

npm init
npm install express --save

Things to Note:

  • npm init: Creates package.json file
  • npm install –save: will save the install to package.json
  • Each package install went into a new directory called “node_modules”.
Setup app.js:
var express = require('express');
var app = express()

app.use(express.logger('dev'))

app.get('/', function (req, res) {
	res.send("hi");
	res.end();
})
app.listen(5000)

Things To Note:

  • var app = express(): Setups up our express app
  • res.use: is intended for binding middleware to your application
  • res.get: is used for routing.
  • res.send: sends data
  • res.end:Ends the response call.
  • You get find all the API reference here.
Run It!
node app.js

Your site is now running on http://localhost:5000. Notice all it says is “HI”. That is because this is just a demo to show you how to get going. Expand as needed.

Ubuntu: tmux

Tmux (Terminal Multiplexer) is really helpful for running many virtual terminals that stay around even after you exit out of the server.

Some quick cheats are:

  • tmux -V: What version you are running.
  • CTRL b %: Split terminal horizontally.
  • CTRL b “: Split terminal vertically.
  • CTRL b LEFT: Switch between panes to the left.
  • CTRL b RIGHT: Switch between panes to the right.
  • CTRL b DOWN: Switch between panes to the down.
  • CTRL b UP: Switch between panes to the up.
  • CTRL b z: Toggle a pane.
  • CTRL b x: Kill the pane you are on.
  • CTRL b c: Create a session window.
  • CTRL b NUMBER: Enter 1, 0, 2, etc and you can switch to that session.
  • CTRL b d: Detach from tmux keeping it running.
  • tmux ls: Lists all sessions currently running.
  • tmux attach -t 0: Attach to the session by number. If you have two you can select 1 instead of 0.
  • tmux kill-session -t 0: This kills a session by session number.

Flask: React Website

This whole tutorial describes in depth how to create a React website with Python. You must have Eclipse installed before you continue. If you have it already installed and configured you can continue on. Note that you should probably have HTML Editor and TypeScript IDE installed for Eclipse.

FYI: I am using Windows at the moment for this tutorial but you can substitute Ubuntu in if you so chose.

We will require Postgres 9.4, nodejs before you continue. You can get Nodejs from here. You can get Postgres 9.4 from here.

In this tutorial we use Flask. If you are not familiar with it go here. Flask is a lightweight Python web framework based on Werkzeug and Jinja 2.

Folder Structure:
  • You will need to create a folder called “testApp”.
  • Inside testApp create the following:
    • folder: app
      • file: models.py
      • file: __init__.py
      • folder: auth
        • file: __init__.py
        • file: views.py
      • folder: home
        • file: __init__.py
        • file: views.py
        • folder: js
          • file: home.jsx
      • folder: templates
        • file: base.html
        • file: login.html
        • file: register.html
    • file: config.py
    • file: requirements.txt
    • file: webpack.config.js
    • file: run.py
    • folder: instance
      • file: config.py
    • folder: static
      • folder: common
        • folder: css
          • file: base.css
      • file: manifest.json
    • .babelrc
Install Python Packages:

Open cmd/terminal and navigate into your testApp folder and run the following commands.

pip install flask-sqlalchemy && pip freeze > requirements.txt
pip install flask-login && pip freeze > requirements.txt
pip install flask-migrate && pip freeze > requirements.txt
pip install psycopg2 && pip freeze > requirements.txt
pip install flask-Webpack && pip freeze > requirements.txt
pip install Flask-WTF && pip freeze > requirements.txt
pip install flask-bootstrap && pip freeze > requirements.txt

Couple things to note:

  • The “&& pip freeze > requirements.txt” saves the install in requirements.txt.
  • flask-migrate: database migrations package
  • flask-sqlalchemy: model engine for your database.
  • flask-login: provides user session management for Flask
  • psycopg2: Postgres driver
  • flask-Webpack: A Flask extension to manage assets with Webpack
  • flask-WTF: flask form validation
  • flask-bootstrap: An extension that includes Bootstrap in your project, without any boilerplate code
Routing:

For the purpose of this example we are using the basic Flask implementation. IE: @home.route(‘/’). However if you want to do a more advanced routing do Resource.

Database:

Open PgAdmin III and create yourself a database called “testApp”. Also create a user with password granting access to testApp database. Permission as you see fit. Don’t forget to write down your password :).

Setup Config.py:

Read here for SqlAlchemy configruation options. DEBUG is for flask debugging.

class Config(object):
    """
    This is the shared common configurations
    """

    # Put any configurations here that are common across all environments

class DevelopmentConfig(Config):
    """
    This is the development configurations
    """
    
    DEBUG = True
    SQLALCHEMY_ECHO = True #Display queries to console

class ProductionConfig(Config):
    """
    This is the production configurations
    """
    DEBUG = False
    SQLALCHEMY_ECHO = False #Do not Display queries to console

app_config = {
    'development': DevelopmentConfig,
    'production': ProductionConfig
}

Things to Note:

Notice how we don’t have any database connections in this config.py. That is because we really shouldn’t checkin to source control our database connection strings, etc.

Setup instance config.py:

We open the file config.py inside our “instance” folder and add this line only.

SQLALCHEMY_DATABASE_URI = "postgresql://##USER##:##PASSWORD##@##HOST##/testApp"
Setup __init__.py in “app” folder:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_login import LoginManager
from flask_migrate import Migrate
from flask_bootstrap import Bootstrap
import os

#Importing from the config.py
from config import app_config

# db variable initialization
db = SQLAlchemy()

#The flask login manager
login_manager = LoginManager()

webpack = Webpack()

def create_app(config_name):
    #This will be either "development" or "production" mapped to what we write in the config.py application
    #static_folder is where the static folder will be located
    app = Flask(__name__, instance_relative_config=True, static_folder=os.path.join(os.getcwd(), "static"))
    print('Running in %s' % (config_name))
    app.config.from_object(app_config[config_name])
    app.config.from_pyfile('config.py')
    #You need a secret key to be able to utilise the database connection
    app.secret_key = 'Awesome App'
    Bootstrap(app)
    db.init_app(app)
    #This will make it so our chunked js files are able to be loaded on the template
    app.config.update({'WEBPACK_MANIFEST_PATH': '../manifest.json'})
    webpack.init_app(app)

    #if a user tries to access a page that they are not authorized to, it will redirect to the specified view and display the specified message.
    login_manager.init_app(app)
    login_manager.login_message = "You must be logged in to access this page."
    #auth.login is not the route but actually the class path.
    login_manager.login_view = "auth.login"
    
    #This let's us do our migrations
    migrate = Migrate(app, db)

    #Bring in our new tables
    from app import models

    #Our blueprints for our app

    #This is how you get authenticated
    from .auth import auth as auth_blueprint
    app.register_blueprint(auth_blueprint)
	
    #Bring in the home module
    from .home import home as home_blueprint
    app.register_blueprint(home_blueprint)

    return app
Environment Variable Setup:

Flask has two environment variables that you can set which point to the environment to use and the run.py file. These are FLASK_CONFIG and FLASK_APP. I set my environment to “development” and the app to use “run.py”. Notice that “development” maps to the key value pair in config.py.

Setup run.py:

Notice how we utilise the FLASK_CONFIG from the environment variables to setup our environment and grab the right config class.

import os #We need this to get the OS ENV VARIABLE 'FLASK_CONFIG'

#You are going to import the create_app from the __init__.py file
from app import create_app

#In our environment variables we create "FLASK_CONFIG" and set our value either development or production
config_name = os.getenv('FLASK_CONFIG')
app = create_app(config_name)

if __name__ == '__main__':
    app.run()

Now the fun really starts!!!

Setup models.py:

We setup our User model ensuring security of our password. Later on I will show you what happens with flask-migrate.

from flask_login import UserMixin
from werkzeug.security import generate_password_hash, check_password_hash

from app import db

class User(UserMixin, db.Model):
    """
    Create an Users table
    """

    # Ensures table will be named in plural and not in singular
    # as is the name of the model
    __tablename__ = 'users'

    id = db.Column(db.Integer, primary_key=True)
    email = db.Column(db.String(60), index=True, unique=True)
    username = db.Column(db.String(60), index=True, unique=True)
    first_name = db.Column(db.String(60), index=True)
    last_name = db.Column(db.String(60), index=True)
    password_hash = db.Column(db.String(128))

    @property
    def password(self):
        """
        Prevent pasword from being accessed
        """
        raise AttributeError('password is not a readable attribute.')

    @password.setter
    def password(self, password):
        """
        Set password to a hashed password
        """
        self.password_hash = generate_password_hash(password)

    def verify_password(self, password):
        """
        Check if hashed password matches actual password
        """
        return check_password_hash(self.password_hash, password)

    def __repr__(self):
        return ''.format(self.username)
Generate User Table:

Using flask-migrate we can now auto generate our User table into Postgres testApp database.

  1. Run “flask db init” to create our migration repository
    1. A “migrations” folder is created automatically.
  2. Run “flask db migrate”
    1. This generates the migration script
  3. Run “flask db upgrade”
    1. This creates the missing migrations into the database. AKA the users table.
    2. You will notice another table also got created “alembic_version”. This is how it stores the version it is at.
  4. Confirm that the db was migrated successfully.
Create Our Templates:

We use the base.html file for our react pages using the “app” id on the div. The login.html is for our login form and register.html is to register a new user. You can use the base.css file in the static/common/css folder to style it how you want. I recommend using flex.

base.html:

<!DOCTYPE html>
{% import "bootstrap/utils.html" as utils %}
{% import "bootstrap/wtf.html" as wtf %}
{% extends "bootstrap/base.html" %}
<html>
	<head>
		{% block head %}
		{{ super() }}
		<title>{{title}}</title>
		<link rel="stylesheet" type="text/css" href={{ url_for("static", filename="common/css/base.css") }} />
		{% endblock %}
	</head>
	{% block content %}
	<div class="container-app">
		<div class="container-header">Team Link</div>
	    <div class="container-body" id="app"></div>
	    <script type="text/javascript" src={{ asset_url_for(script_name) }}></script>
	</div>
	{% endblock %}
</html>

login.html:

<!DOCTYPE html>
{% import "bootstrap/utils.html" as utils %}
{% import "bootstrap/wtf.html" as wtf %}
{% extends "bootstrap/base.html" %}
<html>
	<head>
		{% block head %}
		{{ super() }}
		<title>{{title}}</title>
		<link rel="stylesheet" type="text/css" href={{ url_for("static", filename="common/css/base.css") }} />
		{% endblock %}
	</head>
	{% block content %}
	<div class="container-app">
		<div class="container-header">My Awesome App</div>
	    <div class="container-body" id="app">
			<div class="panel panel-default">
				<div class="panel-heading">
					<h3 class="panel-title">Login</h3>
				</div>
				<div class="panel-body">
					{{ wtf.quick_form(form) }}
				</div>	    
				{{ utils.flashed_messages() }}
				Click here to <a href="/register">register</a>
			</div>
		</div>
	</div>
	{% endblock %}
</html>

register.html

<!DOCTYPE html>
{% import "bootstrap/utils.html" as utils %}
{% import "bootstrap/wtf.html" as wtf %}
{% extends "bootstrap/base.html" %}
<html>
	<head>
		{% block head %}
		{{ super() }}
		<title>{{title}}</title>
		<link rel="stylesheet" type="text/css" href={{ url_for("static", filename="common/css/base.css") }} />
		{% endblock %}
	</head>
	{% block content %}
	<div class="container-app">
		<div class="container-header">Team Link</div>
	    <div class="container-body" id="app">
			<div class="panel panel-default">
				<div class="panel-heading">
					<h3 class="panel-title">Register</h3>
				</div>
				<div class="panel-body">
    				{{ wtf.quick_form(form) }}
				</div>	    
				{{ utils.flashed_messages() }}
				Click here to <a href="login">login</a>
			</div>
		</div>
	</div>
	{% endblock %}
</html>
Setup home __init__.py:

This creates the blueprint that we have in app.__init__.py.

from flask import Blueprint

home = Blueprint('home', __name__)

#This is the views.py from the home directory.
from . import views
Setup home views.py:

@login_required let’s flask know that you need to be logged in to get to this page. Don’t forget to see “render_template” method. How it has “script_name” in it and it uses base.html template. “script_name” was utilised in base.html. It brings in our js file for us on each page we go to.

from flask import render_template
from flask_login import login_required

#This is our blueprint we setup in __init__.py
from . import home

@home.route('/')
@login_required
def homepage():
    """
    Render the homepage template on the / route
    """
    return render_template('base.html', script_name='home.js', title="Welcome")
Setup React home JSX file:

React uses jsx files. So in my home directory I have a js folder and inside that we have our home.jsx file. Let’s set that up to something really basic. Remember above I said in the “render_template” we use the div id “app”. The ReactDOM will put our class in that spot. I will show you later how that is done.

var React = require("react");
var ReactDOM = require("react-dom");
var createReactClass = require("create-react-class");

var Home = createReactClass({
	render: function() {
		return (<div>Hi</div>);
	}
});

ReactDOM.render(<Home />, document.getElementById("app"));
Node Init:

We need to go to our root directory testApp from command prompt and run the following in the exact order.

npm init
npm install react react-dom --save
npm install webpack webpack-dev-server --save
npm install --save-dev babel babel-core babel-loader babel-preset-es2015 babel-preset-react
npm install create-react-class --save
npm install bootstrap --save
npm install jquery --save
npm install clean-webpack-plugin --save-dev
npm install manifest-revision-webpack-plugin --save-dev
npm install sync-exec --save-dev
npm install uglifyjs-webpack-plugin --save-dev

Things to Note:

  • npm init: Creates package.json file
  • npm install –save: will save the install to package.json
  • Each package install went into a new directory called “node_modules”.
Setup webpack.config.js:

We want to transition our jsx files to .js. Remember back in app.__init__.py we setup the static folder location. Checkout the “path” key below and now we know how it knows where it’s assets are going to be located. Our entry key value pair is the locations of each of our jsx files to create assets from. Then we have our loaders.

var path = require("path");
var webpack = require('webpack');
var ManifestRevisionPlugin = require('manifest-revision-webpack-plugin');
const CleanWebpackPlugin = require('clean-webpack-plugin');

module.exports = {
	entry: {
		"home": [path.resolve(__dirname, "./app/home/js/home.jsx")],
	},
	output: {
        path: path.join(__dirname, "static"),
		publicPath: "/static/",
		filename: "[name]-[hash].js"
	},
	plugins: [
                new CleanWebpackPlugin(["static/*.js", "static/manifest.json"], {root: __dirname, verbose: true, dry: false, exclude: ["base.css"]}),
		new ManifestRevisionPlugin(path.resolve(__dirname, "./manifest.json"), {rootAssetPath: './static', ignorePaths: ['./node_modules']}),
		new webpack.NoEmitOnErrorsPlugin(),
		new UglifyJsPlugin(),
		new webpack.optimize.AggressiveMergingPlugin(),
		new webpack.HotModuleReplacementPlugin()
	],
	module: {
		loaders: [
			{
				test: /\.jsx?$/,
				exclude: /(node_modules)/,
				loader: 'babel-loader',
				query: {
					presets: ['react','es2015']
				}
			}
		]
	}
};
.babelrc

Set the following in your file at the root directory.

{
 "presets": [ "es2015", "react" ]
}
Let’s Test Node & Webpack:

Open command prompt and navigate to our testApp folder and run “webpack”. You will notice that a “static” folder is created in our root directory. In it we will now see “home.js” file. Remember back above we set __init__.py static folder and in home.views.py file we said in render_template script_name “home.js”. This is how it all maps together.

Setup auth __init__.py:

This creates the blueprint that we have in app.__init__.py.

from flask import Blueprint

auth = Blueprint('auth', __name__)

#This is the views.py from the auth directory.
from . import views

Setup auth views.py:

from flask import flash, redirect, render_template, url_for
from flask_login import login_required, login_user, logout_user
from flask_wtf import FlaskForm
from wtforms import PasswordField, StringField, SubmitField, ValidationError
from wtforms.validators import DataRequired, Email, EqualTo
from .. import db, login_manager
from ..models import User

from . import auth

class RegistrationForm(FlaskForm):
    """
    Form for users to create new account
    """
    email = StringField('Email', validators=[DataRequired(), Email()])
    username = StringField('Username', validators=[DataRequired()])
    first_name = StringField('First Name', validators=[DataRequired()])
    last_name = StringField('Last Name', validators=[DataRequired()])
    password = PasswordField('Password', validators=[
                                        DataRequired(),
                                        EqualTo('confirm_password')
                                        ])
    confirm_password = PasswordField('Confirm Password')
    submit = SubmitField('Register')

    def validate_email(self, field):
        if User.query.filter_by(email=field.data).first():
            raise ValidationError('Email is already in use.')

    def validate_username(self, field):
        if User.query.filter_by(username=field.data).first():
            raise ValidationError('Username is already in use.')

class LoginForm(FlaskForm):
    """
    Form for users to login
    """
    email = StringField('Email', validators=[DataRequired(), Email()])
    password = PasswordField('Password', validators=[DataRequired()])
    submit = SubmitField('Login')

@login_manager.user_loader
def load_user(id):
    #This is the how we locate the user in our testApp database
    return User.query.get(int(id))

@auth.route('/register', methods=['GET', 'POST'])
def register():
    form = RegistrationForm()
    if form.validate_on_submit():
        user = User(email=form.email.data,
                            username=form.username.data,
                            first_name=form.first_name.data,
                            last_name=form.last_name.data,
                            password=form.password.data)

        # add user to the database
        db.session.add(user)
        db.session.commit()
        flash('You have successfully registered! You may now login.')

        # redirect to the login page
        return redirect(url_for('auth.login'))

    # load registration template
    return render_template('register.html', form=form, title='Register')

@auth.route('/login', methods=['GET', 'POST'])
def login():
    form = LoginForm()
    if form.validate_on_submit():
        # check whether user exists in the database and whether
        # the password entered matches the password in the database
        user = User.query.filter_by(email=form.email.data).first()
        if user is not None and user.verify_password(
                form.password.data):
            # log user in
            login_user(user)

            # redirect to the dashboard page after login
            return redirect("/")

        # when login details are incorrect
        else:
            flash('Invalid email or password.')

    # load login template
    return render_template('login.html', form=form, title='Login')

@auth.route('/logout')
@login_required
def logout():
    """
    Handle requests to the /logout route
    Log an user out through the logout link
    """
    logout_user()
    flash('You have successfully been logged out.')

    # redirect to the login page
    return redirect(url_for('auth.login'))
Let’s Run our App:

Open command prompt navigate to our testApp folder and run “flask run”. If no mistakes were made you should now be able to navigate to our site. http://localhost:5000.

 

Resources:

In learning how to incorporate flask into python I used the following part one only tutorial as a guide. Very helpful.

Python IDE Installation for Eclipse

This tutorial will guide you through configuring Eclipse for Python. Ensure you have followed the tutorial on installing Eclipse first.

You need to install PyDev. Open Eclipse. Click Help–>Install New Software. In the “work with” put “http://pydev.org/updates” and click Add. Follow the prompts and you are done for now.

You also need to ensure you have installed Python 3.6. You can get it from here. You should add the environment variable “PYTHON_HOME” and enter the python directory. I would also add “%PYTHON_HOME%\;%PYTHON_HOME%\Scripts” to your “path” environment variable.

I would also at this time install PIP. Just to make sure you have everything you will need. You will need to download “get-pip.py“. You will need to put it into the “Scripts” folder where you install Python 3.6. Navigate into your Scripts folder and then run “python get-pip.py”.

Once PyDev is installed and you have restarted Eclipse. Open PyDev perspective. Go to Window–>Perspective–>Open Perspective–>Other. Then select PyDev and click Ok.

Optional:

TypeScript IDE:

Used for React. It’s really handy. Open Eclipse. Click Help–>Install New Software. In the “work with” put “http://oss.opensagres.fr/typescript.ide/1.1.0/” and click Add. Follow the prompts and you are done for now. make sure you select “Embed Node.js” and “TypeScript IDE”.

HTML Editor:

Used for HTML files. Click Help –> Eclipse Marketplace. Search for “HTML Editor”. Then click install. After it is installed and Eclipse is restarted Click Window –> Preferences –> Web. Under “HTML Files” change encoding to “ISO 10646/Unicode(UTF-8). Under “Editor” add “div”. You can get more info and configuration from here.

Python: Misc

This post will just be for misc basic python stuff.

Let’s say you want to raise 10 to the power of 3.

10 ** 3

If you want to check if a variable is of a specific type. You can use “isinstance”:

#Check if it is int
isinstance(variable, int)
#Check if it is  string
isinstance(variable, basestring)

If you want to test that a dictionary is not empty

bool(my_dict)

Check if a file exists

import os
os.path.isfile(path)

Recursively loop through a directory to get the files with path to a list

import os

x = [os.path.join(r,file) for r,d,f in os.walk("E:") for file in f]

for val in x:
    print(val)

Python: Connect To Hadoop

We can connect to Hadoop from Python using PyWebhdfs package. For the purposes of this post we will use version 0.4.1. You can see all API’s from here.

To build a connection to Hadoop you first need to import it.

from pywebhdfs.webhdfs import PyWebHdfsClient

Then you build the connection like this.

HDFS_CONNECTION = PyWebHdfsClient(host=##HOST## port='50070', user_name=##USER##)

To list the contents of a directory you do this.

HDFS_CONNECTION.list_dir(##HADOOP_DIR##)

To pull a single file down from Hadoop is straight forward. Notice how we have the “FileNotFound” brought in. That is important when pulling a file in. You don’t actually need it but “read_file” will raise that exception if it is not found. By default we should always include this.

from pywebhdfs.errors import FileNotFound

try:
	file_data = HDFS_CONNECTION.read_file(##FILENAME##)
except FileNotFound as e:
	print(e)
except Exception as e:
	print(e)

 

 

Java: Enums

Below is an example of how to create a enum in Java.

public enum MyEnum {
	VAL(0), NEXTVAL(1);

	private int value;

	private MyEnum(int value) {
		setValue(value);
	}

	public int getValue() {
		return value;
	}

	public void setValue(int value) {
		this.value = value;
	}
}

Java: Input/Output Streams

This page is to give some basic examples of how to convert OutputStreams to InputStreams and vise versa.

Maven:

<dependency>
	<groupId>org.logback-extensions</groupId>
	<artifactId>logback-ext-loggly</artifactId>
	<version>0.1.4</version>
</dependency>

To convert an InputStream to OutputStream we can do it using IoUtils.copy as demonstrated below.

import ch.qos.logback.ext.loggly.io.IoUtils;
import java.io.InputStream;

InputStream input = ##INPUTSTREAM##;

//Convert InputStream to OutputStream

try (FileOutputStream out = new FileOutputStream(file)) {
	IoUtils.copy(input, out);
} catch (final IOException e) {
}

To convert a ByteArrayOutputStream to a ByteArrayInputStream we can do it as demonstrated below.

final ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
//Convert to ByteArrayInputStream
final ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());

AWS: Java Kinesis Lambda Handler

This entry is part 2 of 5 in the series AWS & Java

If you want to write a Lambda for AWS in Java that connects to a Kinesis Stream. You need to have the handler.

Maven:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.11.109</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.7.1</version>
</dependency>

This is the method that AWS Lambda will call. It will look similar to the one below.

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.events.KinesisEvent;
import com.amazonaws.services.lambda.runtime.events.KinesisEvent.KinesisEventRecord;
import com.amazonaws.services.kinesis.model.Record;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;

public void kinesisRecordHandler(KinesisEvent kinesisEvent, Context context) {
	final String awsRequestId = context.getAwsRequestId();
	final int memoryLimitMb = context.getMemoryLimitInMB();
	final int remainingTimeInMillis = context.getRemainingTimeInMillis();

	for (final KinesisEventRecord kinesisRec : kinesisEvent.getRecords()) {
		final Record record = kinesisRec.getKinesis();

		//We get the kinesis data information
		final JsonNode recData = new ObjectMapper().readValue(record.getData().array(), JsonNode.class);

		final String bucketName = recData.get("bucket").asText();
		final String key = recData.get("key").asText();
	}
}

The thing to note when you setup you Lambda is how to setup the “Handler” field in the “Configuration” section on AWS. It is in the format “##PACKAGE##.##CLASS##::##METHOD##”.

AWS: Java S3 Lambda Handler

This entry is part 1 of 5 in the series AWS & Java

If you want to write a Lambda for AWS in Java that connects to S3. You need to have the handler.

Maven:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.11.109</version>
</dependency>

This is the method that AWS Lambda will call. It will look similar to the one below.

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.event.S3EventNotification.S3Entity;
import com.amazonaws.services.s3.event.S3EventNotification.S3EventNotificationRecord;

public void S3Handler(S3Event s3e, Context context) {
	final String awsRequestId =  context.getAwsRequestId();
	final int memoryLimitMb = context.getMemoryLimitInMB();
	final int remainingTimeInMillis = context.getRemainingTimeInMillis();

	for (final S3EventNotificationRecord s3Rec : s3e.getRecords()) {
		final S3Entity record = s3Rec.getS3();
		
		final String bucketName = record.getBucket().getName()
		final String key = record.getObject().getKey();
	}
}

The thing to note when you setup you Lambda is how to setup the “Handler” field in the “Configuration” section on AWS. It is in the format “##PACKAGE##.##CLASS##::##METHOD##”.

Java: Jackson Json

When you want to work with JSON the package of choice is Jackson. I find it so useful and easy to use.

Maven:

<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-core</artifactId>
	<version>2.7.1</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-annotations</artifactId>
	<version>2.7.1</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-databind</artifactId>
	<version>2.7.1</version>
</dependency>

There are so many things that we can do with Jackson. It’s actually very exciting.

Let’s start with converting a json string to ObjectNode.

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;

//We need an object mapper to help with the conversion
final ObjectMapper mapper = new ObjectMapper();
//Next we read the json string into ObjectNode
final ObjectNode node = (ObjectNode)mapper.readTree(jsonString);

//You can check if the objectnode has a key by
node.has("my_key")

//You can get the value by
node.get("my_key")

//To get the value as a string using 
.asText() at the end of the get

//To get as int
.asInt()

//To get as boolean
.asBoolean()

//To get as double
.asDouble()

//To get as Long
.asLong()

Next let’s convert a json string to JsonNode

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

//You need an object mapper to help with the conversion
final ObjectMapper mapper = new ObjectMapper();
//Next we read the json string into a jsonnode
final JsonNode node = mapper.readTree(jsonString);
//We can if you want to switch it to an object node fairly easily.
final ObjectNode objNode = (ObjectNode)node;

//Put value in the object node
objNode.put("my_key", 1);

//You can also set json node
objNode.set("my_key", myJsonNode);

You can also create a new object node by using JsonNodeFactory. See below.

import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.fasterxml.jackson.databind.node.ObjectNode;

//If you want to create an instance of ObjectNode
final ObjectNode objNode = JsonNodeFactory.instance.objectNode();

If you want to work with json array’s it’s also pretty straight forward. Again we work with JsonNodeFactory to declare it out.

import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;

private ArrayNode arrNode = JsonNodeFactory.instance.arrayNode();

//We can then add to it
arrNode.add();

Jackson also has generators which allow us to create json on the fly.

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonProcessingException;

//It throws a "JsonProcessingException'

final JsonFactory jfactory = new JsonFactory();
ByteArrayOutputStream out=new ByteArrayOutputStream();

try (final JsonGenerator jg = jfactory.createGenerator(out, JsonEncoding.UTF8)) {
	jg.setCodec(objectMapper);

	jg.writeStartObject();

	jg.writeFieldName("my_key");
	jg.writeString("hi");

	jg.writeFieldName("next_key");
	jg.writeObject(my_obj);

	jg.writeFieldName("third_key");
	jg.writeTree(my_obj_node);

	jg.writeEndObject();
}

Jackson JsonViews are really cool in my opinion. We can have an object that has multiple properties and we can set a view to each one of these properties. Which means when we serialize the object we will only get those fields that are associated to that view and vise versa on the deserialize.

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonView;

//You can however many views we want to work with in our objects. They must all be classes. 
//If you extend a class then it will contain all properties of that view and the view it extends.
public class MyClassViews {
	public static class ViewA {}
	public static class ViewB extends ViewA {}
}

//Let's create an object for our serialization and deserialization
public class MyClass {
	private final int id;
	private final String value;

	//This constructor is what the deserializer would use.
	@JsonCreator
	public MyClass(@JsonProperty("id") int id, @JsonProperty("value") String value) {
	}

	@JsonView(MyClassViews.ViewA.class)
	@JsonProperty("id") //You don't have to put the name you could leave it as @JsonProperty
	public int getId() {
		return id;
	}

	@JsonView(MyClassViews.ViewB.class)
	@JsonProperty("value")
	public String getValue() {
		return value;
	}
}

I didn’t come up with this next part but it’s so awesome that I wanted to include it. Let’s say you had an object that was populated not using any views but the object does contain views that will eventually pass up to a UI but you only want to pass using a specific view. Then you can do the following.

import javax.ws.rs.core.StreamingOutput;
import com.fasterxml.jackson.databind.ObjectMapper;

final StreamingOutput jsonStream = new StreamingOutput() {
	@Override
	public void write(OutputStream out) throws IOException {
		final ObjectMapper mapper = new ObjectMapper();
		mapper.writerWithView(MyClassViews.ViewB.class).writeValue(out, MyClass);
	}
};

If you want to write the object to string using a view. All you need to do is

final String jsonString = new ObjectMapper().writerWithView(ViewB).writeValueAsString(MyClass);

If you then want to convert our new jsonString to the object using the view do the following:

new ObjectMapper().readerWithView(ViewB.class).forType(MyClass.class).readValue(jsonString);

If you want to convert a json string to object.

new ObjectMapper().readValue(myJsonString, MyClass.class)

Postgres: Vacuum

You should set autovacuum on. If it is turned off (which is the default) it will require manual vacuuming and analyzing to be performed.

You can run vacuuming by:

VACUUM (VERBOSE, ANALYZE)

You can set auto vacuuming on by the below commands. You will need to modify the “postgresql.conf” file for this. Windows is located “C:\Program Files\PostgreSQL\9.4\data\postgresql.conf” and on Ubuntu it is “/etc/postgresql/9.4/main/postgresql.conf”. In the off chance that it isn’t then run the below first.

find / -type f -name "postgresql.conf"

Locate the following line of text “# AUTOVACUUM PARAMETERS” and apply the following. You should note that turning on automatically will also run “VACUUM ANALYZE” command to update statistics.

autovacuum = on (remove #)
autocavuum_analyze_threshold = 100
autovacuum_vacuum_threshold = 100
track_counts = on

Now you will need to restart postgresql service.

Ubuntu:

/etc/init.d/postgresql restart
invoke-rc.d postgresql restart

Windows:

services.msc restart service "postgresql-x64-9.4"

AWS: Node Kinesis Stream

This entry is part 2 of 2 in the series AWS & Node

If you haven’t already done so please refer to the AWS Node Setup tutorial as part of this series. In this tutorial we will just put something on the Kinesis queue.

We will utilise the AWS variable we created during the setup as part of this series.

First we need to create the variable that connects to our Kinesis in our region.

var kinesis = new AWS.Kinesis({region : ##REGION##});

Next we need to setup a record to send on the Kinesis stream. This will contain our data, key and the stream name.

var recordParams = {
	Data: ##DATA##,
	PartitionKey: ##FILENAME_OR_ID##,
	StreamName: ##STREAM_NAME##
};

Next we need to put the record onto the stream. This is a very basic implementation. Feel free to expand as you need to.

kinesis.putRecord(recordParams, function(err, data) {
	if (err) {
		console.error(err);
	}
	else {
		console.log("done");
	}
});

AWS: Node Setup

This entry is part 1 of 2 in the series AWS & Node

Using Node we can setup connections to AWS. As time goes on I will keep this section updated.

First we need to install the aws-sdk and save in our dependencies. This will make it show up in our package.json file.

npm install aws-sdk --save

Next we need to require the aws-sdk.

var AWS = require('aws-sdk')

Next we update the config to utilise our keys.

AWS.config.update({accessKeyId: ##ACCESS_ID##, secretAccessKey: ##SECRET_KEY##});

 

 

 

AWS: Python S3

This entry is part 3 of 3 in the series AWS & Python

If you haven’t already done so please refer to the AWS setup section which is part of this series. As time goes on I will continually update this section.

To work with S3 you need to utilise the “connection_s3” you setup already in the previous tutorial on setting up the connection.

To load a file from a S3 bucket you need to know the bucket name and the file name.

connection_s3.Object(##S3_BUCKET##, ##FILE_NAME##).load()

If you want to check if the check if a file exists on S3 you do something like the below. However you will need to import botocore.

import botocore

def keyExists(key):
    file = connection_s3.Object(##S3_BUCKET##, ##FILE_NAME##)
    
    try:
        file.load()
    except botocore.exceptions.ClientError as e:
        exists = False
    else:
        exists = True
    
    return exists, file

If you want to copy a file from one bucket to another or sub folder you can do it like below.

connection_s3.Object(##S3_DESTINATION_BUCKET##, ##FILE_NAME##).copy_from(CopySource=##S3_SOURCE_BUCKET## + '/' + ##FILE_NAME##)

If you want to delete the file you can use the “keyExists” function above and then just call “delete”.

##FILE##.delete()

If you want to just get a bucket object. Just need to specify what bucket and utilise the S3 connection.

bucket = connection_s3.Bucket(##S3_BUCKET##)

To upload a file to S3’s bucket. You need to set the body, type and name. Take a look at the below example.

bucket.put_object(Body=##DATA##,ContentType="application/zip", Key=##FILE_NAME##)

If you want to loop over the objects in a bucket. It’s pretty straight forward.

for key in bucket.objects.all():
	file_name = key.key
	response = key.get()
	data = response_get['Body'].read()

If you want to filter objects in a bucket.

for key in bucket.objects.filter(Prefix='##PREFIX##').all():
        file_name = key.key
	response = key.get()
	data = response_get['Body'].read()

AWS: Python Kinesis Streams

This entry is part 2 of 3 in the series AWS & Python

If you haven’t already done so please refer to the AWS setup section which is part of this series. As time goes on I will continually update this section.

To put something on the Kinesis Stream you need to utilise the “connection_kinesis” you setup already in the previous tutorial on setting up the connection. You will need to set the partition key, data and stream.

response = connection_kinesis.put_record(StreamName=##KINESIS_STREAM##, Data=##DATA##, PartitionKey=##FILE_NAME##)

Depending on your data you may need to utf8 encode. For example below.

bytearray(##MY_DATA##, 'utf8')

To read from the kinesis stream you need to setup the shard iterator then retrieve the data from the stream. Not forgetting to grab the new shard iterator from the returned records. Remember to not query against the queue to fast.

#shardId-000000000000 is the format of the stream
shard_it = connection_kinesis.get_shard_iterator(StreamName=##KINESIS_STREAM##, ShardId='shardId-000000000000', ShardIteratorType='LATEST')["ShardIterator"]

recs = connection_kinesis.get_records(ShardIterator=shard_it, Limit=1)

#This is the new shard iterator returned after queueing the data from the stream.
shard_it = out["NextShardIterator"]