Installing Jenkins Using Docker on Windows

This post is how to install Jenkins using Docker in Windows.

Install Docker

Go to Docker and download “Docker Desktop for Windows – x86_64”. Then run as administrator and install. If you don’t have an account you can create one.

Then open Docker Desktop and go to settings and enable the following “Enable Docker terminal”.

Once you are done you need to start the Docker Engine.

 

 

 

If you get the following error “Failed to initialize: unable to resolve docker endpoint: open C:\Users\<USER>\.docker\machine\machines\default\ca.pem: The system cannot find the path specified.” Then go to environment variables and modify DOCKER_CERT_PATH and set to “C:\Users\<USER>\.docker\machine\certs”.

Then open the docker terminal.

Install Jenkins

Create a network called “jenkins”. This will give you back the id.

docker network create jenkins

You can check that it created successfully by running the following command

docker network ls

Next we run a docker:dind Docker image. If you don’t have the docker image it will download it.

A dind is a Docker In Docker allows developers to run a Docker container within an already running Docker container to support CI/CD pipelines and create sandboxed container environments.

docker run --name jenkins-docker --rm --detach --privileged --network jenkins --network-alias docker --env DOCKER_TLS_CERTDIR=/certs --volume jenkins-docker-certs:/certs/client --volume jenkins-data:/var/jenkins_home --publish 2376:2376 docker:dind

Next we create a Dockerfile

FROM jenkins/jenkins:2.462.2-jdk17
USER root
RUN apt-get update && apt-get install -y lsb-release
RUN curl -fsSLo /usr/share/keyrings/docker-archive-keyring.asc \
  https://download.docker.com/linux/debian/gpg
RUN echo "deb [arch=$(dpkg --print-architecture) \
  signed-by=/usr/share/keyrings/docker-archive-keyring.asc] \
  https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
RUN apt-get update && apt-get install -y docker-ce-cli
USER jenkins
RUN jenkins-plugin-cli --plugins "blueocean docker-workflow"

Next we will build a new docker image with the Dockerfile from the previous step. Be sure to name it appropriately and include versions. If you don’t have the right image it will get it for you. Make sure you are in the directory where you saved your Dockerfile.

docker build -t jenkins-blueocean:2.462.2-1 .

Next we run the image as a container in Docker

docker run --name jenkins-blueocean --restart=on-failure --detach --network jenkins --env DOCKER_HOST=tcp://docker:2376 --env DOCKER_CERT_PATH=/certs/client --env DOCKER_TLS_VERIFY=1 --volume jenkins-data:/var/jenkins_home --volume jenkins-docker-certs:/certs/client:ro --publish 8080:8080 --publish 50000:50000 jenkins-blueocean:2.462.2-1

Docker Container Logs using the below command. This will show you the password.

docker logs jenkins-blueocean

Record the password located “/var/jenkins_home/secrets/initialAdminPassword”.

docker exec jenkins-blueocean cat /var/jenkins_home/secrets/initialAdminPassword

or

docker exec jenkins-blueocean bash
cat /var/jenkins_home/secrets/initialAdminPassword
exit

Then navigate to http://localhost:8080 and enter the password in the “Unlock Jenkins” screen.

 

 

 

 

 

Then select the packages you want to install and click install.

Once the packages install then you need to create the admin user.

Then select your url and click “Save and Finish”.

 

 

 

 

 

Shell: Functions

In this tutorial I will show you how to work with functions.

Define Function

You return 0 for success and 1 for failure

function test()
{
    return 0
}

Call function with no arguments

test

Call function with arguments

val1='test'
val2='test2'
test $val1 $val2

Call function with arguments that have spaces in the value

val1='test'
val2='test2 test23'
test "${val1}" "${val2}"

 

 

Shell: Misc Commands

In this tutorial I will show a few useful commands when working with linux shell.

Check Directory Exists:

if [ -d /opt/test/ ]; then
    echo 'Directory Exists'
fi

Check Directory Not Exists:

if [ ! -d /opt/test/ ]; then
    echo 'Directory Does Not Exist'
fi

Check File Exists:

if [ -f /opt/test/test.log ]; then
    echo 'File Exists'
fi

Check File Not Exists:

if [ ! -f /opt/test/test.log ]; then
    echo 'File Does Not Exist'
fi

Lowercase Variable:

val='TEXT'
echo "${val,,}"

Echo Variable:
This will print the value of “test”. Notice we use double quotes.

test='My Test Val'
echo "$test"

Echo String:

echo 'My test string'

Split:
This will split on the comma into an array list and then loop through it.

test='test 1,test 2'
split_test=(${test//,/ })

for val in "${split_test[@]}"
do
    echo $val
done

Date:
This will print the date in the format YYYY-MM-dd

my_date="$(date +Y-%m-%d)"
echo "$my_date"

Remove Space From Variable:

VAL='test string'
echo "${VAL//\ /}"

Increment Variable:

index=0
index=$((index+1))

Substring

VAL='test string'
echo "${VAL:4:4}"

If value is equal to

VAL='hello'
if [ "$VAL" == 'hello' ] ; then
    echo 'Is Equal'
fi

If with OR

VAL='hello'
if [ "$VAL" == 'hello' ] || [ "$VAL" != 'hi' ] ; then
    echo 'Is Hello'
fi

If Variable is Empty

VAL=''
if [ -z "$VAL" ] ; then
    echo 'Is Empty'
fi

Append to File

echo 'Hi' >> file_to_log_to.log

Write to File

echo 'Hi' > file_to_log_to.log

While Loop: Increment to 10

This will loop till the value is 9 then exit.

i=0
while [ $i -lt 10 ];
do
    echo "$i"
done

whoami

USER=$(whoami)

If Variable Contains Text

VAL='my Test String'
if [[ "${VAL,,}" == *"test"* ]] ; then
    echo "Found test"
fi

Color Coding

NoColor=$'\033[0m'
READ=$'\033[0;31m'
GREEN=$'\033[0;32m'
YELLOW=$'\033[1;33;40m'

printf "%s Variable Not Set %s\n" "${RED}" "${NoColor}"

Get Log to a logfile and console

SOME_COMMAND 2>&1 | tee -a "${LOG_FILE_PATH}"

Read a JSON config

JSON=$(cat "/path/to/json/file.json")
export MY_VAR=$(echo "${JSON}" | python -c 'import json,sys;obj=json.load(sys.stdin);print(obj["MyKey"])')

Extract tar to Folder

sudo tar -xvf /the/location/file.tar -C /to/location/ --force-local --no-same-owner

Update Certificates

This will update certificates. After you put a certificate in /usr/local/share/ca-certificates/

update-ca-certificates

PipeStatus

somecommand
RETURN_CODE=${PIPESTATUS[0]}

Shell: AWK Command

In this tutorial I will show you some useful awk uses.

Determine Line Number

This will find the line number of the text_to_find.

lineNumber=`awk '/TEXT_TO_FIND/{print NR}' /file/path/name.extension`
Split Text By Space

If you want to split a word by a space and then select the second one

val='test test2'
echo $val | awk '{ print $2 }'
Split Text By Comma

If you want to split a word by a space and then select the second one

val='test test2'
echo $val | awk -F, '{ print $2 }'

 

Shell: SED Command

In this tutorial I will show a few useful examples of using “sed”.

-i: inline replace
-e: command to run

Add Line At End

Notice ($) and (a after $). Match last line (‘$’) and append (‘a’)

sudo sed -i -e "\$amy new line" /file/path/name.extension
Match Replace

This will replace the text on the left with the text on the right. Notice the “/” at the beginning and “/c\” in the middle.

sudo sed -i '/myCustomText=/c\myCustomText=newtext' /file/path/name.extension
Remove Matched Line

This will remove the line that matches the text. Notice the “/d” at the end and the “/” at the beginning.

sudo sed -i '/RemoveMyCustomText/d' /file/path/name.extension
Remove All Occurrences

This will remove all occurrences of “NIFI.APACHE.ORG” and replace with “REALM.CA”. Notice “/g” at end for global replace. Notice “s/” at the beginning. If means substitute. The “/” in the middle is used to denote the separation of the text to find and the text to replace.

sudo sed -i 's/NIFI.APACHE.ORG/REALM.CA/g' /file/path/name.extension
Handle ” Quotes / Handle XML / Add Tab

So the below example will go a few things. It will show you how to escape double quotes. It shows you how to add a tab and shows you how to handle xml. It also will add a line under text.

sudo sed -i -e "/<property name=\"Default Realm\">REALM.CA<\/property>/a \ \t<property name=\"Kerberos Config File\">/etc/krb5.conf<\/property>" /file/path/name.extension
Delete Lines By Number
sudo sed -i -e "$lineNumber,$maxLineNumber d" /file/path/name.extension

VirtualBox

In this post I will just walk through a couple settings that you may need to configure. For these examples I am using Ubuntu 16.04.

Networking

If you want your virtualbox os to have it’s own IP on the network and accessable outside the vm then use “Bridged Adapter”.

  • Select “Enable Network Adapter”
  • Attached to: Bridged Adapter
  • Expand Advanced
  • Promiscuous Mode: Allow All
  • Select Cable Connected
  • Adapter Type: Desktop adapter.

If you want to just forward a port to localhost then use “NAT”.

  • Select “Enable Network Adapter”
  • Attached to: NAT
  • Expand Advanced
  • Adapter Type: Desktop adapter.
  • Click “Port Forwarding”
  • Add new Rule
    • Host IP: 127.0.0.1
    • Host Port: 22
    • Guest IP: Leave empty
    • Guest Port: 22

Shared Folders

If you want to share a folder from Host to Guest.

  • Click Add Share
  • Folder Path: Select folder from Host machine
    • I will us C:\data
  • Folder Name: What the share will be on the Guest
    • data
  • Turn on VM
  • Download https://download.virtualbox.org/virtualbox/5.2.12/VBoxGuestAdditions_5.2.12.iso
  • Mount VBoxGuestAdditions_5.2.12.iso
  • On the Guest Ubuntu server run
    • sudo mount /dev/cdrom /media/cdrom
  • Install build-essential
    • sudo apt-get install build-essential linux-headers-`uname –r`
  • Run VBoxGuestAdditions
    • sudo /media/cdrom/VBoxLinuxAdditions.run
  • sudo mkdir -p /mnt/data
  • sudo reboot
  • Mount Share
    • Ubuntu Server
      • sudo mount –t vboxsf data /mnt/data
    • Ubuntu Desktop
      • sudo mount -t vboxsf -o uid=$UID,gid=$(id -g) data /mnt/data/
  • cd /mnt/data
    • Now we can share files between host and guest.

NGINX + UWSGI + Ubuntu 16

If you use Python as part of your stack in developing a web architecture you most likely will also be utilizing NGINX with UWSGI together. In the following article I will give step by step instructions on how to set this up assuming you are using sockets as part of your installation.

What is UWSGI?: A full stack for building application servers for python and various other languages.

What is NGINX?: Is simply a web server or proxy server for various protocols but can also do load balancing, etc.

NGINX Installation:

sudo apt-get install nginx

#The next command checks that nginx is running
ps -auxw | grep nginx

#Stop the service
sudo service nginx stop
#Start the service
sudo service nginx start
#Restart the service
sudo service nginx restart
#nginx status
sudo service nginx status

Once it is installed you will have two folders. One called “sites-enabled” and one called “sites-available” located under /etc/nginx/ folder. The “sites-enabled” folder usually holds a symbolic link to the actual file sometimes in “sites-available” folder. But really can be hosted anywhere on the machine.

If you want to remove the default site link you can

sudo rm /etc/nginx/sites-enabled/default

Create our symbolic link in sites-enabled if we already have a nginx.conf file ready.

cd /etc/nginx/sites-enabled/

sudo ln -s ~/path/to/nginx/conf/nginx.conf nginx.conf

It’s important to note that you may run into permissions errors depending on where you are running your nginx.conf from. So you may need to grant permissions to your users home folder.

sudo chmod 755 /home/user_your_run_your_app_from/

UWSGI Installation:

sudo apt-get install uwsgi

#Stop uwsgi
sudo service uwsgi stop
#Start uwsgi
sudo service uwsgi start
#Restart uwsgi
sudo service uwsgi restart
#uwsgi status
sudo service uwsgi status

It you want to log to a custom folder your uwsgi.ini file will need “logto” folder specified and the correct permissions set.

cd /var/log/
sudo touch /var/log/mylogfile
sudo chmod 666 /var/log/mylogfile

You uwsgi.ini will also have a pid file “pidfile” which we will need to create.

cd /tmp/
touch mypid.pid

#Ownership
sudo chown THEUSER:www-data mypid.pid

#Ensure permissions are set to -rw-r--r--

UWSGI INI Options:

You can add whatever options you prefer as part of your uwsgi.ini setup file. However the below are usually the ones you should use.

[uwsgi]
socket = :5006 #or whatever socket you want
chmod-socket = 777
pidfile=/tmp/mypid.pid
master = True
chdir = /home/user_your_run_your_app_from/

#NOTE that you may or may not need this depending on how you setup your server.
plugin = python3
wsgi_file = my_python_class.py #File that runs your application
module = whatever
callable = app #Set default WSGI callable name.
logto = /var/log/mylogfile

These are not all the options you can pick. You can also set “stats”, “buffer-size”, “processes”, “threads”, “env”, etc. Just review the docs and pick what is right for you.

The option “env” you can add many times and for each time you use it you can set environment variable for use with your application.

NOTE: If you are running flask and use send_file and run as Python3.5 you may get the error “SystemError: <built-in function uwsgi_sendfile> returned a result with an error set”. If you do then you must add the following to your uwsgi.ini file “wsgi-disable-file-wrapper = true”.

NGINX Conf Example:

upstream flask {
    server 0.0.0.0:5006;
}
server {
    client_max_body_size 1G;
    listen 0.0.0.0:8081;
    server_name localhost;
    access_log /var/log/nginx/access_log combined;
	
    location / {
        include uwsgi_params;
        uwsgi_pass flask;
    }
    location /static/ {
        alias /path/to/my/app/static/;
    }
}
fastcgi_intercept_errors on;

UWSGI Service:

Sometimes we want to run our app as a service if you do then below are the steps for that.

sudo nano /etc/systemd/system/myawesomeservice.service

#Enter the contents below

[Unit]
Description=uWSGI instance to serve My Service
After=network.target
 
[Service]
User=user_to_run_as
Group=www-data
WorkingDirectory=/path/to/my/app/
ExecStart=/usr/bin/env bash -c 'uwsgi --ini /path/to/my/app/uwsgi.ini --uid user_to_run_as --gid www-data'
Restart=always
RestartSec=3
 
[Install]
WantedBy=multi-user.target

Next you need to ensure the user has access to your folder with correct ownerships.

chown -R user_to_run_as:www-data /path/to/my/app/*

Next enable the service and start the service.

sudo systemctl enable myawesomeservice
sudo systemctl daemon-reload
sudo systemctl start myawesomeservice
sudo service myawesomeservice status

unRaid: Midnight Commander

If you are not familiar with unRaid go here. unRaid is a raid 4 implementation. If is basically a software raid.

If you want to work with the file system of unRaid whether it to be to move files or for another reason use midnight commander.

Start putty and connect to the unRaid server. Then once logged in run the following command for midnight commander.

mc

 

Ubuntu: tmux

Tmux (Terminal Multiplexer) is really helpful for running many virtual terminals that stay around even after you exit out of the server.

Some quick cheats are:

  • tmux -V: What version you are running.
  • CTRL b %: Split terminal horizontally.
  • CTRL b “: Split terminal vertically.
  • CTRL b LEFT: Switch between panes to the left.
  • CTRL b RIGHT: Switch between panes to the right.
  • CTRL b DOWN: Switch between panes to the down.
  • CTRL b UP: Switch between panes to the up.
  • CTRL b z: Toggle a pane.
  • CTRL b x: Kill the pane you are on.
  • CTRL b c: Create a session window.
  • CTRL b NUMBER: Enter 1, 0, 2, etc and you can switch to that session.
  • CTRL b d: Detach from tmux keeping it running.
  • tmux ls: Lists all sessions currently running.
  • tmux attach -t 0: Attach to the session by number. If you have two you can select 1 instead of 0.
  • tmux kill-session -t 0: This kills a session by session number.

Various Commands

Below are some useful commands for those either just learning Ubuntu 14.04 or a resource for various commands. I will continually expand on them.

Directory Exists:
if [ ! -d "##DIRECTORY##" ]; then
fi
Format a Number as two digits even if it is 1:
$(printf %02d $variable)
Increment a Number:
variable=`expr $variable + 1`
Create a new file:
touch ##FILENAME##
Check if a file contains a string:

This will return 0 if it is not found.

$(echo `grep -c '##TextToSearch##' ##FileNameToSearch##`)
Create a link to a file:
ln -s ##OriginalFile## ##LinkedFile##
List all packages installed:
dpkg -l
Remove installed package:
sudo apt-get --purge remove ##PACKAGENAME##
Install Package:
sudo apt-get install ##PACKAGENAME##
Package Exists:

This will return 0 if it is not found.

$(echo `dpkg-query -l | grep -c ##PACKAGE_NAME##`)
Python Package Exists:

This will return 0 if it is not found.

$(echo `pip freeze | grep -c ##PACKAGE_NAME##`)
Get IP Address:
ip addr show
Find a file:
find / -type f -name "##FILENAME##"
Restart Service:
/etc/init.d/postgresql restart
invoke-rc.d postgresql restart
Kill a process:
kill ##PID##
Terminal Loop:

In the terminal let’s say you want to loop and display a value. You can do it like below. I am just printing free memory. But really you can do anything.

while true; do free -m | grep /+ | gawk '{ print $4 }'; sleep 2; done
Switch to root:
sudo su root
Add Text to File:
echo "##TextToAdd##" >> ##FileNameToAddTo##
Add Text Above Other Text:
sed -i 's/^##LookForText##/##TextToAdd##\n&/' ##FileNameToAddTo##
Loop:
until [ $Variable -gt 3 ]
do
done