Ubuntu HA Cluster with lsyncd, remote MariaDB, Apache Reverse Proxy Setup Guide

Ubuntu HA Cluster with lsyncd, remote MariaDB, Apache Reverse Proxy Setup Guide

The following document describes how to setup High Availability and failover ERPNext Cluster based on the latest stable distribution of Ubuntu Server 14.04 LTS x86-64.

Before we Start

It is expected that you:

• have enough experience in Ubuntu server administration but hate vim and love mc (Midnight Commander),
• must understand what each provided command is doing,
• follow this guide step-by-step, and replace certain values matching your network structure
• you have your own dedicated servers located in the same local TCP/IP network (i.e. 192.168.2.0/24),
• already set up and tweaked your Ubuntu servers,
• with password-less SSH (RSA or DSA key) root login enabled between the machines,
• have at least two servers identically set up with two different public IP addresses (preferably in different subnets) to act as balancers/failover/gateways and as ERPNext Application Servers,
• Bind9 service setup on each of these machines, providing round-robin/failover DNS resolution of your ERPNext FQDN names (top level domains with wildcard sub-domains, for easy SaaS service setups),
• third server as dedicated MariaDB MySQL & Memcached "Master" servers properly tuned for best performance (MariaDB MySQL & Memcached must be allowed to listen/bind on 0.0.0.0 address), available locally to frontend Application Servers,
• taken care of backing up your data prior to starting the setup,
• and have lots of patience.

In the the example setup we use 3 servers as follows:

Server 1: PUBLIC "GATEWAY" with local IP address 192.168.2.1
Server 2: PUBLIC "GATEWAY2" with local IP address 192.168.2.2
Server 3: LOCAL DATABASE "DB" SERVER with local IP address 192.168.2.21

Lets begin...

Add to /etc/hosts on each server:

192.168.2.1  gateway
192.168.2.2  gateway2
192.168.2.21 db

I recommend installing low latency kernel (at the time of writing the guide the kernel version was 3.13.0-36):

su -i
apt-get install linux-image-lowlatency linux-image-3.13.0-36-lowlatency
update-initramfs -ck all
update-grub2
reboot

If you have /var/www mounted as separate partition, make sure /etc/fstab entry for it looks similar to this:

# /var/www was on /dev/sda4 during installation
UUID=d1bb10a1-0f00-4595-b8e2-13a53c8aa534 /var/www        ext4    noatime,nodiratime,relatime 0       2

Install required software to begin with:

su -i
apt-get update
apt-get upgrade
apt-get purge apparmor
apt-get install python-software-properties
apt-get install apache2 apache2-bin apache2-data apache2-utils
a2dismod mpm_prefork mpm_event
apt-get install apache2-mpm-worker
a2enmod mpm-worker
apt-get install mc htop git socat ufw sysv-rc-conf lsyncd python-dev python-setuptools build-essential python-mysqldb git ntp screen mariadb-common libmariadbclient-dev libxslt1.1 libxslt1-dev redis-server libssl-dev libcrypto++-dev postfix supervisor python-pip python-setproctitle python-concurrent.futures python-eventlet python-greenlet python-celery
pip install gunicorn
cd /tmp && wget http://downloads.sourceforge.net/project/wkhtmltopdf/0.12.1/wkhtmltox-0.12.1_linux-trusty-amd64.deb
dpkg -i wkhtmltox-0.12.1_linux-trusty-amd64.deb

Create system user for ERPNext:

useradd erpnext -U -m -r -b /var/www -s /bin/bash
chmod o+x /var/www/erpnext
chmod o+r /var/www/erpnext

If using UFW firewall, add these rules on your GATEWAY servers:

I suggest completely disabling UFW on DATABASE server since it is bound within local TCP/IP network.

Still editing the document!!!