Ubuntu HA Cluster with lsyncd, remote MariaDB, Apache Reverse Proxy Setup Guide

Ubuntu HA Cluster with lsyncd, remote MariaDB, Apache Reverse Proxy Setup Guide for Dummies

The following document describes how to setup High Availability and failover ERPNext Cluster based on the latest stable distribution of Ubuntu Server 14.04 LTS x86-64.

Before we Start

It is expected that you:

• have enough experience in Ubuntu server administration but hate vim and love mc (Midnight Commander),
• must understand what each provided command is doing,
• follow this guide step-by-step, and replace certain values matching your network structure
• you have your own dedicated servers located in the same local TCP/IP network (i.e. 192.168.2.0/24),
• already set up and tweaked your Ubuntu servers,
• with password-less SSH (RSA or DSA key) root login enabled between the machines,
• have at least two servers identically set up with two different public IP addresses (preferably in different subnets) to act as balancers/failover/gateways and as ERPNext Application Servers,
• Bind9 service setup on each of these machines, providing round-robin/failover DNS resolution of your ERPNext FQDN names (top level domains with wildcard sub-domains, for easy SaaS service setups),
• third server as dedicated MariaDB MySQL & Memcached "Master" servers properly tuned for best performance (MariaDB MySQL & Memcached must be allowed to listen/bind on 0.0.0.0 address), available locally to frontend Application Servers,
• taken care of backing up your data prior to starting the setup,
• and have lots of patience.

In the the example setup we use 3 servers as follows:

Server 1: PUBLIC "GATEWAY" with local IP address 192.168.2.1
Server 2: PUBLIC "GATEWAY2" with local IP address 192.168.2.2
Server 3: LOCAL DATABASE "DB" SERVER with local IP address 192.168.2.21

Lets begin...

Add to /etc/hosts on each server:

192.168.2.1  gateway
192.168.2.2  gateway2
192.168.2.21 db

I recommend installing low latency kernel (at the time of writing the guide the kernel version was 3.13.0-36):

su -i
apt-get install linux-image-lowlatency linux-image-3.13.0-36-lowlatency
update-initramfs -ck all
update-grub2
reboot

If you have /var/www mounted as separate partition, make sure /etc/fstab entry for it looks similar to this:

# /var/www was on /dev/sda4 during installation
UUID=d1bb10a1-0f00-4595-b8e2-13a53c8aa534 /var/www        ext4    noatime,nodiratime,relatime 0       2

Install required software to begin with:

su -i
apt-get update
apt-get upgrade
apt-get purge apparmor
apt-get install python-software-properties
apt-get install apache2 apache2-bin apache2-data apache2-utils
a2dismod mpm_prefork mpm_event
apt-get install apache2-mpm-worker
a2enmod mpm-worker
apt-get install mc htop git socat ufw sysv-rc-conf lsyncd nano python-dev python-setuptools build-essential python-mysqldb git ntp screen mariadb-common libmariadbclient-dev libxslt1.1 libxslt1-dev redis-server libssl-dev libcrypto++-dev postfix supervisor python-pip python-setproctitle python-concurrent.futures python-eventlet python-greenlet python-celery
pip install gunicorn
cd /tmp && wget http://downloads.sourceforge.net/project/wkhtmltopdf/0.12.1/wkhtmltox-0.12.1_linux-trusty-amd64.deb
dpkg -i wkhtmltox-0.12.1_linux-trusty-amd64.deb

If using UFW firewall, make sure your ports 22 and 8000 on Application Servers are open bidirectionally for local network. I suggest completely disabling UFW on DATABASE server since it is bound within local TCP/IP network.

Create system user for ERPNext:

useradd erpnext -U -m -r -b /var/www -s /bin/bash
chmod o+x /var/www/erpnext
chmod o+r /var/www/erpnext

Basic install of Frappe Bench and Initialization thereof:

su -l erpnext
git clone https://github.com/frappe/bench bench-repo
exit

Should be still logged in as root, and continue:

pip install -e /var/www/erpnext/bench-repo
su -l erpnext
bench init frappe-bench && cd frappe-bench
bench get-app erpnext https://github.com/frappe/erpnext
bench get-app shopping_cart https://github.com/frappe/shopping-cart

Will continue initializing sites a bit later after done with setting up basic cluster synchronization with lsyncd, and making some tricks with socat to locally relay remotely running MariaDB and Memcached (we should not have these services up and running on Application Servers, i.e. GATEWAY & GATEWAY2).

Lets fake mysql user on Application Servers, if it does not exist there:

useradd mysql -U -M -r 

Open /etc/rc.local file and add these lines before last line "exit 0":

# Start relaying remote MySQL to local socket
mkdir /var/run/mysqld
chown mysql: /var/run/mysqld
socat UNIX-LISTEN:/var/run/mysqld/mysqld.sock,fork,reuseaddr,unlink-early,user=mysql,group=mysql,mode=777 TCP:192.168.2.21:3306 &

# Start relaying remote memcached to localhost
socat TCP-LISTEN:11211,reuseaddr,fork,user=proxy,group=proxy,perm=0777 TCP:192.168.2.21:11211 &  

Now execute this /etc/rc.local file and then run:

htop
Press "F4"
Start typing "socat"
Now you should see two "socat" processes running.
Exit "htop" by pressing ESC feq types and then "q"

Check socat relaying remote memcached with telnet, to see if it is connecting:

telnet localhost 11211
^]
quit

Now check mysql socket for relayed remote mysql service:

stat /var/run/mysqld/mysqld.sock

And this way to make sure it is working:

mysqlreport --user root --password YOURMYSQLROOTPASSWORD

Good stuff. So far, we are more then half way away! Now lets setup lsyncd synchronization of /var/www/erpnext sub-directories:

cd /etc/lsyncd
nano lsyncd.conf.lua

If the file is empty, then add these lines, where "gatewayXYZ" is the local name (you set it up previously in /etc/hosts) of "the other" Application Server where you want to synch the data to. You need to perform these edits on each Application Servers to make the data sync to and fro.

Best way to sync data is the STAR pattern where you have ""master** lsyncd instance, and slaves, allowing you to do all maintenance and further setups on master server with automatic syncing to the slaves i.e.:

• GATEWAY (1st App Server) <--> GATEWAY2 (2st App Server)
• GATEWAY (1st App Server) <--> APPSERVER3 (3st App Server)
• GATEWAY (1st App Server) <--> APPSERVER4 (4st App Server) etc.

Here is the code to add to /etc/lsyncd/lsyncd.conf.lua:

settings{
    logfile = "/var/log/lsyncd.log",
    statusFile = "/var/log/lsyncd.status",
    statusInterval=1,
    delay=1,
    maxDelays = 1,
    insist = 1,
    maxProcesses = 3
    }
sync{default.rsync, source="/var/www/erpnext/bench-repo/", target="root@gatewayXYZ:/var/www/erpnext/bench-repo/",
    rsync = {acls=true, verbose=true, archive = true, owner = true, perms = true, group = true, compress = false, whole_file = true, rsh="/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"}
    }
sync{default.rsync, source="/var/www/erpnext/frappe-bench/", target="root@gatewayXYZ:/var/www/erpnext/frappe-bench/", exclude = {"*.log"},
    rsync = {acls=true, verbose=true, archive = true, owner = true, perms = true, group = true, compress = false, whole_file = true, rsh="/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"}
    }

I recommend adding /etc/ssl, /etc/bind, /etc/apache2/sites-available and /etc/apache2/sites-enabled to frontend GATEWAYs (still the same /etc/lsyncd/lsyncd.conf.lua) for uniform essentials synchronizations:

sync{default.rsync, source="/etc/apache2/sites-available/", target="root@gatewayXYZ:/etc/apache2/",
    rsync = {acls=true, verbose=true, archive = true, owner = true, perms = true, group = true, compress = false, whole_file = true, rsh="/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"}
    }
sync{default.rsync, source="/etc/apache2/sites-enabled/", target="root@gatewayXYZ:/etc/apache2/",
    rsync = {acls=true, verbose=true, archive = true, owner = true, perms = true, group = true, compress = false, whole_file = true, rsh="/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"}
    }
sync{default.rsync, source="/etc/bind/", target="root@gatewayXYZ:/etc/bind/",
    rsync = {acls=true, verbose=true, archive = true, owner = true, perms = true, group = true, compress = false, whole_file = true, rsh="/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"}
    }
sync{default.rsync, source="/etc/ssl/", target="root@gatewayXYZ:/etc/ssl/",
    rsync = {acls=true, verbose=true, archive = true, owner = true, perms = true, group = true, compress = false, whole_file = true, rsh="/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"}
    }

Save the lsyncd.conf.lua file and start/restart the lsyncd service:

service lsyncd restart

You may check the lsyncd log files to make sure everything is working smoothly. Also, worthy to note, that lsync, though being ways better then unison and such, has a bug, where exclude rules do not perform as expected, so, in my further writing you will see I cheated it a little bit, in order to escape conflicts on frappe-bench's celerybeat.pid and scheduler.schedule files overwrites. Each Application server will generate its own *.pid and *.schedule file to escape any conflicts.

tail -n 100 /var/log/lsyncd.log
tail -n 100 /var/log/lsyncd.status

---

Still editing the document bellow!!!

And now is the interesting, but tricky part - setting up ERPNext sites with remote MariaDB MySQL server:

WILL ADD INFO HERE

Test-Driving the frappe-bench. There should be no errors excreted, just nice :

su -l erpnext
cd frappe-bench/
bench start

If everything is OK, then just kill the processes, executing the command a few times:

killall -9 python

Tuning frappe-bench:

WILL ADD INFO HERE

Setting up supervisor application to run ERPNext unattended in our fancy-dancy ERPNext High Availability / Failover Cluster:

WILL ADD INFO HERE

Last Steps...

Setting up Apache Reverse Proxy.

Making sure everything needed is installed:

WILL ADD INFO HERE

Editing the /etc/apache2 configuration ...

WILL ADD INFO HERE

The End

If you notice any errors of suggest anything to be added, please email directly at "victor at devdesco.com"