Improve the deployment times

[vjrj]

Analysing the deployments times of a full testing portal, it takes 2 hours to deploy completely.

Doing a quick summary confirms my opinion that nginx tasks are currently the slower tasks:

• nginx tasks take 18.07 min
• package and other installs: 8 min
• download: 3.97 min
• apt related tasks 2.58 min
• exec-jar: 2.52 min
• restarts: 1.84 min
• waits: 1.49 min

A summary of biggest times:

nginx_vhost : add location https fragments ----------------------------- 11.58s                                                                        [40/12796]
nginx_vhost : add location https fragments ----------------------------- 11.81s
postgresql : Install Postgres ------------------------------------------ 12.86s
install mysql for ansible and python 3 (Debian) ------------------------ 12.96s
nginx_vhost : add basic https fragments -------------------------------- 13.04s
nginx_vhost : add basic https fragments -------------------------------- 13.05s
nginx_vhost : add basic https fragments -------------------------------- 13.05s
nginx_vhost : add basic https fragments -------------------------------- 13.06s
nginx_vhost : add basic https fragments -------------------------------- 13.07s
nginx_vhost : add basic https fragments -------------------------------- 13.07s
nginx_vhost : add basic https fragments -------------------------------- 13.08s
nginx_vhost : add basic https fragments -------------------------------- 13.08s
nginx_vhost : add basic https fragments -------------------------------- 13.09s
Add nagios plugins to do test after deploy ----------------------------- 13.15s
nginx_vhost : add basic https fragments -------------------------------- 13.16s
nginx_vhost : add location https fragments ----------------------------- 13.18s
nginx_vhost : add location https fragments ----------------------------- 13.21s
nginx_vhost : add basic https fragments -------------------------------- 13.22s
nginx_vhost : add basic https fragments -------------------------------- 13.26s
nginx_vhost : add basic https fragments -------------------------------- 13.27s
nginx_vhost : add basic https fragments -------------------------------- 13.32s
cassandra3 : install Cassandra (Debian) -------------------------------- 13.37s
nginx_vhost : add basic https fragments -------------------------------- 13.57s
nginx_vhost : add location https fragments ----------------------------- 13.66s
nginx_vhost : add basic https fragments -------------------------------- 13.73s
nginx_vhost : add basic https fragments -------------------------------- 13.76s
ansible-elasticsearch : restart elasticsearch -------------------------- 13.79s
ansible-elasticsearch : restart elasticsearch -------------------------- 14.40s
postgresql : install python-pip ---------------------------------------- 14.64s
install ca-certificates & openjdk 8 java (Debian) ---------------------- 14.85s
bie-index : copy all species list configurations ----------------------- 14.92s
postgresql : Install Postgres ------------------------------------------ 15.01s
install ca-certificates & openjdk 8 java (Debian) ---------------------- 15.45s
install ca-certificates & openjdk 8 java (Debian) ---------------------- 15.77s
biocache3-service : copy all config.properties ------------------------- 15.99s
biocache-cli : copy solr-template -------------------------------------- 16.03s
logger-service : copy transient files to tmp (schemas etc) ------------- 16.89s
restart tomcat --------------------------------------------------------- 17.17s
nginx_vhost : remove upstream fragment if not needed ------------------- 17.31s
nginx_vhost : add location https fragments ----------------------------- 17.44s
nginx_vhost : add location https fragments ----------------------------- 17.52s
nginx_vhost : add location https fragments ----------------------------- 18.21s
pipelines : dpkg-reconfigure to get rid of configuration changes ------- 18.55s
Update apt and install pipelines --------------------------------------- 18.96s
exec-jar : Download apikey JAR (unauthenticated) ----------------------- 19.11s
nginx_vhost : add location https fragments ----------------------------- 19.71s
nginx_vhost : add location https fragments ----------------------------- 19.75s
exec-jar : Download doi-service JAR (unauthenticated) ------------------ 19.80s
exec-jar : Download cas-management JAR (unauthenticated) --------------- 19.87s
nginx_vhost : add location https fragments ----------------------------- 20.46s
namedata : get custom source name data --------------------------------- 21.62s
biocache-cli : download biocache store jar ----------------------------- 22.52s
Gathering Facts -------------------------------------------------------- 23.30s
namedata : unpackage custom source name data --------------------------- 24.51s
exec-jar : Download cas JAR (unauthenticated) -------------------------- 24.58s
exec-jar : Download userdetails JAR (unauthenticated) ------------------ 28.49s
pipelines : Install required system packages --------------------------- 29.92s
wait for apikey is running to create the apikeys ----------------------- 30.53s
biocache-db : ensure cassandra 3.x is running -------------------------- 30.86s
cas5 : wait for cas is running to create the first admin user ---------- 30.92s
cas5 : Ensure cas is in a running state -------------------------------- 31.23s
Ensure cas-management is in a running state ---------------------------- 31.24s
Ensure apikey is in a running state ------------------------------------ 31.26s
Ensure userdetails is in a running state ------------------------------- 31.26s
exec-jar : Download image-service JAR (unauthenticated) ---------------- 32.43s
biocache-cli : copy vocabs --------------------------------------------- 35.35s
biocache-properties : Download SDS layers ------------------------------ 36.03s
ansible-elasticsearch : Debian - Ensure elasticsearch is installed ----- 40.71s
solr7 : copy SOLR config files for BIE and Biocache -------------------- 52.69s
pipelines : Install ala-namematching-service --------------------------- 56.73s
pipelines : dpkg-reconfigure to get rid of configuration changes ------- 61.55s
start postgresql and configure to startup automatically ---------------- 62.95s
Upgrade all apt packages ----------------------------------------------- 78.38s
nginx_vhost : add location https fragments ----------------------------- 91.74s
nginx_vhost : add location https fragments ----------------------------- 91.94s
nginx_vhost : add location https fragments ----------------------------- 94.81s
pipelines : Install the ala-sensitive-data-service package ------------- 95.84s

In a docker based deployment, the docker deployment itself is quite fast (I recommend to use the docker-service tag in re-deploys). But, if the role have a nginx vhost task, the time of deployment increases a lot.

I recommend to use:
https://github.com/nginxinc/ansible-role-nginx-config
instead of nginx_vhost or to not use fragments in nginx_vhost and use a template with conditionals as ansible-role-nginx-config does.

I use ansible-role-nginx-config in gbif.es for the rest of our services, and it's quite fast. In seconds I can reconfigure ~20 subdomains.

Download tasks can be faster in reruns if we use checksums to avoid to download things twice.