Build instructions

[howff]

I had to do a lot of work to get this to build. Here's my instructions

Clone the repo (demo-debian branch):

git clone -b demo-debian https://github.com/PublicaMundi/labs.geodata.gov.gr.git

Add deb package sources:

for postgresql:
wget https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/plain/pgdg/apt.postgresql.org.sh
sudo bash apt.postgresql.org.sh
for ansible:
echo "deb http://ftp.debian.org/debian wheezy-backports main" >> /etc/apt/sources.list

The ansible yml script clobbers apt/sources.list (sets it to .gr) so install packages manually first:

apt-get install python-pip python-virtualenv
apt-get install htop python python-virtualenv python-setuptools python-dev build-essential vim git screen sudo less rsync curl wget unzip
apt-get install openjdk-7-jdk tomcat6 tomcat6-admin
apt-get install apache2 libapache2-mod-wsgi
apt-get install tidy libtidy-dev supervisor
apt-get install postgresql-9.3 postgresql-9.3-postgis-2.1 postgresql-common python-psycopg2 libpq-dev libxml2-dev libxslt1-dev unzip p7zip-full python-gdal
apt-get install python-pip python-virtualenv python-setuptools python-dev

The ansible yml script requires a host called "dev": use your own IP address here:

echo "[dev]" >> /etc/ansible/hosts
echo "42.42.42.42 " >> /etc/ansible/hosts

Fix postgres if your locale is not en_US (echo $LANG) en_GB.UTF-8

NB this fixes the local copy which gets copied into /etc
sudo sed -i "s/en_US.UTF-8/$LANG/" etc/postgresql/9.3/main/postgresql.conf

Create home dir for user (not sure why it doesn't get created):

sudo mkdir /home/ckaner
sudo chown ckaner /home/ckaner

This file was empty so give it valid contents:

wget -O - https://github.com/PublicaMundi/ckanext-publicamundi/blob/master/ckanext/__init__.py >> /var/local/ckan/default/pyenv/src/ckanext-multilingual-datastore/ckanext/multilingual_datastore/init.py

Error TypeError: init() takes exactly 2 arguments (4 given)

when installing into ubuntu-14.04, try editing:
sudo vi /var/local/ckan/default/pyenv/src/ckan/setup.py
sudo -u ckaner sh -c '(cd /var/local/ckan/default/pyenv/src/ckan ; ../../bin/python setup.py develop)'
Solved by installing debian 7 wheezy instead of ubuntu 14.04

Fix pip which tries to use http: but pypi.python.org requires https now

(don't edit inside /var/local/ckan/default/pyenv/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/*.py)
(don't fix requirements.txt sed -i '1i--index-url=https://pypi.python.org/simple/' /var/local/ckan/default/pyenv/src/ckan/requirements.txt)
(because it gets overwritten when pulled from git)
Append extra_args="-i https://pypi.python.org/simple/" to action: pip lines
sed -i '/action: *pip/s/$/ extra_args="-i https://pypi.python.org/simple/"/' geodatagovgr_dev.yml

The dev version of postgres is v10 but v9.3 is installed!! Remove it

otherwise pg_config returns 10.0 when only 9.3 is avail
and that causes psycopg to barf. then install -dev otherwise pg_config barfs.
sudo dpkg -r libpq-dev
sudo apt-get install postgresql-server-dev-9.3

html5lib latest requires setuptools 18 but we have 0.6

Could install bleach 1.4.3 (not 1.4.2) which gets the right html5lib
See ckan/ckan@69408d2
(could use: sudo -u ckaner bash -c "source /var/local/ckan/default/pyenv/bin/activate; pip install -i https://pypi.python.org/simple/ bleach==1.4.3")
but we don't need bleach for this version of ckan, so:
sudo mv /var/local/ckan/default/pyenv/build/html5lib /var/local/ckan/default/pyenv/build/html5lib-broken
sudo -u ckaner bash -c "source /var/local/ckan/default/pyenv/bin/activate; pip install -i https://pypi.python.org/simple/ html5lib==0.9999999"

Due to this error installing dictdiffer (0.7.0) install 0.4.0:

File "/var/local/ckan/default/pyenv/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/package_index.py", line 475, in fetch_distribution
AttributeError: 'NoneType' object has no attribute 'clone'
Complete output from command python setup.py egg_info:
Couldn't find index page for 'pytest-runner' (maybe misspelled?)

sudo mv /var/local/ckan/default/pyenv/build/dictdiffer /var/local/ckan/default/pyenv/build/dictdiffer-broken
sudo -u ckaner bash -c "source /var/local/ckan/default/pyenv/bin/activate; pip install -i https://pypi.python.org/simple/ dictdiffer==0.4.0"

ua-parser requires pyyaml, not found, maybe PyYAML instead?

Downloading/unpacking ua-parser>=0.4.1 (from user-agents)
Downloading ua-parser-0.7.3.tar.gz
Running setup.py egg_info for package ua-parser
Couldn't find index page for 'pyyaml' (maybe misspelled?)
Try changing it to PyYAML but it doesn't help.
(sudo sed -i 's/pyyaml/PyYAML/' /var/local/ckan/default/pyenv/build/ua-parser/setup.py)
So install pyyaml manually then ua-parser manually
sudo -u ckaner bash -c "source /var/local/ckan/default/pyenv/bin/activate; pip install -i https://pypi.python.org/simple/ pyyaml"
sudo -u ckaner bash -c "source /var/local/ckan/default/pyenv/bin/activate; pip install -i https://pypi.python.org/simple/ ua-parser"

fix up the yml file geodatagovgr_dev.yml

eg. add:
sudo: True
Then apply diffs:

69c69
<       shell: "mkdir /home/solr && cp -R ~/solr-4.2.1/example/solr/* /home/solr && chown -R tomcat6:tomcat6 /home/solr"
---
>       shell: "mkdir -p /home/solr && cp -R ~/solr-4.2.1/example/solr/* /home/solr && chown -R tomcat6:tomcat6 /home/solr"
99c99
<       shell: "cd ~/ && unzip -e geoserver-2.7.4-war.zip && cp geoserver.war /var/lib/tomcat6/webapps/geoserver.war"
---
>       shell: "cd ~/ && unzip -o geoserver-2.7.4-war.zip && cp geoserver.war /var/lib/tomcat6/webapps/geoserver.war"
191c191
<       action: command psql -d ckan -c 'CREATE EXTENSION "postgis";'
---
>       action: command psql -d ckan -c 'CREATE EXTENSION IF NOT EXISTS "postgis";'
196c196
<       action: command psql -d ckan_tests -c 'CREATE EXTENSION "postgis";'
---
>       action: command psql -d ckan_tests -c 'CREATE EXTENSION IF NOT EXISTS "postgis";'
201c201
<       action: command psql -d mapclient -c 'CREATE EXTENSION "postgis";'
---
>       action: command psql -d mapclient -c 'CREATE EXTENSION IF NOT EXISTS "postgis";'
230c230
<       shell: "cd /var/local/ckan/default && mkdir log && mkdir run && mkdir tmp && mkdir -p files/resources && mkdir -p files/storage"
---
>       shell: "cd /var/local/ckan/default && mkdir -p log && mkdir -p run && mkdir -p tmp && mkdir -p files/resources && mkdir -p files/storage"
253c253
<       action: pip requirements=/var/local/ckan/default/pyenv/src/ckan/requirements.txt virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip requirements=/var/local/ckan/default/pyenv/src/ckan/requirements.txt virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
274c274
<       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-archiver/pip-requirements.txt virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-archiver/pip-requirements.txt virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
290c290
<       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-datastorer/pip-requirements.txt virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-datastorer/pip-requirements.txt virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
306c306
<       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-spatial/pip-requirements.txt virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-spatial/pip-requirements.txt virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
331c331
<       action: pip name=pyproj virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip name=pyproj virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
336c336
<       action: pip name=geolinks virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip name=geolinks virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
386c386
<       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-publicamundi/requirements.txt virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-publicamundi/requirements.txt virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
401c401
<       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-multilingual-datastore/requirements.txt virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip requirements=/var/local/ckan/default/pyenv/src/ckanext-multilingual-datastore/requirements.txt virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
416c416
<       shell: ln -s /var/local/ckan/default/pyenv/src/ckan/development.ini /var/local/ckan/default/pyenv/src/ckan/config.ini
---
>       shell: ln -sf /var/local/ckan/default/pyenv/src/ckan/development.ini /var/local/ckan/default/pyenv/src/ckan/config.ini
474c474
<       action: pip name=user-agents virtualenv=/var/local/ckan/default/pyenv/
---
>       action: pip name=user-agents virtualenv=/var/local/ckan/default/pyenv/ extra_args="-i https://pypi.python.org/simple/"
514c514
<       shell: ln -s /var/local/ckan/default/pyenv/src/mapclient/development.ini /var/local/ckan/default/pyenv/src/mapclient/config.ini
---
>       shell: ln -sf /var/local/ckan/default/pyenv/src/mapclient/development.ini /var/local/ckan/default/pyenv/src/mapclient/config.ini

Now run the installation script:
ansible-playbook --ask-sudo-pass -i "dev," -c local geodatagovgr_dev.yml

[kalxas]

Thanks for the feedback!

[howff]

It tries to use mqcdn for the map tiles. I patched every occurrence of mqcdn I could find (more than 20) and changed them to openstreetmap, i.e.
sed -i 's,otile{s}.mqcdn.com/tiles/1.0.0/osm,a.tile.openstreetmap.org,'
sed -i 's,otile{1-4}-s.mqcdn.com/tiles/1.0.0/,a.tile.openstreetmap.org,'
but
a) I don't know exactly which files I should have edited
b) the second substitution may not work as the URL will have a layer name appended
c) Due to (a) I don't know who is responsible for this and whether they've already patched it

[howff]

More packages to install: rabbitmq-server

Also, possibly need to fix a bug in celeryctl.py?
from celeryutils
should be
from celery.utils