Add meaningful robot.txt

Currently the robot.txt is hardcoded and does tell crawlers to ignore the site (because it is a test system). 

For deployment we would need to
a) setup a correct robot.txt
b) maybe add the option for administrators between the production and test robot.txt if they want to?