Merge pull request #159 from elastic/pip-and-https-restaurant

Pip and https fix for restaurant

Author: Dale McDiarmid

Exploring Public Datasets/nyc_restaurants/scripts/README.md

@@ -1,33 +1,73 @@
 ## Ingest Data using Python scripts
 
-If you want to ingest data into Elasticsearch starting with the raw CSV data files follow the instructions below:
+If you want to ingest data into Elasticsearch starting with the raw CSV data
+files follow the instructions below:
 
-##### 1. Download the following files: <br>
-- `ingestRestaurantData.py` - Python script to process and ingest.  This script downloads the required dataset.
+#### 1. Download the following files:
+
+- `ingestRestaurantData.py` - Python script to process and ingest.  Note that this script downloads the required dataset.
 - `inspection_mapping.json` contains mapping for Elasticsearch index
 
 #### 2. Install and Configure Python
 
 Requires Python 3.
-Install Dependencies using pip i.e. `pip install -r requirements.txt`
+Install Dependencies using pip i.e. 
+```shell
+pip install -r requirements.txt
+```
+
+Note that MacOS users may need to `brew install python3`, 
+which would change the pip command to 
+```shell
+pip3 install -r requirements.txt
+```
+#### 3. Optionally, configure the Python script for SSL
+
+If your instance of Elasticsearch requires SSL, is not running locally, or both,
+you can tweak the script to enable it.
+
+Inside the script you will notice the connection string for Elasticsearch:
+
+```code
+ es = elasticsearch.Elasticsearch(
+ #     ['host1'],
+ #     http_auth=('myuser', 'mypassword'),
+ #     port=443,
+ #     use_ssl=True
+)
+```
 
+Replace the host entry with the name of your Elasticsearch endpoint (if more
+than one endpoint you can use a comma-separated list).  For additional arguments
+see the Elasticsearch Python Client documentation
+(https://elasticsearch-py.readthedocs.io/en/master/api.html)
+
+#### 4. Run Python script to process, join data and index data
+
+Run `ingestRestaurantData.py` (requires Python 3). When the script is done
+running, you will have a `nyc_restaurants` index in your Elasticsearch instance
 
-##### 2. Run Python script to process, join data and index data<br>
-Run `ingestRestaurantData.py` (requires Python 3). When the script is done running, you will have a `nyc_restaurants` index in your Elasticsearch instance
 ```
-  python3 ingestRestaurantData.py
+python3 ingestRestaurantData.py
 ```
+
 NOTE:
 - The script makes a call to Google geocoding API to get the lat/lon information for restaurants addresses. (a) You might need to sign up for a API key to avoid hitting usage limits. (b) Depending on your internet connection and the size of the inspection dataset, this step might take a 30 minutes to a few hours to complete.
 - We have also included a iPython Notebook version of the script `ingestRestaurantData.ipynb` in case you prefer running in a cell-by-cell mode.
 
-##### 3. Check if data is available in Elasticsearch
+#### 5. Check if data is available in Elasticsearch
+
 Check to see if all the data is available in Elasticsearch. If all goes well, you should get a `count` response of `473039` when you run the following command.
 
-  ```shell
-  curl -H "Content-Type: application/json" -XGET localhost:9200/nyc_restaurants/_count -d '{
-  	"query": {
-  		"match_all": {}
-  	}
-  }'
-  ```
+```shell
+curl -H "Content-Type: application/json" -XGET localhost:9200/nyc_restaurants/_count -d '{
+"query": {
+	"match_all": {}
+}
+}'
+```
+
+NOTE:
+
+If you are using https you will likely need to also use the
+`--user username:password` option with your curl command

Exploring Public Datasets/nyc_restaurants/scripts/ingestRestaurantData.py

@@ -6,8 +6,17 @@
 import elasticsearch
 import json
 import re
-
-es = elasticsearch.Elasticsearch()
+import certifi
+
+# If you are using the Elastic cloud, or need https/ssl, toggle the below 
+# commented sections.  Note that the Elastic cloud may be using port 9243
+# 
+es = elasticsearch.Elasticsearch(
+#     ['host1'],
+#     http_auth=('myuser', 'mypassword'),
+#     port=443,
+#     use_ssl=True
+)
 
 # In this example, we use the [Google geocoding API](https://developers.google.com/maps/documentation/geocoding/) to translate addresses into geo-coordinates. Google imposes usages limits on the API. If you are using this script to index data, you many need to sign up for an API key to overcome limits.
 

Exploring Public Datasets/nyc_restaurants/scripts/requirements.txt

@@ -1,8 +1,10 @@
 elasticsearch==5.0
+cython==0.26
 geopy==1.11.0
 numpy==1.11.2
 pandas==0.19.0
 python-dateutil==2.5.3
 pytz==2016.7
 six==1.10.0
-urllib3==1.18
+urllib3==1.18
+certifi==2017.7.27.1