update pipe to add/del input data on run, add some additions to docum…

…entation

README.md

@@ -4,17 +4,23 @@ a pipeline to take vietnam admin unit shapefiles and insert them as postgis tabl
 
 ### install
 
+#### node packages
 `$ yarn install`
 
-### configuration
+#### s3 cli
+`$ pip install awscli`
+
+### docker
+[mac](https://docs.docker.com/docker-for-mac/install/#where-to-go-next)
+[pc](https://docs.docker.com/docker-for-windows/install/)
 
 #### data
 
-create a folder `./data/input` and add in the necessary `vietnam-communes.shp` file and its related files
+create a folder `./data/input`. the pipeline handles downloading data from s3.
 
 #### database
 
-add a a file `./db/local/index.js` of the following spec
+add a file `./db/local/index.js` of the following spec
 
 ```javascript
 module.exports = {
@@ -25,6 +31,10 @@ module.exports = {
 }
 ```
 
+#### adding an additional processing module
+
+TBA
+
 # run
 
 `yarn run start`

admin-tables-pipe.sh

@@ -1,10 +1,9 @@
 # Synopysis: links a set of I/O geoprocessing scripts that transform a commune level shapefile of Vietnam admin areas
 ###########  into three postgis tables at commune, district, and province levels/
 
-# copy input shapefiles from the s3 bucket in which they live
-echo --- downloading input boundaries from s3 ---
-aws s3 cp s3://openroads-vn-boundaries ./data/input --recursive
 
+# input directory that holds to which the initial shapefiles are downloaded from s3
+INPUT_DIR=./data/input
 # output directory that holds the final output of linked processes
 OUT_DIR=./data/output
 # the base processing directory that includes sub directories that I/O data for each process
@@ -19,6 +18,12 @@ rm -rf ${PROCESSING_BASE_DIR}
 # make handoff and process directories for current pipeline run
 mkdir ${PROCESSING_BASE_DIR}
 mkdir ${HNDF_DIR}
+mkdir ${INPUT_DIR}
+
+# copy input shapefiles from the s3 bucket in which they live
+echo --- downloading input boundaries from s3 ---
+aws s3 cp s3://openroads-vn-boundaries ${INPUT_DIR} --recursive
+
 
 # make directories in ${PROCESSING_BASE_DIR} for each process's I/O these process scripts live in ./processing
 for FILE in ./processing/*
@@ -69,4 +74,5 @@ done
 # clean up temp directories and remove the input data
 rm -rf ${HNDF_DIR}
 rm -rf ${PROCESSING_BASE_DIR}
-rm -f ./data/input/*
+rm -R ${INPUT_DIR}
+