Instagram Location Crawler

A non API python program to scrape number of posts at a location, as well as locations in a city from Instagram
I am coding this for use in my Seminar "Computational Social Sciences" to acquire data for my group

Full usage:

usage: LocationCrawler.py [-h] [-l LOCATION] [-c CITY] [-fromFile FILE TYPE]
                          [-fromDir DIR SUFFIX] [-d DATE] [-t TIMEWINDOW]
                          [-dir DIRPREFIX] [-threads THREADCOUNT]
                          [-max MAXPOSTS] [-drv DRIVERPATH]
                          [-drvProfile DRIVERPROFILEPREFIX]

Instagram Location Scraper

optional arguments:
  -h, --help            show this help message and exit
  -l LOCATION, --location LOCATION
                        Location Number to scrape, eg. -l 214335386/ for
                        Englischer Garten
  -c CITY, --city CITY  City to scrape location links from, eg. -c c579270/
                        for Munich
  -fromFile FILE TYPE   File containing a list of locations/cities to scrape,
                        specify with c or l, eg. -fromFile cities.txt c
  -fromDir DIR SUFFIX   Directory containing files with lists of locations to
                        scrape with suffix to specify which files to scrape,
                        eg. -fromDir ./data/Locations _Locations.txt
  -d DATE, --date DATE  Date up till which to scrape, eg. -d
                        2017-06-01T10:00:00 ; default: now
  -t TIMEWINDOW, --timeWindow TIMEWINDOW
                        Timeframe to check number of posts in hours, eg. -t
                        1.0 ; default: 1.0
  -dir DIRPREFIX, --dirPrefix DIRPREFIX
                        directory to save results to, eg. -dir ./data/ ;
                        default: ./data/
  -threads THREADCOUNT, --threadCount THREADCOUNT
                        how many threads (each one instance of chromedriver)
                        to use, eg. -threads 4 ; default: 1
  -max MAXPOSTS, --maxPosts MAXPOSTS
                        maximum number of posts to scrape, mainly due to
                        performance reasons, eg. -max 1500 ; default: -1
                        (unlimited)
  -drv DRIVERPATH, --driverPath DRIVERPATH
                        path to chromedriver, eg. -drv ./chromedriver.exe ;
                        default: ./chromedriver.exe
  -drvProfile DRIVERPROFILEPREFIX, --driverProfilePrefix DRIVERPROFILEPREFIX
                        prefix for scraper chrome profiles, eg -drvProfile
                        ./Scraper ; default: ./Scraper

Installation

download chromedriver here: https://sites.google.com/a/chromium.org/chromedriver/ either place chromedriver.exe in same directory as LocationCrawler.py, edit CHROMEDRIVER_PATH in LocationCrawler.py, or pass it as an argument with -drv (eg: "-drv ./chromedriver.exe")

then install the requirements (you need to have a python 3 distribution installed):

$ pip install -r requirements.txt