scraper_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Webscraping example 2: KAREN MILLEN website"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Some importing \n",
    "import time\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from selenium import webdriver\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn; seaborn.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The URL we are trying to scrape is http://www.karenmillen.com/gb/womens/new-in/clothing/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "URL = 'http://www.karenmillen.com/gb/womens/new-in/clothing/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The problem with dynamic websites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dynamic websites contain Web pages that are generated in real-time. These pages include Web scripting code, such as Javascript, PHP or ASP. When a dynamic page is accessed, the code within the page is parsed on the Web server and the resulting HTML is sent to the client's Web browser.\n",
    "\n",
    "Because of a simpler scraper retrieve the content of a webpage as it is, it just not work well in this case. \n",
    "\n",
    "To overcome this, we can use Selenium, a toolkit to simulate a browser a drive it from your favorite programming language. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can download any (firefox/chrome/Edge) webdriver from these websites. We need the driver because selenium uses the web browser to scrape the data from the websites.\n",
    "\n",
    "https://sites.google.com/a/chromium.org/chromedriver/downloads\n",
    "\n",
    "https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/\n",
    "\n",
    "https://github.com/mozilla/geckodriver/releases\n",
    "\n",
    "\n",
    "Next step is put this driver executable in $PATH variable so it can be used from anywhere. To do so execute this command in terminal,\n",
    "\n",
    "For MAC/Linux user,\n",
    "\n",
    "```shell\n",
    "\"export PATH=$PATH:path/to/webdriver\" >> $HOME/.bash_profile\n",
    "```\n",
    "\n",
    "Now, the first step is to load the webdriver which we downloaded before. I will be using chrome webdriver for this example but you can use the same procedure for other web drivers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Selenium Chrome driver\n",
    "driver = webdriver.Chrome()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Open the URL\n",
    "driver.get(URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Close the driver\n",
    "driver.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Selenium to overcome the scroolling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def scroll_down():\n",
    "    script = \"window.scrollTo(0, document.body.scrollHeight); return document.body.scrollHeight;\"\n",
    "    len_page = driver.execute_script(script)\n",
    "    while True:\n",
    "        last_count = len_page\n",
    "        print('lenOfPage %d' % len_page)\n",
    "        time.sleep(5)\n",
    "        len_page = driver.execute_script(script)\n",
    "        if last_count == len_page:\n",
    "            break\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lenOfPage 2810\n",
      "lenOfPage 4795\n",
      "lenOfPage 6793\n",
      "lenOfPage 8769\n",
      "lenOfPage 10745\n",
      "lenOfPage 12743\n",
      "lenOfPage 14741\n",
      "lenOfPage 16739\n",
      "lenOfPage 18265\n"
     ]
    }
   ],
   "source": [
    "scroll_down()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the products details in the page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_product(el):\n",
    "    product = dict()\n",
    "\n",
    "    name = el.find_element_by_xpath(\".//a[@class='name-link']\")\n",
    "    if name:\n",
    "        product['name'] = name.text\n",
    "    else:\n",
    "        product['name'] = ''\n",
    "\n",
    "    price = el.find_element_by_xpath(\".//div[@class='product-price']\")\n",
    "    if price:\n",
    "        product['price'] = float(price.text.replace(\"£\", \"\").replace(\",\", \"\"))\n",
    "    else:\n",
    "        product['price'] = np.nan\n",
    "\n",
    "    return product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_elements():\n",
    "    return driver.find_elements_by_xpath(\".//div[@class='name-price-wrapper']\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-1\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-2\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-3\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-4\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-5\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-6\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-7\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-8\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-9\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-10\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-11\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-12\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-13\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-14\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-15\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-16\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-17\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-18\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-19\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-20\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-21\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-22\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-23\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-24\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-25\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-26\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-27\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-28\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-29\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-30\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-31\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-32\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-33\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-34\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-35\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-36\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-37\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-38\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-39\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-40\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-41\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-42\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-43\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-44\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-45\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-46\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-47\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-48\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-49\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-50\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-51\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-52\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-53\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-54\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-55\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-56\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-57\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-58\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-59\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-60\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-61\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-62\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-63\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-64\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-65\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-66\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-67\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-68\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-69\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-70\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-71\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-72\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-73\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-74\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-75\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-76\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-77\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-78\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-79\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-80\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-81\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-82\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-83\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-84\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-85\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-86\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-87\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-88\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-89\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-90\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-91\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-92\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-93\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-94\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-95\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-96\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-97\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-98\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-99\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-100\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-101\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-102\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-103\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-104\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"234fefffb1daf768ca92af128f9330d6\", element=\"0.9946675079520271-105\")>]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "items = get_elements()\n",
    "items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'FAUX FUR WRAP-AROUND COAT', 'price': 399.0}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_product(items[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting all together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_data(URL):\n",
    "    # Open the url\n",
    "    driver.get(URL)\n",
    "    time.sleep(5)\n",
    "\n",
    "    # Scroll page until end\n",
    "    scroll_down()\n",
    "\n",
    "    # Get all list of items\n",
    "    items = get_elements()\n",
    "\n",
    "    # Extract information for each item in the items list\n",
    "    products = []\n",
    "    for item in items:\n",
    "        products.append(get_product(item))\n",
    "\n",
    "    return products"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "driver = webdriver.Chrome()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lenOfPage 2810\n",
      "lenOfPage 4795\n",
      "lenOfPage 6793\n",
      "lenOfPage 8769\n",
      "lenOfPage 10745\n",
      "lenOfPage 12743\n",
      "lenOfPage 14741\n",
      "lenOfPage 16739\n",
      "lenOfPage 18265\n"
     ]
    }
   ],
   "source": [
    "p = get_data(URL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting the data in a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>FAUX FUR WRAP-AROUND COAT</td>\n",
       "      <td>399.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TAILORED CULOTTE JUMPSUIT</td>\n",
       "      <td>235.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ITALIAN WOOL MIX COAT</td>\n",
       "      <td>350.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>WOOL RICH TRENCH COAT</td>\n",
       "      <td>350.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GEO-PRINT A-LINE KNIT DRESS</td>\n",
       "      <td>140.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          name  price\n",
       "0    FAUX FUR WRAP-AROUND COAT  399.0\n",
       "1    TAILORED CULOTTE JUMPSUIT  235.0\n",
       "2        ITALIAN WOOL MIX COAT  350.0\n",
       "3        WOOL RICH TRENCH COAT  350.0\n",
       "4  GEO-PRINT A-LINE KNIT DRESS  140.0"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(p)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 105 entries, 0 to 104\n",
      "Data columns (total 2 columns):\n",
      "name     105 non-null object\n",
      "price    105 non-null float64\n",
      "dtypes: float64(1), object(1)\n",
      "memory usage: 1.7+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     105.000000\n",
       "mean      216.171429\n",
       "std       192.015948\n",
       "min        55.000000\n",
       "25%       110.000000\n",
       "50%       145.000000\n",
       "75%       250.000000\n",
       "max      1400.000000\n",
       "Name: price, dtype: float64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.price.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10eec08d0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGaCAYAAAD5HsxRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFiBJREFUeJzt3X1slfX9x+HPoQ04SjshqckWxIAPmcS4TRm4BMEt07pE\ns82ghZo6g9nU6bBGTJFHjU8QIouyOGXJtgR0G1Ozh2zJpsyMIUqM2zRW3aLZ2IbOodbYVgK1vX9/\nLHaTQvs5/GjPEa7rL3qfg/32bdEXd+G0VBRFEQAADGtMpQ8AAPBhIZwAAJKEEwBAknACAEgSTgAA\nSbWj8U527+4adG3ixPHR2fnuaLz7Dz1blcdeebYqj73ybFUee+WNxlaNjfUHfaxid5xqa2sq9a4/\ndGxVHnvl2ao89sqzVXnslVfprXypDgAgSTgBACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJ\nwgkAIEk4AQAkCScAgCThBACQJJwAAJKEEwBAknACAEgSTgAASbWVPsDhtHD1byt9hIiI+N6Sz1f6\nCADACHDHCQAgSTgBACQJJwCAJOEEAJCU+sPhX/nKV2LChAkRETF58uRobm6O22+/PWpqamL27Nlx\n7bXXjughAQCqwbDhtHfv3iiKIjZu3Dhw7Utf+lKsX78+jj/++Pj6178eL7zwQkyfPn1EDwoAUGnD\nfqnupZdeij179sTChQvjsssui6effjr27dsXU6ZMiVKpFLNnz47t27ePxlkBACpq2DtOxxxzTFxx\nxRVx8cUXx9/+9rf42te+Fg0NDQOP19XVxT/+8Y8h/xkTJ46P2tqaQdcbG+sP4cjVbyQ+riN1q5Fi\nrzxblcdeebYqj73yKrnVsOE0derUOOGEE6JUKsXUqVOjvr4+3n777YHHe3p6PhBSB9LZ+e6ga42N\n9bF7d9chHLn6He6P60jeaiTYK89W5bFXnq3KY6+80dhqqDAb9kt1Dz30UKxevToiIl5//fXYs2dP\njB8/Pv7+979HURSxbdu2mDFjxuE7LQBAlRr2jtO8efPipptuigULFkSpVIo77rgjxowZE4sXL46+\nvr6YPXt2fPKTnxyNswIAVNSw4TR27Ni46667Bl3fvHnziBwIAKBaeQFMAIAk4QQAkCScAACShBMA\nQJJwAgBIEk4AAEnCCQAgSTgBACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkAIEk4AQAk\nCScAgCThBACQJJwAAJKEEwBAknACAEgSTgAAScIJACBJOAEAJAknAIAk4QQAkCScAACShBMAQJJw\nAgBIEk4AAEnCCQAgSTgBACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkAIEk4AQAkCScA\ngCThBACQJJwAAJKEEwBAknACAEgSTgAAScIJACBJOAEAJAknAIAk4QQAkCScAACShBMAQJJwAgBI\nEk4AAEnCCQAgSTgBACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkAICkVTm+++WbMnTs3\nXnnlldi5c2csWLAgWlpaYtWqVdHf3z/SZwQAqArDhlNvb2+sXLkyjjnmmIiIuPPOO6OtrS0efPDB\nKIoitmzZMuKHBACoBsOG05o1a2L+/Plx3HHHRURER0dHzJw5MyIi5syZE9u3bx/ZEwIAVInaoR58\n5JFHYtKkSXH22WfHhg0bIiKiKIoolUoREVFXVxddXV3DvpOJE8dHbW3NoOuNjfWHcuaqNxIf15G6\n1UixV56tymOvPFuVx155ldxqyHB6+OGHo1QqxZNPPhkvvvhitLe3x1tvvTXweE9PTzQ0NAz7Tjo7\n3x10rbGxPnbvHj66PowO98d1JG81EuyVZ6vy2CvPVuWxV95obDVUmA0ZTg888MDAj1tbW+Pmm2+O\ntWvXxo4dO2LWrFmxdevWOOussw7fSQEAqljZL0fQ3t4e69evj+bm5ujt7Y2mpqaROBcAQNUZ8o7T\n/9q4cePAjzdt2jQihwEAqGZeABMAIEk4AQAkCScAgCThBACQJJwAAJKEEwBAknACAEgSTgAAScIJ\nACBJOAEAJAknAIAk4QQAkCScAACShBMAQJJwAgBIEk4AAEnCCQAgSTgBACQJJwCAJOEEAJAknAAA\nkoQTAECScAIASBJOAABJwgkAIEk4AQAkCScAgCThBACQJJwAAJKEEwBAknACAEgSTgAAScIJACBJ\nOAEAJAknAIAk4QQAkCScAACShBMAQJJwAgBIEk4AAEnCCQAgSTgBACQJJwCAJOEEAJAknAAAkoQT\nAECScAIASBJOAABJwgkAIEk4AQAkCScAgCThBACQJJwAAJKEEwBAknACAEgSTgAAScIJACBJOAEA\nJAknAIAk4QQAkCScAACShBMAQJJwAgBIqh3uCX19fbF8+fL461//GqVSKW655ZYYN25cLFmyJEql\nUpx88smxatWqGDNGgwEAR7Zhw+nxxx+PiIgf/ehHsWPHjvjWt74VRVFEW1tbzJo1K1auXBlbtmyJ\nc889d8QPCwBQScPeJvrCF74Qt956a0REvPrqq9HQ0BAdHR0xc+bMiIiYM2dObN++fWRPCQBQBYa9\n4xQRUVtbG+3t7fHoo4/GPffcE0888USUSqWIiKirq4uurq4hf/7EieOjtrZm0PXGxvpDOHL1G4mP\n60jdaqTYK89W5bFXnq3KY6+8Sm6VCqeIiDVr1sTixYvjkksuib179w5c7+npiYaGhiF/bmfnu4Ou\nNTbWx+7dQwfXh9Xh/riO5K1Ggr3ybFUee+XZqjz2yhuNrYYKs2G/VPfTn/407r///oiI+MhHPhKl\nUilOO+202LFjR0REbN26NWbMmHGYjgoAUL2GveN03nnnxU033RSXXnppvPfee7F06dI48cQTY8WK\nFbFu3bqYNm1aNDU1jcZZAQAqathwGj9+fNx9992Drm/atGlEDgQAUK28+BIAQJJwAgBIEk4AAEnC\nCQAgSTgBACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkAIEk4AQAkCScAgCThBACQJJwA\nAJKEEwBAknACAEgSTgAAScIJACBJOAEAJAknAIAk4QQAkCScAACShBMAQJJwAgBIEk4AAEnCCQAg\nSTgBACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkAIEk4AQAkCScAgCThBACQJJwAAJKE\nEwBAknACAEgSTgAAScIJACBJOAEAJAknAIAk4QQAkCScAACShBMAQJJwAgBIEk4AAEnCCQAgSTgB\nACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJtZU+wJFo4erfVvoI8b0ln6/0EQDgiOOOEwBA\nknACAEgSTgAASUP+Gafe3t5YunRp7Nq1K/bt2xdXX311nHTSSbFkyZIolUpx8sknx6pVq2LMGP0F\nABz5hgynn//853HsscfG2rVr4+23344vf/nL8YlPfCLa2tpi1qxZsXLlytiyZUuce+65o3VeAICK\nGfJW0fnnnx/XXXddREQURRE1NTXR0dERM2fOjIiIOXPmxPbt20f+lAAAVWDIO051dXUREdHd3R2L\nFi2Ktra2WLNmTZRKpYHHu7q6hn0nEyeOj9ramkHXGxvrD+XMJBzt2x7tH385bFUee+XZqjz2yqvk\nVsO+jtNrr70W11xzTbS0tMSFF14Ya9euHXisp6cnGhoahn0nnZ3vDrrW2Fgfu3cPH10cmqN5W59b\nebYqj73ybFUee+WNxlZDhdmQX6p74403YuHChXHjjTfGvHnzIiJi+vTpsWPHjoiI2Lp1a8yYMeMw\nHhUAoHoNGU733XdfvPPOO3HvvfdGa2trtLa2RltbW6xfvz6am5ujt7c3mpqaRuusAAAVNeSX6pYv\nXx7Lly8fdH3Tpk0jdiAAgGrlBZgAAJKEEwBAknACAEgSTgAAScIJACBJOAEAJAknAIAk4QQAkCSc\nAACShBMAQJJwAgBIEk4AAEnCCQAgSTgBACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkA\nIEk4AQAkCScAgCThBACQJJwAAJKEEwBAknACAEgSTgAAScIJACBJOAEAJAknAIAk4QQAkCScAACS\nhBMAQJJwAgBIEk4AAEnCCQAgqbbSB2BkLFz920ofISIivrfk85U+AgAcNu44AQAkCScAgCThBACQ\nJJwAAJKEEwBAknACAEjycgSMKC+LAMCRxB0nAIAk4QQAkCScAACShBMAQJJwAgBIEk4AAEnCCQAg\nSTgBACQJJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkAIEk4AQAkCScAgCThBACQlAqnZ599\nNlpbWyMiYufOnbFgwYJoaWmJVatWRX9//4geEACgWgwbTt/97ndj+fLlsXfv3oiIuPPOO6OtrS0e\nfPDBKIoitmzZMuKHBACoBsOG05QpU2L9+vUDb3d0dMTMmTMjImLOnDmxffv2kTsdAEAVqR3uCU1N\nTfHPf/5z4O2iKKJUKkVERF1dXXR1dQ37TiZOHB+1tTWDrjc21pdzVjhkPtcOzjblsVeercpjr7xK\nbjVsOO1vzJj/3qTq6emJhoaGYX9OZ+e7g641NtbH7t3DRxccDj7XDsyvw/LYK89W5bFX3mhsNVSY\nlf236qZPnx47duyIiIitW7fGjBkzDv1kAAAfImWHU3t7e6xfvz6am5ujt7c3mpqaRuJcAABVJ/Wl\nusmTJ8fmzZsjImLq1KmxadOmET0UAEA18gKYAABJwgkAIEk4AQAkCScAgCThBACQJJwAAJKEEwBA\nknACAEgSTgAAScIJACBJOAEAJAknAIAk4QQAkCScAACShBMAQJJwAgBIEk4AAEnCCQAgSTgBACQJ\nJwCAJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkAIEk4AQAkCScAgCThBACQJJwAAJKEEwBAknAC\nAEgSTgAAScIJACBJOAEAJAknAIAk4QQAkCScAACShBMAQJJwAgBIEk4AAEnCCQAgSTgBACQJJwCA\nJOEEAJAknAAAkoQTAECScAIASBJOAABJwgkAIEk4AQAkCScAgKTaSh8ARsPC1b+t9BEiIuJ7Sz5f\n6SMA8P/gjhMAQJJwAgBIEk4AAEnCCQAgSTgBACQJJwCAJC9HAFSEl4j4oGrZo1pUy78X/qtaPkd/\ncdeXKvr+3XECAEgSTgAAScIJACDpkP6MU39/f9x8883x5z//OcaOHRu33XZbnHDCCYf7bAAAVeWQ\n7jg99thjsW/fvvjxj38cN9xwQ6xevfpwnwsAoOocUjg988wzcfbZZ0dExKc+9al4/vnnD+uhAACq\nUakoiqLcn7Rs2bI477zzYu7cuRERcc4558Rjjz0WtbVe3QAAOHId0h2nCRMmRE9Pz8Db/f39ogkA\nOOIdUjidccYZsXXr1oiI+NOf/hSnnHLKYT0UAEA1OqQv1b3/t+r+8pe/RFEUcccdd8SJJ544EucD\nAKgahxROAABHIy+ACQCQJJwAAJKEEwBA0qiHU39/f6xcuTKam5ujtbU1du7cOdpHqDq9vb1x4403\nRktLS8ybNy+2bNkSO3fujAULFkRLS0usWrUq+vv7IyLi29/+dsybNy/mz58fzz33XIVPXllvvvlm\nzJ07N1555RV7DeH++++P5ubmuOiii+InP/mJrYbQ29sbN9xwQ8yfPz9aWlp8bh3Es88+G62trRER\nZe1zsOce6f53rxdffDFaWlqitbU1rrjiinjjjTciImLz5s1x0UUXxSWXXBKPP/54RES89dZbsXDh\nwmhpaYm2trbYs2dPxT6G0fK/W73vF7/4RTQ3Nw+8XfGtilH261//umhvby+Koij++Mc/FlddddVo\nH6HqPPTQQ8Vtt91WFEVRdHZ2FnPnzi2uvPLK4qmnniqKoihWrFhR/OY3vymef/75orW1tejv7y92\n7dpVXHTRRZU8dkXt27ev+MY3vlGcd955xcsvv2yvg3jqqaeKK6+8sujr6yu6u7uLe+65x1ZDePTR\nR4tFixYVRVEU27ZtK6699lp77WfDhg3FBRdcUFx88cVFURRl7XOg5x7p9t/r0ksvLV544YWiKIri\nhz/8YXHHHXcU//73v4sLLrig2Lt3b/HOO+8M/PjWW28tHn744aIoiuL+++8vvv/971fqwxgV+29V\nFEXR0dFRXHbZZQPXqmGrUb/j5Nu1DHb++efHddddFxERRVFETU1NdHR0xMyZMyMiYs6cObF9+/Z4\n5plnYvbs2VEqleLjH/949PX1xVtvvVXJo1fMmjVrYv78+XHcccdFRNjrILZt2xannHJKXHPNNXHV\nVVfFOeecY6shTJ06Nfr6+qK/vz+6u7ujtrbWXvuZMmVKrF+/fuDtcvY50HOPdPvvtW7dujj11FMj\nIqKvry/GjRsXzz33XHz605+OsWPHRn19fUyZMiVeeumlD/z/8mjYa/+tOjs7Y926dbF06dKBa9Ww\n1aiHU3d3d0yYMGHg7ZqamnjvvfdG+xhVpa6uLiZMmBDd3d2xaNGiaGtri6IoolQqDTze1dU1aLv3\nrx9tHnnkkZg0adLAL5KIsNdBdHZ2xvPPPx9333133HLLLbF48WJbDWH8+PGxa9eu+OIXvxgrVqyI\n1tZWe+2nqanpA98popx9DvTcI93+e73/m70//OEPsWnTprj88suju7s76uvrB55TV1cX3d3dH7h+\nNOz1v1v19fXFsmXL4qabboq6urqB51TDVqP+fVJ8u5YDe+211+Kaa66JlpaWuPDCC2Pt2rUDj/X0\n9ERDQ8Og7Xp6ej7wCXS0ePjhh6NUKsWTTz4ZL774YrS3t3/gd/v2+q9jjz02pk2bFmPHjo1p06bF\nuHHj4l//+tfA47b6oB/84Acxe/bsuOGGG+K1116Lr371q9Hb2zvwuL0GGzPmv7//Hm6fAz33aPSr\nX/0qvvOd78SGDRti0qRJB93r/evHHHPMUbdXR0dH7Ny5M26++ebYu3dvvPzyy3H77bfHWWedVfGt\nRv2Ok2/XMtgbb7wRCxcujBtvvDHmzZsXERHTp0+PHTt2RETE1q1bY8aMGXHGGWfEtm3bor+/P159\n9dXo7++PSZMmVfLoFfHAAw/Epk2bYuPGjXHqqafGmjVrYs6cOfY6gDPPPDN+//vfR1EU8frrr8ee\nPXvis5/9rK0OoqGhYSCAPvrRj8Z7773n1+IwytnnQM892vzsZz8b+O/X8ccfHxERp59+ejzzzDOx\nd+/e6OrqildeeSVOOeWUOOOMM+J3v/tdRPxnrzPPPLOSRx9Vp59+evzyl7+MjRs3xrp16+Kkk06K\nZcuWVcVWo36r59xzz40nnngi5s+fP/DtWo529913X7zzzjtx7733xr333hsREcuWLYvbbrst1q1b\nF9OmTYumpqaoqamJGTNmRHNz88DfTuQ/2tvbY8WKFfbaz+c+97l4+umnY968eVEURaxcuTImT55s\nq4O4/PLLY+nSpdHS0hK9vb1x/fXXx2mnnWavIZTza+9Azz2a9PX1xe233x4f+9jH4pvf/GZERHzm\nM5+JRYsWRWtra7S0tERRFHH99dfHuHHj4uqrr4729vbYvHlzTJw4Me66664KfwSV19jYWPGtfMsV\nAIAkL4AJAJAknAAAkoQTAECScAIASBJOAABJwgkAIEk4AQAk/R8/7QQ+J+jhZAAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10eebf5f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.price.hist(bins=15, figsize=(10,7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closing the driver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "driver.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}