13_bs4_requests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b3edeaf6-cf15-4876-8903-61232e581781",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# BeautifulSoup - Scrape emails from URL\n",
    "\n",
    "Largely inspired by https://github.com/jupyter-naas/awesome-notebooks/blob/master/BeautifulSoup/BeautifulSoup_Scrape_emails_from_URL.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5bdadc0-3346-4222-a74d-9a9237a2075f",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Description:** This notebook will show how to scrape emails stored in HTML webpage using BeautifulSoup."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14ecfcca-6d08-45fe-9806-a71fd35b5269",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<u>References:</u>\n",
    "- [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n",
    "- [Regular Expression Documentation](https://docs.python.org/3/library/re.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5b2693c-e417-4a03-ae5c-3c777de5c7db",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": "## Import libraries"
  },
  {
   "cell_type": "code",
   "id": "22494cd6-648e-4107-88ef-f446282e9c50",
   "metadata": {
    "papermill": {},
    "tags": [],
    "ExecuteTime": {
     "end_time": "2024-10-13T18:52:12.895620Z",
     "start_time": "2024-10-13T18:52:12.893260Z"
    }
   },
   "source": [
    "import re\n",
    "from collections import deque\n",
    "from urllib.parse import urlsplit\n",
    "\n",
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ],
   "outputs": [],
   "execution_count": 17
  },
  {
   "cell_type": "markdown",
   "id": "2128459d-e6e6-4c46-8e52-0c5bbeeb68c3",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup Variables\n",
    "- `url`: URL of the webpage to scrape\n",
    "- `limit`: number of emails found to stop scraping"
   ]
  },
  {
   "cell_type": "code",
   "id": "fc07a613-0018-403e-a1ca-0678f4e274c5",
   "metadata": {
    "papermill": {},
    "tags": [],
    "ExecuteTime": {
     "end_time": "2024-10-13T18:52:12.900418Z",
     "start_time": "2024-10-13T18:52:12.896751Z"
    }
   },
   "source": [
    "url = \"https://www.brentozar.com/\"\n",
    "limit = 3"
   ],
   "outputs": [],
   "execution_count": 18
  },
  {
   "cell_type": "markdown",
   "id": "68493b82-d57b-4509-a71c-45c198da4d77",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": "## Scrape emails from URL"
  },
  {
   "cell_type": "markdown",
   "id": "8893303c-85d5-4784-bc07-cf72ac4d655d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "We will use the `requests` library to get the HTML content of the webpage and the `BeautifulSoup` library to parse the HTML content. We will use a regular expression to extract the emails from the HTML content."
   ]
  },
  {
   "cell_type": "code",
   "id": "15f1e143-fbb8-4fac-98f0-afc3ad659343",
   "metadata": {
    "papermill": {},
    "tags": [],
    "ExecuteTime": {
     "end_time": "2024-10-13T18:53:48.865373Z",
     "start_time": "2024-10-13T18:52:12.901322Z"
    }
   },
   "source": [
    "unscraped = deque([url])\n",
    "\n",
    "scraped = set()\n",
    "\n",
    "emails = set()\n",
    "\n",
    "while len(unscraped):\n",
    "    url = unscraped.popleft()\n",
    "    scraped.add(url)\n",
    "\n",
    "    parts = urlsplit(url)\n",
    "\n",
    "    base_url = \"{0.scheme}://{0.netloc}\".format(parts)\n",
    "    if '/' in parts.path:\n",
    "        path = url[:url.rfind('/') + 1]\n",
    "    else:\n",
    "        path = url\n",
    "\n",
    "    print(\"Crawling URL: %s\" % url)\n",
    "    try:\n",
    "        response = requests.get(url)\n",
    "    except (requests.exceptions.MissingSchema, requests.exceptions.ConnectionError):\n",
    "        continue\n",
    "\n",
    "    exclude = [\"google.com\", \"gmail.com\", \"example.com\"]\n",
    "    # Get emails from URL\n",
    "    new_emails = re.findall(r\"[a-z0-9\\.\\-+_]+@[a-z0-9\\.\\-+_]+\\.+[a-z]{1,3}\", url)\n",
    "    for email in new_emails:\n",
    "        for e in exclude:\n",
    "            if not email.endswith(e):\n",
    "                emails.update([email])\n",
    "\n",
    "    # Get emails from content\n",
    "    new_emails = set(re.findall(r\"[a-z0-9\\.\\-+_]+@[a-z0-9\\.\\-+_]+\\.+[a-z]{1,3}\", response.text, re.I))\n",
    "    for email in new_emails:\n",
    "        for e in exclude:\n",
    "            if not email.endswith(e):\n",
    "                emails.update([email])\n",
    "\n",
    "    if len(emails) >= limit:\n",
    "        break\n",
    "\n",
    "    soup = BeautifulSoup(response.text, 'html.parser')\n",
    "    for anchor in soup.find_all(\"a\"):\n",
    "        if \"href\" in anchor.attrs:\n",
    "            link = anchor.attrs[\"href\"]\n",
    "        else:\n",
    "            link = ''\n",
    "\n",
    "        if link.startswith('/'):\n",
    "            link = base_url + link\n",
    "\n",
    "        elif not link.startswith('http'):\n",
    "            link = path + link\n",
    "\n",
    "        if not link.endswith(\".gz\"):\n",
    "            if not link in unscraped and not link in scraped:\n",
    "                unscraped.append(link)\n",
    "\n",
    "print(emails)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Crawling URL: https://www.brentozar.com/\n",
      "Crawling URL: https://www.brentozar.com/#\n",
      "Crawling URL: https://www.brentozar.com/log-in/\n",
      "Crawling URL: https://www.brentozar.com/contact/\n",
      "Crawling URL: https://www.brentozar.com/cart/\n",
      "Crawling URL: https://www.youtube.com/c/BrentOzarUnlimited\n",
      "Crawling URL: https://www.linkedin.com/in/brentozar/\n",
      "Crawling URL: https://www.facebook.com/brentozar\n",
      "Crawling URL: https://www.twitch.tv/brentozar\n",
      "Crawling URL: https://github.com/BrentOzarULTD\n",
      "Crawling URL: https://www.brentozar.com/sql-critical-care/\n",
      "Crawling URL: https://www.brentozar.com/sql/sql-server-performance-tuning/\n",
      "Crawling URL: https://www.brentozar.com/remote-dba-services-for-microsoft-sql-server/\n",
      "Crawling URL: https://www.brentozar.com/remote-dba-services-for-microsoft-sql-server/sql-server-upgrades-and-migrations/\n",
      "Crawling URL: https://training.brentozar.com/p/the-consultant-toolkit\n",
      "Crawling URL: https://www.brentozar.com/training/\n",
      "Crawling URL: https://www.brentozar.com/training/my-videos/\n",
      "Crawling URL: https://www.brentozar.com/my-account/\n",
      "Crawling URL: https://www.brentozar.com/corporate-training/\n",
      "Crawling URL: https://www.brentozar.com/training/training-faq/\n",
      "Crawling URL: https://www.brentozar.com/sql-constantcare/\n",
      "Crawling URL: https://www.brentozar.com/first-aid/\n",
      "Crawling URL: https://www.brentozar.com/blitz/\n",
      "Crawling URL: https://www.brentozar.com/blitzcache/\n",
      "Crawling URL: https://www.brentozar.com/askbrent/\n",
      "Crawling URL: https://www.brentozar.com/blitzindex/\n",
      "Crawling URL: https://www.brentozar.com/first-aid/sp_blitzwho/\n",
      "Crawling URL: https://www.brentozar.com/pastetheplan/\n",
      "Crawling URL: https://www.brentozar.com/blog/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/development/t-sql/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/development/t-sql/query-exercises/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/development/execution-plans/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/indexing/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/videos/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/architecture/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/backup-and-recovery/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/cloud-computers/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/development/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/first-responder-kit/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/high-availability/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/humor/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/development/locking-blocking-and-isolation-levels/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/development/parameter-sniffing/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/production-database-administration/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/professional-development/\n",
      "Crawling URL: https://www.brentozar.com/archive/category/sql-constantcare/\n",
      "Crawling URL: https://training.brentozar.com\n",
      "Crawling URL: https://www.brentozar.com/training/think-like-sql-server-engine/\n",
      "Crawling URL: https://training.brentozar.com/p/recorded-class-season-pass-fundamentals\n",
      "Crawling URL: https://training.brentozar.com/p/recorded-class-season-pass-masters-classes\n",
      "Crawling URL: https://www.brentozar.com/privacy-policy/\n",
      "Crawling URL: https://www.brentozar.com/training/refund-and-cancellation-policy-for-training/\n",
      "Crawling URL: https://www.brentozar.com/log-in/#\n",
      "Crawling URL: https://training.brentozar.com/sign_in\n",
      "Crawling URL: https://www.brentozar.com/my-account/lost-password/\n",
      "Crawling URL: https://www.brentozar.com/contact/#\n",
      "Crawling URL: https://www.brentozar.com/support/\n",
      "Crawling URL: https://www.brentozar.com/product/sponsorship/\n",
      "Crawling URL: https://www.tiktok.com/@brentozarultd\n",
      "Crawling URL: https://www.brentozar.com/contact/mailto:Help@BrentOzar.com\n",
      "Crawling URL: https://www.brentozar.com/cart/#\n",
      "Crawling URL: https://www.brentozar.com/shop/\n",
      "Crawling URL: https://www.youtube.com/\n",
      "Crawling URL: https://www.youtube.com/about/\n",
      "Crawling URL: https://www.youtube.com/about/press/\n",
      "Crawling URL: https://www.youtube.com/about/copyright/\n",
      "Crawling URL: https://www.youtube.com/t/contact_us/\n",
      "Crawling URL: https://www.youtube.com/creators/\n",
      "Crawling URL: https://www.youtube.com/ads/\n",
      "Crawling URL: https://developers.google.com/youtube\n",
      "Crawling URL: https://support.google.com/youtube/contact/de_cancellation?hl=fr\n",
      "{'8149a85a83fa4ec69640c43ddd69017d@sentry.io', 'asxvmprobertest@gmail.com', 'Help@BrentOzar.com'}\n"
     ]
    }
   ],
   "execution_count": 19
  },
  {
   "cell_type": "markdown",
   "id": "2c37e870-430c-4f33-8980-8da03df22cdf",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": "## Display result"
  },
  {
   "cell_type": "code",
   "id": "a788c4a5-1559-49ce-8b97-589dc4ef1b58",
   "metadata": {
    "papermill": {},
    "tags": [],
    "ExecuteTime": {
     "end_time": "2024-10-13T18:53:48.868477Z",
     "start_time": "2024-10-13T18:53:48.866352Z"
    }
   },
   "source": [
    "print(f\"🚀 {len(emails)} founded on {url}\")\n",
    "print(emails)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🚀 3 founded on https://support.google.com/youtube/contact/de_cancellation?hl=fr\n",
      "{'8149a85a83fa4ec69640c43ddd69017d@sentry.io', 'asxvmprobertest@gmail.com', 'Help@BrentOzar.com'}\n"
     ]
    }
   ],
   "execution_count": 20
  },
  {
   "cell_type": "markdown",
   "id": "7ec95b4b-a905-4a9c-9ab9-e0824c582302",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    " ### Exercice 1\n",
    " \n",
    "Développer un script qui permet de faire d'extraire les données sur les dividendes de l'action TOTAL SE en utilisant leur site web https://totalenergies.com/fr/actionnaires/action-et-dividende/dividende pour les années 2020 à 2024.\n",
    "Le script doit retourner un tableau de données avec les colonnes suivantes :\n",
    "- Nom du coupon\n",
    "- Montant, un `float` avec 2 décimales\n",
    "- Date de détachement, un objet `datetime`, parsé avec le module distant `dateparser`\n",
    "- Date de paiement, un objet `datetime`, parsé avec le module distant `dateparser`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "naas": {
   "notebook_id": "12ef03c9788da13b430b0d23171c6c0949ff23c86a70281d36e19d8a6237b135",
   "notebook_path": "BeautifulSoup/BeautifulSoup_Scrape_emails_from_URL.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.4.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}