10-2_Scrapy_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7d43b5e-524e-4f46-8dcd-b6f74041b818",
   "metadata": {},
   "source": [
    "# Lecture 10-2\n",
    "\n",
    "# Webscraping with scrapy\n",
    "\n",
    "## Week 10 Wednesday\n",
    "\n",
    "## Miles Chen, PhD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9db388d8-6ef7-4b66-959d-e3cbe08cfd2e",
   "metadata": {},
   "source": [
    "# Resources\n",
    "\n",
    "- Scrapy documentation: https://docs.scrapy.org/en/latest/\n",
    "- Scrapy overview: https://docs.scrapy.org/en/latest/intro/overview.html\n",
    "- Scrapy tutorial: https://docs.scrapy.org/en/latest/intro/tutorial.html\n",
    "\n",
    "- github repository from scrapy workshop at PyCon US 2024: https://github.com/rennerocha/pyconus2024-tutorial\n",
    "- "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b40d151-a87c-4c22-a221-f2b9c071f80a",
   "metadata": {},
   "source": [
    "## installing scrapy\n",
    "\n",
    "from the command line, run: \n",
    "\n",
    "```\n",
    "pip install scrapy\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "641bfeb2-a5a4-43df-badf-6616ae1bf778",
   "metadata": {},
   "source": [
    "## Beginning a project\n",
    "\n",
    "Scrapy is controlled through the `scrapy` command-line tool\n",
    "\n",
    "First, you will have to set up a new Scrapy project. \n",
    "\n",
    "From the command line, run:\n",
    "```\n",
    "scrapy startproject scrapydemo\n",
    "```\n",
    "\n",
    "Where `scrapydemo` is the name of whatever scrapy projectal"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d509ab2-b545-436d-8473-020767f68302",
   "metadata": {},
   "source": [
    "After running this command, scrapy will create a folder called `scrapydemo` with files and additional folders and files inside.y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c20591af-7ce5-423c-acaf-a2a6a4d4cbe5",
   "metadata": {},
   "source": [
    "### Scrapy Project Structure\n",
    "\n",
    "The main components of a Scrapy project:\n",
    "\n",
    "+ `spiders/`: Directory containing spider definitions.\n",
    "+ `items.py`: Define data structures.\n",
    "+ `middlewares.py`: Handle requests and responses.\n",
    "+ `pipelines.py`: Post-process data (e.g., save to database).\n",
    "+ `settings.py`: Project settings and configurations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a602c32-fae5-472c-b9ad-6ed7cccc4cc9",
   "metadata": {},
   "source": [
    "Inside the `scrapydemo` folder, there you will find another folder called `scrapydemo` and inside that, there will be another folder called `spiders`. You will write python scripts inside the `spiders` folder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "563b5acf-bab3-4322-ae80-5473ec1381b7",
   "metadata": {},
   "source": [
    "### Using Scrapy\n",
    "\n",
    "To use scrapy, you will need to create a spider.\n",
    "\n",
    "Change into the directory you created.\n",
    "\n",
    "Run the command to create a spider script:\n",
    "\n",
    "`scrapy genspider <spidername> <domain to scrape>`\n",
    "\n",
    "We will scrape a site at http://quotes.toscrape.com. We will call our spider \"quotes\"\n",
    "\n",
    "```\n",
    "scrapy genspider quotes quotes.toscrape.com\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb9091e-aaba-4870-91d6-77bb80cc8495",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Edit the spider file\n",
    "\n",
    "Now modify the spider file.\n",
    "\n",
    "A spider file must have a `parse` method, which will be called on the response, which is the html from the server.\n",
    "\n",
    "Within the `parse` method, there should be a `yield` which produces a dictionary that defines the content to be stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bd378e63-5c9e-4cc9-8153-6955656d8894",
   "metadata": {
    "editable": true,
    "raw_mimetype": "",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Couldn't find program: 'false'\n"
     ]
    }
   ],
   "source": [
    "%%script false # this cell is not run, but contains a commented version of the quotes.py script\n",
    "import scrapy\n",
    "\n",
    "\n",
    "class QuotesSpider(scrapy.Spider):\n",
    "    # Defines a new class QuotesSpider that inherits from scrapy.Spider. This class will contain the logic for our spider.\n",
    "    name = \"quotes\"\n",
    "    # Sets the name of the spider to \"quotes\". This is how you will refer to this spider when running it from the command line.\n",
    "    allowed_domains = [\"quotes.toscrape.com\"]\n",
    "    # attribute is used to restrict the spider to only crawl URLs from specified domains.\n",
    "    # prevents the spider from accidentally crawling other websites\n",
    "    start_urls = [\n",
    "        'http://quotes.toscrape.com/page/1/',\n",
    "    ]\n",
    "    # A list of URLs where the spider will begin its crawling.\n",
    "    # In this case, it starts at the first page of the quotes website.\n",
    "    def parse(self, response):\n",
    "        # Defines the parse method, which will be called with the response object of each request made.\n",
    "        # This is where the main parsing logic of the spider resides.\n",
    "        for quote in response.css('div.quote'):\n",
    "            # Iterates over each div element with the class quote found in the response. \n",
    "            # This is where individual quotes are located on the page.\n",
    "            yield {\n",
    "                'text': quote.css('span.text::text').get(),\n",
    "                'author': quote.css('small.author::text').get(),\n",
    "                'tags': quote.css('div.tags a.tag::text').getall(),\n",
    "            }\n",
    "            # yield : Generates a dictionary containing the extracted data for each quote. \n",
    "            # The yield statement is used to return the data without stopping the spider.\n",
    "                # 'text': quote.css(...): Extracts the text of the quote by selecting the span element with the class text\n",
    "                # and retrieves its text content.\n",
    "                # 'author': quote.css(...): Extracts the author's name by selecting the small element\n",
    "                # with the class author and retrieves its text content.\n",
    "                # 'tags': quote.css(...),: Extracts all tags associated with the quote by selecting all a \n",
    "                # elements with the class tag within the div element with the class tags, and retrieves their text content as a list.\n",
    "\n",
    "        next_page = response.css('li.next a::attr(href)').get()\n",
    "        # next_page = response.css('li.next a::attr(href)').get(): \n",
    "                # Finds the URL of the next page by selecting the a element within the li element with the class next and retrieving its href attribute.\n",
    "        if next_page is not None:\n",
    "            yield response.follow(next_page, self.parse)\n",
    "        # if next_page is not None:: Checks if there is a next page.\n",
    "        # yield response.follow(next_page, self.parse): If there is a next page, \n",
    "                # the spider follows the link and calls the parse method on the response of the next page.\n",
    "                # This allows the spider to continue scraping subsequent pages."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c673b7fa-d244-4c48-a21a-7d07ae293630",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Running the spider\n",
    "\n",
    "Once you have the script written, you can execute it from the command line with:\n",
    "\n",
    "`scrapy crawl <spidername> -o <output_file_name>`\n",
    "\n",
    "We will run our spider called \"quotes\" and store the results in `quotes.csv`\n",
    "\n",
    "`scrapy crawl quotes -o quotes.csv `\n",
    "\n",
    "a lowercase `-o` will scrape and append to an existing file (or create a new file).\n",
    "\n",
    "an uppercase `-O` will scrape and replace / overwrite an existing file (or create a new file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63410a03-640e-4a90-b5b0-078e192a53bf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}