Scrapy project

All

27 repositories

scrapy
Public
Scrapy, a fast high-level web crawling & scraping framework for Python.
python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python
Python
•
BSD 3-Clause "New" or "Revised" License
•11k•54k•443•197•Updated Jan 2, 2025Jan 2, 2025
form2request
Public
Python 3.8+ library to build HTTP requests out of HTML forms
Python
•
BSD 3-Clause "New" or "Revised" License
•3•4•2•0•Updated Dec 24, 2024Dec 24, 2024
scrapyd
Public
A service daemon to run Scrapy spiders
Python
•
BSD 3-Clause "New" or "Revised" License
•570•3k•7•0•Updated Dec 18, 2024Dec 18, 2024
parsel
Public
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
css python xml scraping selectors xpath lxml hacktoberfest
Python
•
BSD 3-Clause "New" or "Revised" License
•147•1.2k•30•12•Updated Dec 16, 2024Dec 16, 2024
itemadapter
Public
Common interface for data container classes
python metadata python3 scrapy hacktoberfest python-dataclasses python-attrs
Python
•
BSD 3-Clause "New" or "Revised" License
•13•61•6•4•Updated Dec 12, 2024Dec 12, 2024
scrapy.org
Public
The scrapy.org website
html hacktoberfest
HTML
•140•60•1•1•Updated Dec 9, 2024Dec 9, 2024
protego
Public
A pure-Python robots.txt parser with support for modern conventions.
python robots-txt hacktoberfest robots-parser
DIGITAL Command Language
•
BSD 3-Clause "New" or "Revised" License
•28•57•5•1•Updated Nov 15, 2024Nov 15, 2024
w3lib
Public
Python library of web-related functions
python hacktoberfest
Python
•
BSD 3-Clause "New" or "Revised" License
•105•394•11•4•Updated Oct 16, 2024Oct 16, 2024
itemloaders
Public
Library to populate items using XPath and CSS with a convenient API
hacktoberfest
Python
•
BSD 3-Clause "New" or "Revised" License
•16•45•17•4•Updated Oct 16, 2024Oct 16, 2024
cssselect
Public
CSS Selectors for Python
css python selectors hacktoberfest
Python
•
Other
•61•292•17•4•Updated Oct 16, 2024Oct 16, 2024
queuelib
Public
Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python
python python3 persistent hacktoberfest queues non-persistent
Python
•
BSD 3-Clause "New" or "Revised" License
•54•273•3•2•Updated Oct 16, 2024Oct 16, 2024
scrapyd-client
Public
Command line client for Scrapyd server
Python
•
BSD 3-Clause "New" or "Revised" License
•145•769•5•0•Updated Oct 11, 2024Oct 11, 2024
loginform
Public
Fill HTML login forms automatically
Python
•79•270•11•3•Updated Apr 24, 2024Apr 24, 2024
xtractmime
Public
https://mimesniff.spec.whatwg.org/ implementation for Python
Python
•
BSD 3-Clause "New" or "Revised" License
•2•14•0•0•Updated Jan 16, 2024Jan 16, 2024
quotesbot
Public
This is a sample Scrapy project for educational purposes
Python
•
MIT License
•778•1.3k•0•6•Updated Nov 29, 2023Nov 29, 2023
booksbot
Public
A crawler for http://books.toscrape.com
Python
•946•40•0•4•Updated Aug 8, 2023Aug 8, 2023
scrapely
Public
A pure-python HTML screen-scraping library
HTML
•272•1.9k•26•6•Updated Apr 4, 2022Apr 4, 2022
scrapy-bench
Public
A CLI for benchmarking Scrapy.
python web-crawler scrapy command-line-tool benchmark-suite scrapy-bench
Python
•
MIT License
•15•31•6•2•Updated Feb 24, 2021Feb 24, 2021
scrapy-itemloader
Public archive
[Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
Python
•
BSD 3-Clause "New" or "Revised" License
•7•6•2•0•Updated May 5, 2020May 5, 2020
scurl
Public
Performance-focused replacement for Python urllib
python cython chromium gurl urlparse
Python
•
Apache License 2.0
•6•21•10•0•Updated Oct 2, 2018Oct 2, 2018
url-chromium
Public
url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url
chromium gurl
C++
•
BSD 3-Clause "New" or "Revised" License
•2•2•0•0•Updated Aug 7, 2018Aug 7, 2018
base-chromium
Public
base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
C++
•
BSD 3-Clause "New" or "Revised" License
•3•7•0•0•Updated Jul 31, 2018Jul 31, 2018
dirbot
Public
Scrapy project to scrape public web directories (educational) [DEPRECATED]
Python
•1.1k•1.6k•0•0•Updated Oct 27, 2017Oct 27, 2017
scrapy-bench-speedcenter
Public
Codespeed for scrapy-bench
Python
•2•2•0•0•Updated Aug 28, 2017Aug 28, 2017
pypydispatcher
Public
A fork of http://pydispatcher.sourceforge.net/ with PyPy support
Python
•
Other
•4•16•1•0•Updated Jul 3, 2017Jul 3, 2017
slybot
Public
57•224•5•0•Updated Apr 27, 2015Apr 27, 2015
gsoc2014-integration-tests
Public
GSoC2014 - Scrapy Integration tests project
Shell
•3•3•0•0•Updated Mar 18, 2014Mar 18, 2014