Skip to content

GateNLP/ultimate-sitemap-parser

Repository files navigation

Ultimate Sitemap Parser

PyPI - Python Version PyPI - Version Conda Version Pepy Total Downloads

Ultimate Sitemap Parser (USP) is a performant and robust Python library for parsing and crawling sitemaps.

Features

Installation

pip install ultimate-sitemap-parser

or using Anaconda:

conda install -c conda-forge ultimate-sitemap-parser

Usage

from usp.tree import sitemap_tree_for_homepage

tree = sitemap_tree_for_homepage('https://www.example.org/')

for page in tree.all_pages():
    print(page.url)

sitemap_tree_for_homepage() will return a tree of AbstractSitemap subclass objects that represent the sitemap hierarchy found on the website; see a reference of AbstractSitemap subclasses. AbstractSitemap.all_pages() returns a generator to efficiently iterate over pages without loading the entire tree into memory.

For more examples and details, see the documentation.