Skip to content

Small script designed to crawl a website for links. Designed to research spammy websites who market themselves as government support.

Notifications You must be signed in to change notification settings

mtworth/link_crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 

Repository files navigation

link_crawler

Small script designed to crawl a website for links. Designed to research spammy websites who market themselves as government support.

Input two parameters:

start_url = 'https://governmentassistanceonline.com/'
max_depth = 2

And returns dataframe of URLs and their main domain.

Maybe I'll make this into something someday, but there probably is already a package out there I'm missing.

About

Small script designed to crawl a website for links. Designed to research spammy websites who market themselves as government support.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published