[Request] Add robots.txt parsing

[joshua-bn]

Would be nice to have the ability to parse robots.txt like RSS feeds. $web->robots

https://github.com/bopoda/robots-txt-parser is a library. Not sure if it is the one to use here but it seems to do the job

[spekulatius]

Yeah, that's something to consider. I would opt for https://github.com/spatie/robots-txt instead as it's better maintained. What exactly do you want to achieve with the information?

[joshua-bn]

Personally, I am looking for sitemaps declared in robots.txt but I think there's also value in checking for rules for crawling.

[spekulatius]

Fair enough, that's definitely another use-case. I'll see how we can get both working

…

On Thu, Jan 12, 2023, 15:58 Joshua Dickerson ***@***.***> wrote: Personally, I am looking for sitemaps declared in robots.txt but I think there's also value in checking for rules for crawling. — Reply to this email directly, view it on GitHub <#177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAK7M45YFZADMOUK6LOEHLWSALZZANCNFSM6AAAAAATW5RGTE> . You are receiving this because you commented.Message ID: ***@***.***>