Skip to content

hoogle needs to be less botcrawlable #442

@gbaz

Description

@gbaz

In particular, bots and crawlers (which the ai bubble has thrown up in great numbers) seem to get lost in a maze of links when they search for something like "id" then just keep exploring all of the is:[pkg] left hand links. adding nofollow might help, but these are lawless bots and i doubt it.

suggest that if there is is:[pkg] in the query string already, then no further left hand package links be shown, so there's not a branching maze path that traps bots.

relevant code: https://github.com/ndmitchell/hoogle/blob/master/src/Action/Server.hs#L203

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions