Add max url page request limit

[zerokeeper]

Please describe your feature request:

add -ul/-url-limit (default 1000) to set katana crawl max url page in every site,avoid to crawl long time in some sites.

Describe the use case of this feature:

katana -u https://example.com -ul 1000

[Mzack9999]

With this feature you really risk to lose a lot of results that might be in the upper range of the threshold. Have you tried playing with the depth parameter? If you are interested in a rapid flyover without too much nesting you can limit the crawling to a depth of 2 or 3 with standard crawling.