v4.7

• v4.7

  • New

    • BUG FIX: If an input domain has unicode in, e.g xñl.uk, then it will be converted to the punycode version, e.g. xn--xl-zja.uk to use that as the input instead. This will ensure the URLs and responses are correctly retrieved from the archive sources.

v4.6

• v4.6

  • New

    • Add argument -ft to specify a list of MIME Types to filter. This will override the FILTER_MIME list in config.yml. NOTE: This will NOT be applied to Alien Vault OTX and Virus Total because they don't have the ability to filter on MIME Type. Sometimes URLScan does not have a MIME Type defined - these will always be included. Consider excluding sources if this matters to you..
    • Add argument -mt to specify a list of MIME Types to match. This will be used instead of the default filtering using FILTER_MIME list in config.yml, or filtering using -ft. NOTE: This will NOT be applied to Alien Vault OTX and Virus Total because they don't have the ability to filter on MIME Type. Sometimes URLScan does not have a MIME Type defined - these will always be included. Consider excluding sources if this matters to you..
    • Add argument --providers in the same way as gau. A comma separated list of source providers that you want to get URLs from. The values can be wayback,commoncrawl,otx,urlscan and virustotal. Passing this will override any exclude arguments (e.g. -xwm,-xcc, etc.) passed to exclude sources, and reset those based on what was passed with this argument.

  • Changed

    • When argument --verbose has been used and the options are shown, show the name of providers that will be searched instead of the exclude arguments, e.g.-xwm, -xcc, etc.
    • Change HTTP_ADAPTER_CC used for Common Crawl requests to use retries+3 instead of reties+20. This was originally suggested by Common Crawl, but there are so many issues it can just take forever to get anything from their API, and often fail anyway.
    • Change the default of -lcc to 1 instead of 3 because of so many problems with Common Crawl.
    • BUG FIX: If a connection error occurs when getting the Common Crawl index file, then error ERROR getCommonCrawlUrls 1: object of type 'NoneType' has no len() is displayed. This will now be suppressed.
    • BUG FIX: If arg -mc was not passed and -ft was, when options were shown to the user (in showOptions function), the value of -mc was shown for -ft.
    • BUG FIX: When a MIME type is used in a filter for Wayback Machine that has a + in it (e.g. image/svg+xml), then the + was replaced because that'#s the only way Wayback recognises it. However, it was being escaped first and was being converted to image/svg\.xml instead of image/svg.xml so was not recognised in the filter.

v4.5

• Change

  • BUG FIX: When -f/--filter-responses-only is used, and retrieving Wayback Archive links, the links were still being filtered for URL exclusions, e.g. the extensions. This has been fixed and should return more links in that situation.
  • BUG FIX: If there is an invalid response from Alien Vault, then the error ERROR: getAlienVaultUrls 1: Expecting value: line 1 column 1 (char 0) is raised. This will be handled properly.
  • BUG FIX: If there is an invalid response from URLScan, then the error ERROR getURLScanUrls 1: local variable 'jsonResp' referenced before assignment is raised. This will be handled properly.
  • BUG FIX: If there is an invalid response from Virus Total, then the error ERROR getVirusTotalUrls 1: Expecting value: line 1 column 1 (char 0) is raised. This will be handled properly.
  • BUG FIX: When retrieving links from the Wayback Archive, and the user presses Ctrl-C to cancel the program, the error [ ERR ] Error getting response for page - local variable 'resp' referenced before assignment was displayed. This will no longer be shown.

v4.4

• New

  • When using -mode R, if input was used that does find results, but then those reults don't match the input given, then display a message. For example, if input is www.hackerone.com/xnl then wayback machine returns links for http://hackerone.com/xnl (without the www.). These don't match so aren't returned, but a message will give the user and clue as to what to change the input to if they did want those.

• Changed

  • BUG FIX:Rewrite the logic in linksFoundAdd and correct a typo that always made a runtime error occur and always add a link, without doing the check to see if the domain matches what was searched for (it's rare other URLs are included anyway). Also use new linksFoundResponseAdd with similar logic, but remove the prefixed timestamp which occurs with response links.
  • BUG FIX: If a URL is passed (instead of just a domain) as input for -mode R to download archived responses, it would not download anything because it would check the result contains the input, but the default port number is included in wayback results, but not included in the input. This has been corrected.
  • Remove argparse from setup.py and requirements.txt because it is a standard Python module.

v4.3

See CHANGELOG

v4.2

BUG FIXES: See CHANGELOG

v4.1

Bug fix - remove reference to tqdm library

v4.0

See CHANGELOG for details

v3.7

See CHANGELOG

v3.5

Minor non functional changes. See CHANGELOG