[known issue] the redundant data in the search results

[ax4]

As we observe, redundant data may occur, due to the following reasons:

• Sending search request too often
• weixin.sogou sometimes gives redundant response

To solve:

1. use a statics package, to observe the redundant rate
2. modify the crawler strategy in terms of redundant rate