New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[known issue] the redundant data in the search results #6

Open

ax4 opened this issue Jun 13, 2017 · 0 comments

Owner

ax4 commented Jun 13, 2017 •

edited

Loading

As we observe, redundant data may occur, due to the following reasons:

Sending search request too often
weixin.sogou sometimes gives redundant response

To solve:

use a statics package, to observe the redundant rate
modify the crawler strategy in terms of redundant rate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment