Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scrapping 1000's of comments on Instagram #968

Open
Geminy3 opened this issue May 13, 2024 · 3 comments
Open

Scrapping 1000's of comments on Instagram #968

Geminy3 opened this issue May 13, 2024 · 3 comments
Labels
bug Something isn't working question Further information is requested

Comments

@Geminy3
Copy link

Geminy3 commented May 13, 2024

I'm getting an minet.instagram.exceptions.InstagramPublicAPIInvalidResponseError while trying to get all the comments from a post on Instagram, which interrupt the scrapping. I guess it's because there's a lot of comments on this post, because when I connect back to Instagram on a browser, I have a message for suspicial activities on my account.

I used this CL:
minet instagram comments URL -i urls_insta.csv -o comments.csv
With this csv

Is there any way to fix that error ? Or is anyone faced this same issue ?

Thanks !

@Yomguithereal
Copy link
Member

Are you using your Instagram account while the scraper is running? Or are you scraping multiple things at once using the same account. Instagram rate limiting is very fickle and there is no surefire way to make it work without it sometimes failing.

@Yomguithereal Yomguithereal added bug Something isn't working question Further information is requested labels May 15, 2024
@Geminy3
Copy link
Author

Geminy3 commented May 16, 2024

I opened a instagram page while the scrapper is running, but i'm not navigating while minet works. And I'm scrapping only comments from one post at a time, not trying to overload Instagram with a lot of different requests.
Is there any possibility to restart the scrapping from where it fails, meaning starting again at the last comment scraped ?
Thanks !

@Yomguithereal
Copy link
Member

Is there any possibility to restart the scrapping from where it fails, meaning starting again at the last comment scraped ?

It's probably doable because the pagination does not rely on transient ids but rather on id bounds (through a max_id GET parameter), but the nesting of the comment hierarchy between root-level comments and child comments might make this a bit perilous to implement correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants