Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

403 Client Error - ReserveCalifornia blocked on cloud hosting providers #319

Open
DavydLiu opened this issue Jan 31, 2024 · 15 comments
Open
Assignees
Labels
bug Something isn't working triage This will looked at

Comments

@DavydLiu
Copy link

Describe the bug
I got the following error when I run the below command from a linode machine:

HTTPError: 403 Client Error: Forbidden for url: https://calirdr.usedirect.com/rdr/rdr/search/filters

Original Camply Command (with --debug)

camply campsites --provider ReserveCalifornia --campground 597 --campground 598 --start-date 2024-02-10 --end-date 2024-02-11 --search-forever  --notifications pushover --polling-interval 5 --debug

Expected behavior
Running the same command on my laptop works fine

Console Output (with --debug)

[2024-01-31 02:00:45] CAMPLY camply, the campsite finder ⛺️
[2024-01-31 02:00:45] DEBUG Setting up camply debugging
[2024-01-31 02:00:45] DEBUG Camply Version: 0.32.1
[2024-01-31 02:00:45] DEBUG Python Version: 3.11.6
[2024-01-31 02:00:45] DEBUG Platform: linux
[2024-01-31 02:00:45] INFO Using Camply Provider: "ReserveCalifornia"
[2024-01-31 02:00:45] INFO 1 booking nights selected for search, ranging from 2024-02-10 to 2024-02-10
[2024-01-31 02:00:45] DEBUG Refreshing UseDirect Metadata: filters.json
[2024-01-31 02:00:45] DEBUG Starting new HTTPS connection (1): calirdr.usedirect.com:443
[2024-01-31 02:00:45] DEBUG https://calirdr.usedirect.com:443 "GET /rdr/rdr/search/filters HTTP/1.1" 403 520
[2024-01-31 02:00:45] CAMPLY Exiting camply 👋

Additional context

@DavydLiu DavydLiu added the bug Something isn't working label Jan 31, 2024
@github-actions github-actions bot added the triage This will looked at label Jan 31, 2024
@juftin
Copy link
Owner

juftin commented Jan 31, 2024

Oh no! I'll take a look. Hopefully ReserveCalifornia isn't cracking down on tools like camply

@juftin
Copy link
Owner

juftin commented Jan 31, 2024

EDIT: I'm seeing that this is only occurring when coming from linode servers and not from local. The command from above worked for me as well.

This might just be an unfortunate blocking of linode IP addresses by the ReserveCalifornia servers. The best I can do in this case is update camply's user agents to a more recent version of fake-useragent

@DavydLiu
Copy link
Author

I actually tried Linode and DigitalOcean, both had the same issue.

@DavydLiu
Copy link
Author

DavydLiu commented Feb 4, 2024

Tried vultr this morning, still the same error.

@juftin juftin changed the title Encountered 403 Client Error for usedirect.com 403 Client Error - ReserveCalifornia blocked on cloud hosting providers Feb 5, 2024
juftin added a commit that referenced this issue Feb 6, 2024
related to #319 

---------

Co-authored-by: Justin Flannery <[email protected]>
github-actions bot added a commit that referenced this issue Feb 6, 2024
# [v0.32.2](v0.32.1...v0.32.2) (2024-02-06)

## 🐛 Bug Fixes
- [`731e55e`](731e55e)  fake-useragent update (Issues: [`#319`](#319))

[skip ci]
@juftin
Copy link
Owner

juftin commented Feb 6, 2024

😞 I've updated fake-useragent to the latest version and enforced a random User-Agent for each request to ReserveCalifornia. Sadly this is the best I think I'll be able to do - I wouldn't be surprised if ReserveCalifornia universally blocks all major cloud hosting provider's IP ranges.

Let me know if this resolves the issue for you - if not you may want to consider a VPN or similar solution to mask your IP address.

@DavydLiu
Copy link
Author

DavydLiu commented Feb 6, 2024

Thanks, let me git it a try. I'm also curious which cloud provider you'd recommend for hosting camply?

@juftin
Copy link
Owner

juftin commented Feb 6, 2024

I don't have a recommendation sadly - my cloud provider is a small linux server running in my basement that I also use for my homelab. A raspberry pi would probably also work nicely.

This is actually the first I've heard of this cloud provider blocking issue so you'd probably only run into this problem on ReserveCalifornia and not the other camply providers (for now).

@DavydLiu
Copy link
Author

DavydLiu commented Feb 7, 2024

Just tried 0.32.2. Unfortunately, it's still blocked for HTTPError: 403 Client Error: Forbidden for url: https://calirdr.usedirect.com/rdr/rdr/search/filters. Recreation.gov works perfectly. I'm curiously what ReserveCalifornia does differently.

Raspberry PI will be my last resort.

@justin-outdoorithm
Copy link

I had this issue on Digital Ocean and ended up using a proxy

@DavydLiu
Copy link
Author

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

@justin-outdoorithm
Copy link

Smartproxy

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

Smartproxy

@DavydLiu
Copy link
Author

DavydLiu commented Apr 1, 2024

Smartproxy

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

Smartproxy

Thanks, Justin, do you mind share which proxy you were using. Took a look yesterday but unsure which product I should buy.

@justin-outdoorithm
Copy link

justin-outdoorithm commented Apr 11, 2024 via email

@botlahan
Copy link

botlahan commented Aug 2, 2024

I've been using the Residential Pay As You Go plan

On Mon, Apr 1, 2024 at 10:12 AM David Liu @.> wrote: Smartproxy I had this issue on Digital Ocean and ended up using a proxy Which proxy did you use? Smartproxy Thanks, Justin, do you mind share which proxy you were using. Took a look yesterday but unsure which product I should buy. — Reply to this email directly, view it on GitHub <#319 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCRKPZBP5OIRUD4TOY7Z6XDY3GIPJAVCNFSM6AAAAABCSIENPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGE3TONZVGQ . You are receiving this because you commented.Message ID: @.>

I’m having the same issue while trying to use camply on a Google cloud virtual console. Any idea on how to setup a third party residential proxy such as smartproxy on Google cloud compute engines?

@blkwtkns
Copy link

Before I found your project, I had made my own node app to basically do the same stuff, except only for reserveCalifornia, though it's no where near as mature as this awesome project. Thanks for this!

Okay anyways, I had to setup a proxy for running my script, because when I would run it without one, it would give 400/500 type of codes after running once or twice. AND, I wouldn't be able to look through the reserve california site for at least 3+ minutes. So I was finding myself in this box of finding reservations, but not being able to act on them fast enough.

So yeah, I configured a proxy, and it's been mostly good. Until today. something changed today, and my requests are failing hard. So then I started looking around and found your project. I really like it.

So my question in relation to this issue right here - would it be possible to allow users to integrate a proxy into this, instead of just relying on the fake useragent stuff? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage This will looked at
Projects
None yet
Development

No branches or pull requests

5 participants