Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Oauth2 client cannot refresh token #12742

Open
gbarazer opened this issue Jan 21, 2025 · 5 comments
Open

Oauth2 client cannot refresh token #12742

gbarazer opened this issue Jan 21, 2025 · 5 comments
Labels
in linear Issue or PR has been created in Linear for internal review

Comments

@gbarazer
Copy link

Hello,

We are using the generic Oauth2 credentials to run a HTTP request node and we are having failures only some times when the oauth2 token needs to be refreshed.

The credential is used in several active workflows, and it looks like there is a concurrency issue when the HTTP with an input payload of several items, because i notice in the debug logs that the token is refreshed several times in the same second and updated. The next issue is that when doing that the API server can once in a while rate limit or return an error, causing the oauthtokendata to be garbled and effectively disabling the token refresh capability.

I think the HTTP request node does not wait for the token refresh and database credential update to initialize its next run, causing a storm condition and refreshing the access_token several times in parallel, and this scenario cannot run with a refresh token strategy where a refresh token is burned immediately to get another access_token (the classic oauth flow).

Here is a log extract of what happens :

2025-01-21T08:03:37.589Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.589Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.685Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.685Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.692Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.692Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.781Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.781Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.800Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.800Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.824Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.824Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.829Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.829Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.830Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.830Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.831Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.831Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.837Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.837Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.839Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.839Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.843Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.843Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.846Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.846Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.850Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.850Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.855Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.855Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" expired. Should revalidate. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.866Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.866Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.870Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.870Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.870Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.870Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.871Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.871Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.873Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.873Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.875Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.875Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.878Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.878Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been renewed. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.886Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.886Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.889Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.889Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.890Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.890Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.894Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.894Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.896Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
2025-01-21T08:03:37.896Z | debug | OAuth2 token for "oAuth2Api" used by node "HTTP Request flex" has been saved to database successfully. {"file":"NodeExecuteFunctions.js"}
@Joffcom
Copy link
Member

Joffcom commented Jan 21, 2025

Hey @gbarazer,

We have created an internal ticket to look into this which we will be tracking as "N8N-8186"

@Joffcom Joffcom added the in linear Issue or PR has been created in Linear for internal review label Jan 21, 2025
@gbarazer
Copy link
Author

I also want to add that when the Oauth server returns an error, the credential data is completely garbled in N8N, it seems the error handling or parsing in N8N is too light here :

n8n/packages/@n8n/client-oauth2/src/ClientOAuth2.ts

Line 77 in 174cd44

* Attempt to parse response body as JSON, fall back to parsing as a query string.

For diagnosis, I looked at the credential content using the CLI n8n export:credentials, which returns this when the credential is fine and working :

[{"createdAt":"2025-01-17T16:54:00.280Z","updatedAt":"2025-01-21T08:03:37.884Z","id":"REDACTED","name":"REDACTED","data":{"grantT
ype":"authorizationCode","authUrl":"https://REDACTED/connect/authorize","accessTokenUrl":"https://REDACTED/connect/token","clientId":"REDACTED","clientSecret":"REDACTED","scope":"api offline_access","authQueryParameters":"","a
uthentication":"header","ignoreSSLIssues":false,"oauthTokenData":{"access_token":"REDACTED","expires_in":3600,"token_type":"Bearer","refresh_tok
en":"REDACTED","scope":"api offline_access","callbackQueryString":{"scope":"api offline_access"}}},"type":"oAuth2Api","isManaged":false}]

and returns this when the credential is not working anymore :

[{"createdAt":"2025-01-17T16:54:00.280Z","updatedAt":"2025-01-20T19:00:53.189Z","id":"REDACTED","name":"REDACTED","data":{"grantType":"authorizationCode","authUrl"
:"https://REDACTED/connect/authorize","accessTokenUrl":"https://REDACTED/connect/token","client
Id":"REDACTED","clientSecret":"REDACTED","scope":"api offline_access","authQueryParameters":"","authentication":"header","ignoreSSL
Issues":false,"oauthTokenData":{"access_token":"REDACTED","expires_in":3600,"token_type":"Bearer","refresh_token":"REDACTED","scope":"api offline_access","callbackQueryString":{"scope":"api offline_access"},"\r\n\r\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http:
//www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\r\nr\n"]

BIG HTML GARBAGE (probably the error page returned by the oauth server)

}},"
type":"oAuth2Api","isManaged":false}]

@Joffcom
Copy link
Member

Joffcom commented Jan 21, 2025

Hey @gbarazer,

We have an enhancement request opened internally to change how we handle the refresh and to move it to more of a background service rather than relying on a refresh when the auth fails.

Oddly though I have not seen the current approach cause an error like this before, I do know that 1.74 and maybe 1.73 has some issues with oauth so it could be worth trying 1.75 to see if that helps.

Can you share which service you are connecting to as well just so we can think about adding it to the test list if it is public.

@lapms
Copy link

lapms commented Jan 21, 2025

Also getting "Unable to sign without access token" with Google Sheets node since version 1.74.3

@Joffcom
Copy link
Member

Joffcom commented Jan 21, 2025

@lapms your issue is going to be different, Can you update to 1.75 which should work for you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
in linear Issue or PR has been created in Linear for internal review
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gbarazer @Joffcom @lapms