Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection leak with Cosmos DB client SDK #382

Closed
euchungmsft opened this issue Jun 11, 2021 · 2 comments
Closed

Connection leak with Cosmos DB client SDK #382

euchungmsft opened this issue Jun 11, 2021 · 2 comments

Comments

@euchungmsft
Copy link

euchungmsft commented Jun 11, 2021

PoolExhaustedException occurred actually from the connection when it tries to get a cosmos db connection from the pool while available memory's decreasing. Apparently it looks like connection leakage

io.reactivex.netty.client.PoolExhaustedException:
   at io.reactivex.netty.client.ConnectionPoolImpl.performAquire (ConnectionPoolImpl.java177)
   at io.reactivex.netty.client.ConnectionPoolImpl.access$300 (ConnectionPoolImpl.java45)
   at io.reactivex.netty.client.ConnectionPoolImpl$1.call (ConnectionPoolImpl.java139)
   at io.reactivex.netty.client.ConnectionPoolImpl$1.call (ConnectionPoolImpl.java124)

Async SDK For SQL API of Azure Cosmos DB Service - 2.6.13
Spring Boot - 2.1.6
Java 11

It sounds like the situation looks similar to this
ReactiveX/RxNetty#611

@moderakh
Copy link
Contributor

moderakh commented Jun 11, 2021

on v2 sdk, PoolExhaustedException does not indicate a leak. it indicates the number of concurrent requests to GW is more than the configured SDK connection pool-size.

You are on a very old generation of the SDK (v2). our long term recommendation is to move to V4 SDK, where this limitation does not exist:
https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-java-v4

as remedy for particular PoolExhaustedException issue, please note on v2 SDK, If load from SDK to GW is high and if connectionPool size is not big enough you may run into PoolExhaustedException.

see this:
https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-java-async-sdk#connection-pool-exhausted-issue

The following should help to address the issue:

  1. increase http connection pool size (default is 1000, you can increase to 2000)

https://github.com/Azure/azure-cosmosdb-java/blob/master/commons/src/main/java/com/microsoft/azure/cosmosdb/ConnectionPolicy.java#L177

  1. You are using GW mode, by changing to DirectMode, more traffic will go over tcp and less traffic over http, please change to direct mode:

https://github.com/Azure/azure-cosmosdb-java/blob/master/commons/src/main/java/com/microsoft/azure/cosmosdb/ConnectionPolicy.java#L141

please see the recommendation on direct mode configuration:
https://docs.microsoft.com/en-us/azure/cosmos-db/performance-tips-async-java#networking

  1. as mentioned this v2 sdk is very old and generations behind. for long-run you can consider moving to v4 SDK, where a lot of improvement were made and this limitation doesn't exist:
    https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-java-v4

@euchungmsft
Copy link
Author

Thank you so much for the prompt response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants