Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 #26043

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+48 −0
Open

[FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 #26043

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/content/docs/deployment/memory/mem_tuning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,54 @@ the job can fail because the deployment environment can kill the offending conta

See also description of [container memory exceeded]({{< ref "docs/deployment/memory/mem_trouble" >}}#container-memory-exceeded) failure.

## Configure memory for Netty4

As the version of Apache Pekko was updated, now Flink RPC also uses Netty4. Netty4 introduces some changes regarding byte buffers compared to Netty3 that is worth mentioning.
Mainly Netty4 brought pooled byte buffers, which enables better performance, but it allocates slightly more memory.

### Configure byte buffer allocator type

{{< hint warning >}}
Be aware that specifying the byte buffer allocator type will affect both Flink RPC and [Flink's shuffle](({{< ref "docs/dev/datastream/execution_mode#task-scheduling-and-network-shuffle" >}})) performance!
{{< /hint >}}

In highly resource-limited use-cases it might make sense to fine-tune these configurations.
This can be done via setting the following JVM property for the TaskManager(s) and/or JobManager(s): `org.apache.flink.shaded.netty4.io.netty.allocator.type`.
The possible allocator types are the following:

* `pooled`: use `PooledByteBufAllocator.DEFAULT`
* `unpooled`: use `UnpooledByteBufAllocator.DEFAULT`
* `adaptive`: use `AdaptiveByteBufAllocator`

#### Example

```yaml
# In <flink-root-dir>/conf/config.yaml
env:
java:
opts:
jobmanager: -Dorg.apache.flink.shaded.netty4.io.netty.allocator.type=unpooled
taskmanager: -Dorg.apache.flink.shaded.netty4.io.netty.allocator.type=unpooled
```

For more information about these byte buffer allocators please check the relevant parts of the [Netty4 documentation](https://netty.io/wiki/new-and-noteworthy-in-4.0.html#buffer-api-changes).

### Enable reflection in JDK >= 11

Since by default Flink has `--add-opens=java.base/java.lang.reflect=ALL-UNNAMED`, using reflection in Netty4 should also not be a problem in most environments.
Setting `org.apache.flink.shaded.netty4.io.netty.tryReflectionSetAccessible` enables some optimizations that reduces GC pressure, and improves performance.

#### Example

```yaml
# In <flink-root-dir>/conf/config.yaml
env:
java:
opts:
jobmanager: -Dorg.apache.flink.shaded.netty4.io.netty.tryReflectionSetAccessible=true
taskmanager: -Dorg.apache.flink.shaded.netty4.io.netty.tryReflectionSetAccessible=true
```

## Configure memory for state backends

This is only relevant for TaskManagers.
Expand Down