[VL] Maintanence status of the dynamic off-heap sizing feature?

[zhztheplayer]

Previous PRs about dynamic off-heap sizing:

1. [GLUTEN-5438] feat: Dynamically sizing off-heap memory #5439
2. [GLUTEN-7802][CORE] Use original total memory size if enable dynamic offheap #7803

The initial PR 1 was merged in April, and it seems we are still having the OOM risk not solving yet. @supermem613 Do you have any thoughts or updates? Thanks.

@yikf If you also have some use case you are able to share?

cc @zhli1142015 @Yohahaha

[yikf]

@zhztheplayer Thanks for your invitation.

It has been found that when dynamic memory is enabled, the memory requested by Spark from the resource manager is larger than the available memory. In short, it has restricted itself, and the total amount of requested memory has increased. #5439 (comment).