Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete the oldest tracked version metadata files after commit #23940

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

7hong
Copy link

@7hong 7hong commented Oct 28, 2024

Description

When using the iceberg table, the xxx-metadata.json file is generated each time commit is executed. In the iceberg table, we can automatically clean the previous metadata file through configuration.

write.metadata.delete-after-commit.enabled = true
write.metadata.previous-versions-max = 10
However, the metadata file is not automatically cleaned in trino. A large number of metadata.json files are left on the hdfs.

Different from #20863 ,I have followed the configurations in iceberg (write.metadata.delete-after-commit.enabled and write.metadata.previous-version-max ). Instead of adding a new configuration, this keeps it compatible with iceberg.

Due to some troublesome issues encountered while merging the code, I closed #23813 and reopened a clean PR.

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x ) Release notes are required, with the following suggested text:

## Iceberg connector

* Delete the oldest tracked version metadata files after commit. ({issue} #19582 ). 

@cla-bot cla-bot bot added the cla-signed label Oct 28, 2024
@github-actions github-actions bot added the iceberg Iceberg connector label Oct 28, 2024
@7hong
Copy link
Author

7hong commented Oct 29, 2024

I have reopened a new PR, please review it, thank you! @ebyhr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed iceberg Iceberg connector
Development

Successfully merging this pull request may close these issues.

1 participant