Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Supports only execute specified actions during the writing process. #4884

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

LinMingQiang
Copy link
Contributor

@LinMingQiang LinMingQiang commented Jan 11, 2025

Purpose

There are several operations during data writing such as partition-expire,snapshot-expire,create-tag...
Currently, these actions are either executed together or not executed at all in the framework.

The purpose of this PR is to supports skipping specified actions during writing.

For example : set 'write.skip-actions'='partition-expire' , the partition expiration action will be skipped during data writing, but snapshot-expire / auto create tag / full compact will still execute normally.

Linked issue: close #xxx

Tests

API and Format

Documentation

@LinMingQiang LinMingQiang reopened this Jan 12, 2025
@LinMingQiang LinMingQiang force-pushed the master-feat-addconf-toskip-action branch 2 times, most recently from 9d3fcef to ed49eb8 Compare January 12, 2025 02:30
@wwj6591812
Copy link
Contributor

Thanks for @LinMingQiang prepare this pr.
This is a good job, I recently received a similar request from some Paimon user in my company.

Copy link
Member

@xuzifu666 xuzifu666 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job and left some comments.

@LinMingQiang LinMingQiang force-pushed the master-feat-addconf-toskip-action branch from 7250389 to b3867f8 Compare January 12, 2025 15:37
@xuzifu666
Copy link
Member

+1

@LinMingQiang LinMingQiang force-pushed the master-feat-addconf-toskip-action branch from b3867f8 to 99dcebd Compare January 13, 2025 08:22
@LinMingQiang LinMingQiang marked this pull request as draft January 16, 2025 13:56
@LinMingQiang LinMingQiang force-pushed the master-feat-addconf-toskip-action branch 4 times, most recently from 0c0573a to 7cb026d Compare January 18, 2025 08:17
@LinMingQiang LinMingQiang force-pushed the master-feat-addconf-toskip-action branch from 2d26f9d to fd50ccd Compare January 20, 2025 13:03
@LinMingQiang LinMingQiang marked this pull request as ready for review January 20, 2025 13:04
@LinMingQiang LinMingQiang changed the title [core] Supports skipping specified actions during writing. [core] Supports only execute specified actions in write task. Jan 20, 2025
@LinMingQiang LinMingQiang changed the title [core] Supports only execute specified actions in write task. [core] Supports only execute specified actions during the writing process. Jan 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants