Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Support clearing all consumers execept specified consumer in the consumer directory #4599

Open
2 tasks done
chenxinwei opened this issue Nov 27, 2024 · 4 comments · May be fixed by #4893
Open
2 tasks done

[Feature] Support clearing all consumers execept specified consumer in the consumer directory #4599

chenxinwei opened this issue Nov 27, 2024 · 4 comments · May be fixed by #4893
Assignees
@chenxinwei
Labels
enhancement New feature or request

Comments

@chenxinwei
Copy link
Contributor

chenxinwei commented Nov 27, 2024 •
edited
Loading

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Sometimes，we may want to modify the consumerId in a Flink task.However, if we do not delete other consumers that are not actually used at this time, the expired data will not be effectively cleaned up.So I want to support Support clearing all consumers execept specified consumer in the consumer directory.

Solution

I will introduce the ClearConsumersProcedure to support clearing unspecified consumers, clearing specified consumers, and clearing all consumers in a table.

Clear consumers procedure. Usage:

  1. clear all consumers except the specified consumer in the table
    CALL sys.clear_consumers('tableId', 'consumerIds', true)
  2. clear all specified consumers in the table
    CALL sys.clear_consumers('tableId', 'consumerIds') or CALL sys.clear_consumers('tableId', 'consumerIds', false)
  3. clear all consumers in the table
    CALL sys.clear_consumers('tableId')

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@chenxinwei chenxinwei added the enhancement New feature or request label Nov 27, 2024
@ChaomingZhangCN
Copy link
Contributor

Maybe we can use a Procedure to do this. We already have consumer.expiration-time, adding too many options reduces usability. What do you think? @chenxinwei

@chenxinwei
Copy link
Contributor Author

chenxinwei commented Nov 27, 2024 •
edited
Loading

Maybe we can use a Procedure to do this. We already have consumer.expiration-time, adding too many options reduces usability. What do you think? @chenxinwei

The existing consumer.expiration-time parameter causes the consumerId in use to join the expiration process. In some cases, we don't want the consumer to expire.

@ChaomingZhangCN
Copy link
Contributor

I think, clearing a specified consumer by flink Procedure is a better way. @chenxinwei

CALL sys.clear_consumer('tableId', 'consumerId')

@chenxinwei
Copy link
Contributor Author

I think, clearing a specified consumer by flink Procedure is a better way. @chenxinwei

CALL sys.clear_consumer('tableId', 'consumerId')

Thanks for your suggestion, this approach is indeed better.

@chenxinwei chenxinwei linked a pull request Jan 13, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chenxinwei chenxinwei

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[core] Introduce ClearConsumersProcedure to clear consumers chenxinwei/paimon
2 participants
@chenxinwei @ChaomingZhangCN