[Feature] consumer-id in batch reading from spark

[kwontaeheon]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

When storing logs into append-only bucked table and batch reading from spark,
Is there any way to prevent data removal from compaction without turning on write-only or making snapshot-retention longer?

If there isn't, like streaming consumer-id, making a temporary snapshot prevention is needed.

Solution

setting a spark sql hint for storing consumer-id and retention

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!