tx/group compaction fixes #24637
Open
tx/group compaction fixes #24637
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/dt |
CI test resultstest results on build#60037
test results on build#60049
test results on build#60058
test results on build#60103
test results on build#60115
test results on build#60249
|
|
/ci-repeat 3 |
Retry command for Build#60049please wait until all jobs are finished before running the slash command |
Retry command for Build#60058please wait until all jobs are finished before running the slash command |
nvartolomei
reviewed
Dec 23, 2024
nvartolomei
reviewed
Dec 23, 2024
src/v/redpanda/admin/transaction.cc
Outdated
| } | ||
|
|
||
| if (pid_str.empty() || epoch_str.empty() || sequence_str.empty()) { | ||
| throw ss::httpd::bad_param_exception( |
There was a problem hiding this comment.
This should happen out of the box btw. This seems to be called implicitly when handling requests https://github.com/redpanda-data/seastar/blob/09a59a23ff2740a2fa591b0e65d978ca83d2b9e3/include/seastar/http/handlers.hh#L76
There was a problem hiding this comment.
ya pretty conservative check, was thinking to convert to an assert but probably not worth it.
| dedicated fence batch type (group_tx_fence) was used for group | ||
| transaction fencing. | ||
| After this buggy compaction, these uncleaned tx_fence batches are |
There was a problem hiding this comment.
Is the buggy compaction fixed now? Otherwise. shouldn't we just fix the buggy compaction instead?
There was a problem hiding this comment.
we can't fix that easily without a major refactor of the code and we don't need to (as outlined in the comment) as we already moved away from that problematic tx_fence batch infavor of group_tx_fence, so the whole fix revolves around how to get rid of the outdated tx_fence batches.
| model::producer_identity{header.producer_id, header.producer_epoch}, | ||
| header.base_offset); | ||
| model::record_batch_header, kafka::group_tx::offsets_metadata) { | ||
| // Transaction boundaries are determined by fence/commit or abort |
There was a problem hiding this comment.
This change doesn't seem to be explained by the commit message?
There was a problem hiding this comment.
Is split this into a separate commit (and added more changes to it)
ztlpn
reviewed
Dec 23, 2024
ztlpn
reviewed
Dec 23, 2024
ztlpn
reviewed
Dec 23, 2024
ztlpn
reviewed
Dec 23, 2024
ztlpn
reviewed
Dec 23, 2024
ztlpn
reviewed
Dec 23, 2024
| producers.producers.push(std::move(producer_state)); | ||
| co_await ss::coroutine::maybe_yield(); | ||
| } | ||
| co_return ss::json::json_return_type(std::move(producers)); |
There was a problem hiding this comment.
ss::json::stream_range_as_array?
There was a problem hiding this comment.
I wanted to do it originally but I didn't because the final output is not array, It is something like
{
ntp = foo,
producers = [...]
}
it there a better way to do this?
ztlpn
reviewed
Dec 23, 2024
ztlpn
reviewed
Dec 23, 2024
ztlpn
reviewed
Dec 23, 2024
tests/rptest/transactions/verifiers/consumer_offsets_verifier.py
Outdated
Show resolved
Hide resolved
ztlpn
reviewed
Jan 2, 2025
This is unsafe because it does not do any required checks to see if a particular transaction is in progress and is a candidate for abort. For example if a transaction is committed by the coordinator and pending commit on the group, using this escape hatch to abort the transaction can cause correctness issues. To be used with caution as an escape hatch for aborting transactions that the group has lost track of are ok to be aborted. This situation usually is indicative of a bug in the transaction implementation.
bharathv
force-pushed
the
fix_co
branch
3 times, most recently
from
d698d99 to
82b255b
Compare
Retry command for Build#60249please wait until all jobs are finished before running the slash command |
ztlpn
reviewed
Jan 3, 2025
| .begin_offset = offset, | ||
| .batch_ts = begin_ts, | ||
| .timeout = tx_timeout}; | ||
| if (p_state.expired_deprecated_fence_tx()) { |
There was a problem hiding this comment.
Hmm this makes me a bit uncomfortable as this is non-deterministic... But maybe it's okay because these producer states are not expected to be updated ever.
There was a problem hiding this comment.
there is a periodic GC loop added in the next commit that sweeps through this list and expires any old stuff
Consider group_metadata to determine if a group transaction should be considered open. Eg: if a group if tombstoned, any transaction corresponding to the group is ignored. This invariant is also held in the actual group stm to ensure groups are not tombstoned before any pending transactions are cleaned up
This somewhat simplifies cases where the stm needs to disambiguate between legit tx_fence batches and leftover tx_fence batches from compaction. (see next commit)
This will result in hanging transactions and subsequent blocking of compaction.
If a group got tombstoned all the producers to that group should be ignored. The current logic is incorrectly recovering producers and loading them up to expire later.
.. for a given partition, to be hooked up with REST API in the next commit.
/v1/debug/producers/{namespace}/{topic}/{partition}
.. includes low level debug information about producers for
idempotency/transactional state.
.. in this case the state machine proceeds on to applying from the log.
Bumps the supported snapshot version so the existing snapshots are invalidated as they may contain stale max_collectible_offset. This forces the stm to reconstruct the state form the log and recompute correct max_collectible_offset.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Check individual commit message for details
Main changes
/v1/transaction/unsafe_abort_group_transaction/{group_id}Backports Required
Release Notes
Bug Fixes