You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Follow up to #5148 where the cleanup of the checkpoints in the shard API was left aside.
Describe the solution you'd like
Add an extra parameter to the FileSourceSqs config object to configure how long and how many checkpoints should be kept:
During the split commit transaction, update this timestamp
Update the shard API to make the cleanup possible:
update the DeleteShardsRequest to support pruning based on the timestamp / number of messges
message DeleteShardsRequest {
quickwit.common.IndexUid index_uid = 1;
string source_id = 2;
repeated quickwit.ingest.ShardId shard_ids = 3;
// If false, only shards at EOF positions will be deleted.
bool force = 4;
+ // The maximum age of shards to keep.
+ optional uint32 max_age = 5;
+ // The maximum number of shards to keep. Older shards will be deleted first.
+ optional uint32 max_count= 6;
}
The sources call the DeleteShards API but the control plane is in charge of debouncing identical requests
Describe alternatives you've considered
For the shard API update, we could
list all the shards and filter on the client side (source)
instead change ListShardsRequest to support filtering the older shards.
Additional context
Add any other context or information about the feature request here.
Is your feature request related to a problem? Please describe.
Follow up to #5148 where the cleanup of the checkpoints in the shard API was left aside.
Describe the solution you'd like
Add an extra parameter to the
FileSourceSqs
config object to configure how long and how many checkpoints should be kept:deduplication_window_duration_sec
(default 3600)deduplication_window_max_messages
(default 100k)Handling the shard lifecycle:
update_timestamp: sqlx::types::time::PrimitiveDateTime
DeleteShardsRequest
to support pruning based on the timestamp / number of messgesDescribe alternatives you've considered
For the shard API update, we could
Additional context
Add any other context or information about the feature request here.
The text was updated successfully, but these errors were encountered: