Replace removeByPrefix Lua script with Scan #79
Conversation
|
|
| while (keys.Length != 0 || index < chunkSize) { | ||
| total += await RemoveAllAsync(keys).AnyContext(); | ||
| index += chunkSize; | ||
| (cursor, keys) = await ScanKeysAsync(regex, cursor, chunkSize).AnyContext(); |
There was a problem hiding this comment.
Wouldn't this fail if there were more than 2500 batches?
There was a problem hiding this comment.
I'm not sure why it would fail if there were more than 2500 batches.
I've tested with 10.000.000 entries, it worked fine.
(Except I had to split the parameters passed to SetAllAsync by batches of 1.000.000 as the current implementation does not support bigger enumerables - the ~1M limit is a Redis limit).
There was a problem hiding this comment.
I was thinking about the logic of keys.Length != 0 || index < chunkSize it didn't seem like the second part was correct.
| private async Task<(int, string[])> ScanKeysAsync(string prefix, int index, int chunkSize) | ||
| { | ||
| var result = await Database.ExecuteAsync("scan", index, "match", prefix, "count", chunkSize).AnyContext(); | ||
| var uniqueValue = result.ToDictionary().FirstOrDefault(); |
There was a problem hiding this comment.
What is the shape of the result here? If it's two elements, can we speed this up by selecting the array element?
There was a problem hiding this comment.
Good idea.
I've made the change: I'm casting directly to the expected types.
JulienDeroff
force-pushed
the
issue/78
branch
from
a0b5e1a to
5cbc6a6
Compare
|
The error message was never logged either: |
|
Thanks for the PR! |
Resolves the issue #78
Description of the change
Replace the Lua script that removes keys based on a prefix with multiple calls.
Bulk calls made to remove the detected keys by chunks of 2500.
Note: 2500 was chosen empirically after several rounds of testing. The value can be adjusted.
SCAN is being used over KEYS to avoid blocking operations.
Motivation
According to the official documentation:
https://redis.io/commands/eval/
After some testing, the issue happens when trying to remove between 7500 and 8000 entries by prefix.