Skip to content

Commit

Permalink
[#213] fix Redis protocol error too big inline request
Browse files Browse the repository at this point in the history
Redis bulk load commands need to be under 64 KB per line (more or less):

redis/redis-rb#1185

https://github.com/redis/redis/blob/07b292af5e86459e435da5986f87bc4f7415d437/src/server.h#L178

If too many IDs are included in the batch file, the lines are too long,
causing an error:

    ERR Protocol error: too big inline request

Fix by chunking IDs written to lines.

What chunk size to use? 1000 IDs works, and leads to lines of
approximately 23 KB. But Register 2 generates statement IDs of around 20
characters in length, whereas BODS 0.2 and BODS 0.3 says they must be
between 32 and 64 chars (the fact that Register 2 doesn't comply with
the minimum is a separate issue). Multiplying by 20/64 gives a target of
around 20 KB, so reducing to 750 IDs should be comfortable, leading to
lines of around 17 KB (as measured with `wc -c`).
  • Loading branch information
tiredpixel committed Jan 25, 2024
1 parent 1a423ba commit 9b2367e
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion bin/combine
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ dst_f=${2:-$dst_f_}
ns=register-files-combiner
tmp=$(mktemp -d)
tmp_redis=$tmp/batch.redis
chunk_size=750
#-------------------------------------------------------------------------------
function _clean() {
rm -rf "${tmp:?}"
Expand Down Expand Up @@ -42,7 +43,10 @@ for src_f in "${src_fs[@]}" ; do
mapfile -t ids < <(
jq -r '.statementID' < "$src_f"
)
echo "SADD '$src_k' ${ids[*]@Q}" >> "$tmp_redis"
for ((i=0 ; i < ${#ids[@]}; i+=chunk_size)) ; do
chunk=( "${ids[@]:i:$chunk_size}" )
echo "SADD '$src_k' ${chunk[*]@Q}" >> "$tmp_redis"
done
state='*'
src_k_n=${#ids[@]}
else
Expand Down

0 comments on commit 9b2367e

Please sign in to comment.