redis.cluster.RedisCluster with RESP3 and hiredis causes segmentation fault

[jakob-keller]

Version: redis-py 5.0.1 / Redis 7.1 (AWS ElastiCache)

Platform: CPython 3.11.7 on macOS Sonoma 14.2 (M2 Pro)

Description: redis.cluster.RedisCluster instances with protocol=3 (i.e. RESP3 enabled) crash the Python interpreter when performing commands and the hiredis extra dependency is installed in the local environment.

Example to reproduce:

url = "rediss://hostname:6379/0?protocol=3"
client = redis.cluster.RedisCluster.from_url(url)
client.get("test")

Process finished with exit code 139 (interrupted by signal 11:SIGSEGV)

[dvora-h]

@chayim This is the bug we discussed.

[asottile-sentry]

here's the bt:

Program received signal SIGSEGV, Segmentation fault.
0x0000aaaaaab9780c in PyTuple_GetItem ()
(gdb) bt
#0  0x0000aaaaaab9780c in PyTuple_GetItem ()
#1  0x0000fffff6b05920 in tryParentize (obj=0xfffff6735c40, 
    task=0xaaaaab36b5d0) at src/reader.c:81
#2  createArrayObject (task=0xaaaaab36b5d0, elements=<optimized out>)
    at src/reader.c:171
#3  0x0000fffff6b0cb18 in processAggregateItem (r=0xaaaaab334e60)
    at vendor/hiredis/read.c:540
#4  processItem (r=0xaaaaab334e60) at vendor/hiredis/read.c:647
#5  redisReaderGetReply (r=0xaaaaab334e60, reply=reply@entry=0xffffffffd0a8)
    at vendor/hiredis/read.c:763
#6  0x0000fffff6b04acc in Reader_gets (self=0xfffff671b7d0, 
    args=<optimized out>) at src/reader.c:377
#7  0x0000aaaaaab88d60 in ?? ()
#8  0x0000aaaaaab8d3f8 in _PyEval_EvalFrameDefault ()
#9  0x0000aaaaaabb4148 in ?? ()
#10 0x0000aaaaaab8e2ac in _PyEval_EvalFrameDefault ()
#11 0x0000aaaaaaba5348 in _PyFunction_Vectorcall ()
#12 0x0000aaaaaab8d3f8 in _PyEval_EvalFrameDefault ()
#13 0x0000aaaaaabb4288 in ?? ()
#14 0x0000aaaaaab8ed90 in _PyEval_EvalFrameDefault ()
#15 0x0000aaaaaabb4288 in ?? ()
#16 0x0000aaaaaab8ed90 in _PyEval_EvalFrameDefault ()
#17 0x0000aaaaaabb4288 in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--c
#18 0x0000aaaaaab8ed90 in _PyEval_EvalFrameDefault ()
#19 0x0000aaaaaaba5348 in _PyFunction_Vectorcall ()
#20 0x0000aaaaaab8d3f8 in _PyEval_EvalFrameDefault ()
#21 0x0000aaaaaaba5348 in _PyFunction_Vectorcall ()
#22 0x0000aaaaaab8d3f8 in _PyEval_EvalFrameDefault ()
#23 0x0000aaaaaab99d04 in _PyObject_FastCallDictTstate ()
#24 0x0000aaaaaabb0224 in ?? ()
#25 0x0000aaaaaab9ac18 in _PyObject_MakeTpCall ()
#26 0x0000aaaaaab91af4 in _PyEval_EvalFrameDefault ()
#27 0x0000aaaaaaba5348 in _PyFunction_Vectorcall ()
#28 0x0000aaaaaab99da8 in _PyObject_FastCallDictTstate ()
#29 0x0000aaaaaabb0224 in ?? ()
#30 0x0000aaaaaab9b108 in ?? ()
#31 0x0000aaaaaabb4e0c in PyObject_Call ()
#32 0x0000aaaaaab8ed90 in _PyEval_EvalFrameDefault ()
#33 0x0000aaaaaabb4148 in ?? ()
#34 0x0000aaaaaab9177c in _PyEval_EvalFrameDefault ()
#35 0x0000aaaaaac89760 in ?? ()
#36 0x0000aaaaaac895e4 in PyEval_EvalCode ()
#37 0x0000aaaaaacbccbc in ?? ()
#38 0x0000aaaaaacb53c8 in ?? ()
#39 0x0000aaaaaacbc96c in ?? ()
#40 0x0000aaaaaacbbad4 in _PyRun_SimpleFileObject ()
#41 0x0000aaaaaacbb6a0 in _PyRun_AnyFileObject ()
#42 0x0000aaaaaacabfc0 in Py_RunMain ()
#43 0x0000aaaaaac7a748 in Py_BytesMain ()
#44 0x0000fffff7d173fc in __libc_start_call_main (main=main@entry=0xaaaaaac7a720, argc=argc@entry=2, argv=argv@entry=0xffffffffef58) at ../sysdeps/nptl/libc_start_call_main.h:58
#45 0x0000fffff7d174cc in __libc_start_main_impl (main=0xaaaaaac7a720, argc=2, argv=0xffffffffef58, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:392
#46 0x0000aaaaaac7a630 in _start ()
(gdb) py-bt
Traceback (most recent call first):
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/_parsers/hiredis.py", line 128, in read_response
    response = self._reader.gets()
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/connection.py", line 500, in read_response
    response = self._parser.read_response(disable_decoding=disable_decoding)
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/client.py", line 553, in parse_response
    response = connection.read_response()
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/cluster.py", line 1151, in _execute_command
    response = redis_node.parse_response(connection, command, **kwargs)
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/cluster.py", line 1101, in execute_command
    res[node.name] = self._execute_command(node, *args, **kwargs)
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/commands/core.py", line 792, in command
    return self.execute_command("COMMAND", **kwargs)
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/_parsers/commands.py", line 70, in initialize
    commands = r.command()
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/_parsers/commands.py", line 67, in __init__
    self.initialize(redis_connection)
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/cluster.py", line 645, in __init__
    self.commands_parser = CommandsParser(self)
  File "/tmp/y/venv/lib/python3.10/site-packages/redis/cluster.py", line 488, in from_url
    return cls(url=url, **kwargs)
  File "/tmp/y/t.py", line 3, in <module>
    client = redis.cluster.RedisCluster.from_url(url)

[gerzse]

Hi @jakob-keller ,

This is indeed the same as #3145. From the output I can say for sure it is the hiredis-py issue with maps inside sets. If you can uninstall hiredis-py, it will solve the problem. Bug report for the root cause: redis/hiredis-py#188

I'm also curious, what is your main reason for using hiredis-py instead of the built-in RESP parser from redis-py?

[jakob-keller]

Thanks for following up!

I'm also curious, what is your main reason for using hiredis-py instead of the built-in RESP parser from redis-py?

My use case is performance critical and this repo's README states:

For faster performance, install redis with hiredis support, this provides a compiled response parser, and for most cases requires zero code changes.

[jakob-keller]

Thank you for addressing this. A quick test confirms that the issue is fixed by hiredis==3.0.0.