RedisSearch Crash In 8.2.1: Segfault Fix Guide

by SLV Team 47 views
RedisSearch Crash: Redis 8.2.1 Segfault in raxGenericInsert (Signal 11) - A Deep Dive and Fixes

Hey guys! Ever faced the dreaded Redis crash while running RedisSearch? It's a nasty situation, especially when it hits production. We're going to break down a specific crash scenario: a segfault in raxGenericInsert when using Redis 8.2.1 with the RedisSearch module. This guide will walk you through the issue, potential causes, and how to tackle it head-on.

Understanding the Crash

So, what exactly happened? A segmentation fault (SIGSEGV) occurred while Redis was running with the RedisSearch module (redisearch.so). The crash happened within Redis's core radix tree code (raxGenericInsert), which is a critical part of how RedisSearch manages its indexes. The stack trace clearly points to this function as the culprit, indicating a memory access issue during the insertion process within the radix tree. This could stem from several factors, such as a bug within the RedisSearch module, an incompatibility between the module and the Redis core, or even rare concurrency or memory-related problems.

Key Log Excerpts Explained

Let's dissect those log snippets to understand the crash better:

1:M 26 Oct 2025 07:45:51.549 # Redis 8.2.1 crashed by signal: 11, si_code: 1
1:M 26 Oct 2025 07:45:51.549 # Accessing address: (nil)
1:M 26 Oct 2025 07:45:51.549 # Crashed running the instruction at: 0x55b3f56a8817

This section confirms the crash (signal 11) and indicates that the program attempted to access a null address, a classic sign of a segmentation fault. The instruction address 0x55b3f56a8817 pinpoints the exact location in memory where the crash occurred.

Stack Trace Analysis

The stack trace is like a detective's notebook, revealing the chain of function calls leading to the crash:

EIP:
redis-server *:6379(raxGenericInsert+0xb7)[0x55b3f56a8817]
1 redis-server
/usr/lib/libc.so.6(epoll_wait+0x56)[0x7f348e40a7f6]
redis-server *:6379(+0x9d364)[0x55b3f5548364]
redis-server *:6379(aeMain+0xd0)[0x55b3f5548a60]
redis-server *:6379(main+0x4c2)[0x55b3f5542942]
/usr/lib/libc.so.6(+0x2724a)[0x7f348e32924a]
/usr/lib/libc.so.6(__libc_start_main+0x85)[0x7f348e329305]
redis-server *:6379(_start+0x21)[0x55b3f5544321]

The crucial line here is redis-server *:6379(raxGenericInsert+0xb7)[0x55b3f56a8817], which confirms the crash happened within the raxGenericInsert function. The rest of the stack trace shows the call chain, starting from the main function and going through the event loop (aeMain) and other internal Redis functions.

Several other threads are also listed in the stack trace, including IO threads (io_thd_), background threads for closing files (bio_close_file), lazy freeing (bio_lazy_free), and AOF operations (bio_aof), as well as a garbage collection thread (gc-0214). This suggests the crash might be related to concurrent operations within Redis, possibly involving the RedisSearch module and its interaction with the core data structures.

Decoding the INFO Output

The INFO output provides a snapshot of the Redis server's configuration and status:

# Server
redis_version:8.2.1
redis_mode:standalone
[...]
# Modules
module:name=ReJSON,ver=80200,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=vectorset,ver=1,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors|handle-repl-async-load]
module:name=search,ver=80201,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
# search_version
search_version:8.2.1
search_redis_version:8.2.1 - oss
[...]

This tells us we're dealing with Redis 8.2.1 in standalone mode, with the RedisSearch (version 8.2.1), ReJSON (version 8.2.0), and vectorset modules loaded. Notably, ReJSON is used by RedisSearch, indicating a dependency between these modules. This information is crucial for understanding the environment where the crash occurred.

Observations and Environment Details

Key observations leading up to the crash include:

  • Crash within Redis core radix tree logic during RedisSearch module code execution.
  • Repeated ForkGC - got timeout while reading from pipe (Success) messages, potentially linked to RedisSearch index garbage collection.
  • High usage of FT.SEARCH and JSON commands, with significant call counts and p99 latency.
  • IO threads enabled (io-threads 4), with RedisSearch and ReJSON modules loaded.
  • No Out-of-Memory (OOM) errors or maxmemory limit reached; RSS and fragmentation are within normal bounds.

The environment details further clarify the context:

  • Redis: 8.2.1 (Bitnami container)
  • RedisSearch: 8.2.1
  • ReJSON: 8.2.0
  • vectorset: 1
  • OS: Linux 5.14.0-427.85.1.el9_4.x86_64 x86_64
  • Memory: Approximately 3GB used out of 31GB total
  • Workload: High concurrency with heavy usage of FT.SEARCH and JSON.GET/SET commands, along with pub/sub activity.

Recreating the Crash

The steps to reproduce the crash involve:

  1. Running Redis 8.2.1 with RedisSearch 8.2.1 and ReJSON 8.2.0 modules loaded within a Bitnami container.
  2. Enabling IO threads (io-threads 4).
  3. Applying a heavy workload using FT.SEARCH and JSON commands.
  4. Observing repeated ForkGC timeouts, eventually leading to a segfault.

This reproduction scenario suggests that the crash is likely triggered by a combination of high load, concurrent operations, and specific interactions between RedisSearch and the core Redis data structures.

Potential Causes and Troubleshooting

So, what could be causing this headache? Here are a few potential culprits:

  1. Module Bug: The RedisSearch module itself might contain a bug that triggers the segfault under certain conditions. This is a primary suspect, especially given the crash occurs within the raxGenericInsert function, which is used extensively by RedisSearch for indexing.
  2. Module/Core Incompatibility: There might be an incompatibility between the RedisSearch module version (8.2.1) and the Redis core version (8.2.1). Modules interact closely with Redis internals, and even minor version mismatches can lead to unexpected behavior.
  3. Concurrency Issues: Redis, especially with IO threads enabled, handles many operations concurrently. A race condition or other concurrency bug within RedisSearch could lead to memory corruption and the observed segfault. The stack trace, with its multiple IO threads and background processes, hints at this possibility.
  4. Memory Corruption: Although the fast memory test passed, rare memory corruption issues might still occur. These can be difficult to diagnose, especially under heavy load.
  5. Garbage Collection Issues: The repeated ForkGC timeouts suggest potential problems with RedisSearch's garbage collection process. If the GC can't keep up with the rate of data changes, it might lead to memory leaks or other issues that eventually trigger a crash.

Troubleshooting Steps

Here's how we can start digging deeper:

  1. Check Known Issues: Look for known issues related to RedisSearch 8.2.1 and Redis 8.2.1. The Redis and RedisSearch communities are active, and similar crashes might have been reported already. GitHub issues, forums, and mailing lists are great resources.
  2. Simplify the Setup: Try to reproduce the crash in a simpler environment. Remove other modules, reduce the workload, and disable IO threads to see if the crash still occurs. This helps isolate the problem.
  3. Examine the Workload: Analyze the specific FT.SEARCH and JSON commands being executed. Are there any patterns or specific queries that trigger the crash? High cardinality indexes or complex queries might be contributing factors.
  4. Core Dump Analysis: If possible, generate a core dump of the crashed Redis process. This provides a snapshot of the memory at the time of the crash, allowing for detailed debugging with tools like GDB. This is probably the most direct way of pinpointing the source of the issue.
  5. Downgrade or Upgrade: As a workaround, consider downgrading to a more stable Redis and RedisSearch version combination or upgrading to the latest versions, as newer releases often include bug fixes. However, make sure to test the changes in a staging environment first!
  6. Monitor Memory Usage: Keep a close eye on Redis's memory usage, even though no OOM errors were reported. Memory leaks or excessive memory fragmentation can sometimes lead to crashes without triggering OOM.

Workarounds and Solutions

While we hunt down the root cause, let's explore some potential workarounds and solutions:

  1. Disable IO Threads: As a temporary measure, disabling IO threads (io-threads 0) might reduce concurrency and prevent the crash. However, this can impact performance, so it's not a long-term solution.
  2. Reduce Workload: If possible, reduce the load on Redis, especially the number of concurrent FT.SEARCH and JSON commands. This might involve optimizing queries, caching results, or distributing the workload across multiple Redis instances.
  3. Increase GC Frequency: If the ForkGC timeouts are indeed a contributing factor, try increasing the frequency of RedisSearch's garbage collection. However, this can also impact performance, so it's a trade-off.
  4. Implement Circuit Breakers: Use circuit breaker patterns in your application to prevent cascading failures. If Redis crashes repeatedly, the circuit breaker can temporarily stop sending requests to Redis, giving it time to recover.

Seeking Expert Advice

If you're still stuck, don't hesitate to seek help from the Redis and RedisSearch communities. Provide detailed information about your setup, the crash logs, and the steps you've taken to troubleshoot the issue. The more information you provide, the easier it will be for others to assist you.

Conclusion

Dealing with Redis crashes can be frustrating, but understanding the problem and following a systematic approach can help you find the root cause and implement effective solutions. In this case, the segfault in raxGenericInsert points to a potential bug in RedisSearch or an incompatibility issue. By analyzing the logs, understanding the environment, and trying various troubleshooting steps, we can get closer to resolving this issue and keeping our Redis instances running smoothly. Remember, patience and persistence are key! And always back up your data, just in case.

Stay tuned for updates as we continue to investigate this issue. If you've encountered a similar crash, please share your experiences in the comments below!