Data Race In CPython With ThreadSanitizer

Oct 21, 2025 by SLV Team 42 views

Data Race in CPython: ThreadSanitizer Detects Issue During Finalization

Hey guys! Let's dive into a nasty bug report concerning a data race detected by ThreadSanitizer (TSan) in CPython, specifically during the finalization phase. This is a pretty interesting technical problem, so buckle up!

The Core Problem: Data Race in interpreter_clear() vs take_gil()

At the heart of the issue is a data race between the main thread and daemon threads in CPython. The main thread, during finalization, calls interpreter_clear(). This function writes to a specific memory location: _PyRuntime.ceval.eval_breaker. Meanwhile, daemon threads are trying to acquire the Global Interpreter Lock (GIL) using take_gil(). When take_gil() is called, it eventually leads to _Py_unset_eval_breaker_bit(), which performs an atomic operation on the same memory location (eval_breaker).

Imagine this: the main thread is cleaning up, while the daemon threads are still trying to grab the lock to do their thing. Because the access to eval_breaker isn't properly synchronized during this cleanup, a data race occurs. This is a classic concurrency problem, where multiple threads access the same memory location without proper protection, potentially leading to unpredictable behavior and crashes. This is a critical issue that can cause instability in Python programs, especially those that heavily rely on threading and daemon threads.

Where the Race Occurs

Here's a breakdown of the conflicting operations:

Main Thread (Write): interpreter_clear() writes to eval_breaker.
Daemon Thread (Atomic Write): take_gil() calls _Py_unset_eval_breaker_bit(), performing an atomic operation on the same memory address.

This overlapping access is what triggers the data race, as the threads are not properly coordinated when accessing this shared resource during the final stages of the program's lifecycle.

How to Reproduce the Data Race

To reproduce this data race, you'll need to build CPython with specific flags and run a test script. It's a bit involved, but it highlights the importance of careful memory management and synchronization in multithreaded environments. This test is designed to trigger the data race by creating a situation where multiple threads simultaneously try to force the Just-In-Time (JIT) compilation.

Build Configuration

First, you need to configure your CPython build with the following flags. These flags enable ThreadSanitizer and other debugging features that help expose the data race.

CC=clang CXX=clang++ ./configure --with-thread-sanitizer --with-pydebug --enable-experimental-jit=yes --disable-optimizations --with-lto=full
make -j$(nproc)

The --with-thread-sanitizer flag is crucial as it integrates ThreadSanitizer into the build process, enabling it to detect data races. The other flags add debugging information and disable optimizations, which can help to expose the issue more clearly.

Run Command

After building CPython, run the following command to execute the test script.

./python -X dev -X showrefcount bug.py

The -X dev flag enables development mode, which can provide more detailed information during execution. The -X showrefcount flag displays reference counts, which can be useful for debugging. This command runs the bug.py script.

The Test Script (bug.py)

This is where the magic happens. The bug.py script uses threading and a function that will be JIT compiled to create a race condition. The script creates multiple threads that all force the JIT compilation simultaneously. This concurrent activity increases the likelihood of the data race occurring during finalization.

import threading
from functools import lru_cache

@lru_cache(None)
def fib(n):
    """Function that JIT will compile"""
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

def hammer_jit():
    """Call the function many times to force JIT compilation"""
    for i in range(1_000_000):
        fib(i)

# Create a race: all threads simultaneously force JIT compilation
threads = [threading.Thread(target=hammer_jit) for _ in range(16)]
for t in threads: t.start()
for t in threads: t.join()

The fib() function is decorated with @lru_cache(None) to cache results and is designed to trigger the JIT compilation. The hammer_jit() function repeatedly calls fib() to force the JIT compiler to kick in. The script then creates multiple threads, each running hammer_jit(), thereby creating a scenario where many threads are contending for resources.

ThreadSanitizer Output and What It Means

When you run the script with the build and command above, ThreadSanitizer should catch the data race and produce an output similar to the one provided in the original bug report. This output is gold because it tells us exactly where the problem is occurring. This is the smoking gun, and shows the location of the data race within the CPython source code.

The crucial part of the TSan output includes:

The specific memory address where the data race occurs.
The threads involved (main thread and a daemon thread).
The functions involved (interpreter_clear() and take_gil()).

The output clearly indicates that the main thread is writing to the memory location while a daemon thread is also trying to access it, causing the data race. The stack traces provided give a precise location of the code that's causing the issue.

Analysis: Why This Happens

The root cause of this data race lies in how CPython handles finalization and the GIL. When the interpreter is being cleared during finalization, it should ensure that no other threads are actively trying to access shared resources. However, in this case, daemon threads, which might still be running in the background, attempt to acquire the GIL using take_gil(). This operation involves atomic operations on the eval_breaker field, which conflicts with the main thread's write operation during interpreter_clear(). The issue appears during the shutdown process, when daemon threads are still active.

Because there is no proper synchronization during the finalization process, this race condition is able to occur. The main thread doesn't wait for the daemon threads to finish before starting the cleanup process. The underlying problem is the lack of proper synchronization during the interpreter's shutdown phase, when these threads are still active and trying to access shared resources.

Environment Details: Where the Bug Appears

Here are the details of the environment where this bug was observed. This information is important because it helps pinpoint the context in which the problem exists and can influence how it is addressed.

CPython Version: This was found on the main branch of CPython (commit fbf0843e39), meaning it's a recent issue and likely affects the latest versions of Python.
Operating System: Tested on Linux (Ubuntu 25.10). This indicates the bug is likely to be present on other Linux distributions as well.
Compiler: Ubuntu clang version 20.1.8 (0ubuntu4). The compiler and its version are important since they can affect how the code is compiled and optimized.
Build Flags: The use of specific build flags, especially --with-thread-sanitizer, --with-pydebug, and --enable-experimental-jit=yes, is crucial for reproducing the issue. These flags enable the necessary tools and features to expose the data race. The combination of these factors creates a specific environment where the data race is more likely to appear.

Related Issues: What Else is Going On

This bug report references a couple of related issues. Understanding these helps provide context and can potentially offer clues for a fix:

#124878: This issue is similar and involves a race condition during finalization. However, it involves drop_gil and free_threadstate rather than interpreter_clear and take_gil. This suggests that the finalization process itself is a complex area prone to concurrency issues.
#104341: This issue provides background information on interpreter deletion during finalization. Knowing how the interpreter is deleted is important for understanding where and why such race conditions might occur.

These references help to understand the broader context of the problem and the ongoing efforts to address it within the CPython community.

CPython Versions Tested On

The issue has been confirmed on the CPython main branch and version 3.15, meaning this isn't an isolated problem. This increases the severity, as it potentially impacts a wide range of Python users.

Operating Systems Tested On

The bug was successfully reproduced on Linux, highlighting that it's a potential problem for many users who are running Python on Linux systems.

Conclusion: What Does It All Mean?

This data race in CPython, detected by ThreadSanitizer, highlights a critical concurrency issue during interpreter finalization. The interaction between the main thread's interpreter_clear() and daemon threads' attempts to acquire the GIL creates a vulnerability that can lead to unexpected behavior and crashes. Understanding the build configuration, test script, and TSan output is key to grasping the problem. This bug is concerning because it could cause unexpected behavior, especially in multithreaded applications. This issue underscores the complexity of multithreaded programming and the importance of thorough testing and synchronization mechanisms. Keep an eye on updates in the CPython repository, and make sure to test your multithreaded code thoroughly to avoid these types of issues!

That's it, guys! Hope you enjoyed the read, and now you have a better understanding of what to look out for when you're working with threads in Python.