`PyGILState_Ensure` in one thread causes the thread-local storage of other threads to be GCed

[kevin85421]

Bug report

Bug description:

# Download https://gist.github.com/kevin85421/4616545d3bed2ebcfe5d0a01bdfde3ff
g++ example.cc -I/usr/include/python3.12 -lpython3.12 -lpthread -o example
./example

• The above script creates two C++ threads, default_pool and custom_pool, and calls Python code.
  • Phase 1 (void init_python_thread): Each thread
    • Calls PyGILState_Ensure()
    • Runs Python code that assigns the value 1 to the variable a and creates a thread-local state using threading.local().
    • Calls PyEval_SaveThread to release GIL.
  • Phase 2 (void release_gstate): Each thread
    • Calls PyEval_RestoreThread(*tstate) to acquire GIL and restore thread state.
    • Runs Python code to print a
    • Runs Python code to print thread-local state.
    • Calls PyGILState_Release

The thread who is the first thread to call PyGILState_Ensure() will fail to print thread-local state in step 2.

In the following example log,

• default_pool calls PyGILState_Ensure() and writes a=1 and thread-local state.
• custom_pool calls PyGILState_Ensure() and writes a=1 and thread-local state. => My current guess is that PyGILState_Ensure() here makes the thread-local state of default_pool be GCed.

[C++][08:21:37.740] Hello from the default_pool in init_python_thread
[C++][08:21:37.741] Hello from the custom_pool in init_python_thread
Hello from default_pool, a = 1!
[C++][08:21:37.744] Hello from the default_pool in release_gstate
Hello from custom_pool, a = 1!
a = 1 in release_gstate!
[C++][08:21:37.744] Hello from the custom_pool in release_gstate
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: '_thread._local' object has no attribute 'name'
a = 1 in release_gstate!
thread_local.name = custom_pool in release_gstate!

CPython versions tested on:

3.12

Operating systems tested on:

Linux

[ZeroIntensity]

Hi! I'm not familiar with boost::asio::post, but your gist looks a little wrong. (Don't blame yourself, we've done a bad job of documenting how these functions work.)

This may seem a little counterintuitive, but the GIL can also be released randomly in-between execution of Python instructions, not just calls to PyEval_SaveThread. I'm seeing the following behavior based on the output:

• Thread 1 (default_pool) starts, and thread_local is assigned to a threading.local() and the name attribute is set. The GIL is then released explicitly by PyEval_SaveThread.
• Thread 2 (custom_pool) is now unblocked and takes the GIL, but completely overwrites the thread_local from before, losing the prior name attribute.
• At this point, Python decides another thread should execute (based on the switch interval), and releases the GIL without having set the attribute yet.
• release_gstate for thread 1 is now unblocked and gets the GIL, but since thread_local was blown away before, it doesn't have the name anymore, resulting in an AttributeError.
• init_python_thread for thread 2 gets the GIL again and finishes up, then hands off the GIL to release_gstate.

That aside, I would suggest avoiding passing PyGILState_STATE and PyThreadState pointers up to the main function like you're currently doing. It's not wrong, but it's error-prone; if you accidentally use either of those in a different thread than where they were created, you'll get all sorts of hard-to-debug crashes.

---

I've seen several reports before due to PyGILState_Ensure inconsistencies. It would definitely be helpful to tell us where things need to be documented better, so we don't run into problems like these. Where were you reading, and what kind of information would be useful to you?

[kevin85421]

Hi @ZeroIntensity, thank you for the quick reply!

Thread 2 (custom_pool) is now unblocked and takes the GIL, but completely overwrites the thread_local from before, losing the prior name attribute.

This differs from my understanding of how threading.local() behaves. I expected that thread_local.name in default_pool would be stored somewhere like thread_local_dict[default_pool_thread_id]["name"], and the one for custom_pool would be stored somewhere like thread_local_dict[custom_pool_thread_id]["name"]. The following example is also able to show this behavior:

# Download https://gist.github.com/kevin85421/2d4fd5620ffdd7a1dfd36fd734fa7afe
python3 example.py

Worker-0 set value: Worker-0
Worker-1 set value: Worker-1
Worker-2 set value: Worker-2
Worker-0 read value: Worker-0
Worker-1 read value: Worker-1
Worker-2 read value: Worker-2

At this point, Python decides another thread should execute (based on the switch interval), and releases the GIL without having set the attribute yet.

My observation based on the following example is that the GIL doesn't implicitly switch. Instead, the thread holds the thread should call PyEval_SaveThread or PyGILState_Release explicitly. Do I misunderstand anything?

# Download: https://gist.github.com/kevin85421/d34df2fac0af85efe25b3c3e4bf0ceea
g++ switchinterval.cc -I/usr/include/python3.12 -lpython3.12 -lpthread -o switchinterval
./switchinterval

[C++][17:18:34.651] Sleep 10 seconds
[C++][17:18:34.651] Hello from the default_pool in init_python_thread
[C++][17:18:34.651] Hello from the custom_pool in init_python_thread
Hello from default_pool, a = 1!
sys.getswitchinterval() = 0.005
[C++][17:18:44.651] Hello from the default_pool in release_gstate
Hello from custom_pool, a = 1! <---------- `custom_pool` gets the GIL after `default_pool` calls `PyGILState_Release`
sys.getswitchinterval() = 0.005
[C++][17:18:44.651] Hello from the custom_pool in release_gstate
a = 1 in release_gstate!
a = 1 in release_gstate!
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: '_thread._local' object has no attribute 'name'
thread_local.name = custom_pool in release_gstate!

My current guess is that when custom_pool calls PyGILState_Ensure, it accidentally overwrites some data associated with default_pool. Consequently, the thread-local storage detects that the default_pool thread is out of scope and removes the key default_pool_thread_id from the thread-local dictionary.

The following link points to the function that cleans up data related to a thread when it is garbage-collected.

cpython/Lib/_threading_local.py

Line 184 in 0c4248f

wrthread = ref(thread, thread_deleted)

[ZeroIntensity]

This differs from my understanding of how threading.local() behaves. I expected that thread_local.name in default_pool would be stored somewhere like thread_local_dict[default_pool_thread_id]["name"], and the one for custom_pool would be stored somewhere like thread_local_dict[custom_pool_thread_id]["name"]. The following example is also able to show this behavior:

That would normally be right, but the important part about your example is that you redeclare the threading.local() object when calling init_python_thread. The prior thread-local dictionary containing name for thread 1 is completely lost!

In fact, I'm not even really sure what happens when you set a threading.local() attribute outside of the main thread. If the switch interval isn't messing things up, I suspect it's just getting set for the second thread, and then the first thread has no name at all (which would also explain the AttributeError).

You should refactor your code to something more like:

Py_Initialize();
PyRun_SimpleString("import threading; thread_local = threading.local()");
thread1 = run_in_thread("thread_local.name = 'first'");
thread2 = run_in_thread("thread_local.name = 'second'");
// ...
Py_Finalize();

[kevin85421]

That would normally be right, but the important part about your example is that you redeclare the threading.local() object when calling init_python_thread.

Oops, my bad. I don't notice that I redeclare it. The following gist can get the threading local state in both threads.

https://gist.github.com/kevin85421/4b646b0ee07e5168b333d3325a59ca4c

[kevin85421]

Where were you reading, and what kind of information would be useful to you?

This is the doc I read: https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock. There are something possible to improve:

• Provide some runnable small examples (something like the gists) and explain the expected results of the examples.
• What will happen to the C++ threads if PyGILState_ or PyThreadState goes out of scope?
• Summarize the lifecycle of the GIL (when and how to acquire/release it).
• How do I debug? If a deadlock occurs, how can I check whether a thread is waiting for the GIL and determine which thread holds it?

#130394 (comment)

Following up on my comment: Will the GIL be released based on the switch interval in this case?

That aside, I would suggest avoiding passing PyGILState_STATE and PyThreadState pointers up to the main function like you're currently doing. It's not wrong, but it's error-prone; if you accidentally use either of those in a different thread than where they were created, you'll get all sorts of hard-to-debug crashes.

To provide more context, I am working on ray-project/ray, a distributed computing engine. Users add the @ray.remote annotation to their Python functions or classes, and then Ray launches these functions (Ray tasks) or classes (Ray actors) on the underlying C++ threads in the Ray cluster.

Currently, we don't use Python.h. Instead, we pass the Python code as a callback from Python (via Cython) to C++. However, the C++ threads are considered as Python threads only when it executes the Python callback functions.

A C++ thread:

• Executes Python function f1, which writes data to threading.local().
• After f1 finishes, the Python interpreter considers the thread dead and garbage collects its threading local state.
• Executes Python function f2, which attempts to access data from threading.local() → Fails.
  • From the Python interpreter's perspective, the thread with the same ID was dead and then became alive again which should not happen in Python IIUC.

To avoid this issue, I want the Python interpreter to treat the C++ thread as a Python thread, regardless of whether it is executing a Python function. My current solution:

1. When we initialize the C++ thread, call PyGILState_Ensure to register the thread with the Python interpreter. Then, call PyEval_SaveThread to release GIL so that other threads can get GIL. At the same time, hold both gstate and tstate.

2. Call PyEval_RestoreThread to restore thread state before executing Python function.

3. Execute Python function

4. Call PyEval_SaveThread after the Python function finishes

5. Repeat step 2 - 4

6. Call PyGILState_Release before stopping the C++ thread

[ZeroIntensity]

This is the doc I read: https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock. There are something possible to improve:

Thanks for these suggestions. I think the biggest thing that you need to grasp here is that, at least on modern Python versions, a thread state and the GIL are roughly the same thing. Having an attached thread state implies holding the GIL, and vice versa. Treat PyGILState_STATE as an opaque handle to a PyThreadState.

Following up on my comment: Will the GIL be released based on the switch interval in this case?

It could; the GIL is always subject to the switch interval. If it is implicitly switching, that would explain the issue, but if not, then there could be something else that I'm missing.

After f1 finishes, the Python interpreter considers the thread dead and garbage collects its threading local state.

I don't think Python will consider the thread dead until you call PyGILState_Release. If you want it to be "dormant" (waiting on tasks to execute, in your case), you should keep the thread alive with a GILstate available, and then just release the GIL while waiting on tasks. The psuedo-code would be:

PyGILState_STATE gstate = PyGILState_Ensure();
// We have the GIL
while (atomic_load(&thread_should_run)) {
    // Let other threads execute while we wait on tasks--Python will know about this
    // thread, it just won't have the GIL
    Py_BEGIN_ALLOW_THREADS;
    // GIL is released
    PyObject *function_to_execute = queue_get();
    // We have the function, but not the GIL
    Py_END_ALLOW_THREADS;
    // We can call it now!
    PyObject *retval = PyObject_CallNoArgs(function_to_execute);
    Py_XDECREF(retval); // Throw away the return value
    if (retval == NULL) {
        // Do whatever you want with the error--here, let's print it
        PyErr_Print();
    }
}
PyGILState_Release(gstate);

Then, call PyEval_SaveThread to release GIL so that other threads can get GIL. At the same time, hold both gstate and tstate.

Hmm, why? If you call PyGILState_Ensure, you have the GIL and can call the user's function. Why do the extra step, possibly allowing Python to shut down or do something you didn't expect?

---

I'm 95% sure this is user-error because the docs suck, so not much we can fix on our end, but if there's a PR or something that you're seeing failures on, I'd be happy to review it.

[kevin85421]

Thank you for your reply!

It could; the GIL is always subject to the switch interval. If it is implicitly switching, that would explain the issue, but if not, then there could be something else that I'm missing.

You are correct. I printed the timestamps in the Python logs, and they indeed match what you mentioned. The logs are printed to stdout about 10 seconds after the GIL is released.

I don't think Python will consider the thread dead until you call PyGILState_Release.

To provide more context, we currently don't use functions in Python.h (i.e. PyGILState_Ensure, PyEval_SaveThread ... etc) to control C++ threads in Ray. Instead, it uses Cython.

• Python main thread: Python -> Cython -> C++
  • The main thread creates C++ threads to run user-defined Python functions that are passed into C++ via Cython callback functions.
  • The main thread is responsible for submitting user-defined Cython callback functions to other C++ threads.

My observation is that the C++ threads are considered as a Python thread from the Python interpreter's perspective only when the Cython callback functions are executed in the C++ threads.

Hmm, why? If you call PyGILState_Ensure, you have the GIL and can call the user's function. Why do the extra step, possibly allowing Python to shut down or do something you didn't expect?

As I mentioned above, there are two kinds of threads in Ray. One is the main thread (in Python/Cython/C++), which is responsible for receiving user-defined Python functions and submitting them to other pure C++ threads. Hence, the pure C++ threads release the GIL, allowing the main thread to execute and wait for incoming user-defined Python functions.

[ZeroIntensity]

To provide more context, we currently don't use functions in Python.h (i.e. PyGILState_Ensure, PyEval_SaveThread ... etc) to control C++ threads in Ray. Instead, it uses Cython.

I'm a bit confused here--is your issue coming up when using Cython, or are you trying to switch to pure PyGILState_Ensure/PyEval_SaveThread, and that's where the issue is coming from?

My observation is that the C++ threads are considered as a Python thread from the Python interpreter's perspective only when the Cython callback functions are executed in the C++ threads.

That could be right, but I don't think that would be our fault. It's probably Cython's.

[kevin85421]

I'm a bit confused here--is your issue coming up when using Cython, or are you trying to switch to pure PyGILState_Ensure/PyEval_SaveThread, and that's where the issue is coming from?

The current implementation runs Cython callback functions without calling any Python.h functions.

My current attempt is:

• Run PyGILState_Ensure in the pure C++ threads
• Run Cython callback 1
• Run Cython callback 2
• ...
• Run PyGILState_Release

so that thread-local state will not be GCed after "Run Cython callback 1".

By the way, is there an easy way to check which thread holds the GIL? I asked ChatGPT whether py-spy can determine which thread holds the GIL, but the answer seems to be no.

[sergey-miryanov]

By the way, is there an easy way to check which thread holds the GIL? I asked ChatGPT whether py-spy can determine which thread holds the GIL, but the answer seems to be no.

Not the easy way, but idk other - https://www.maartenbreddels.com/perf/jupyter/python/tracing/gil/2021/01/14/Tracing-the-Python-GIL.html