Expand description
Fundamental properties of objects tied to the Python interpreter.
The Python interpreter is not threadsafe. To protect the Python interpreter in multithreaded
scenarios there is a global lock, the global interpreter lock (hereafter referred to as GIL)
that must be held to safely interact with Python objects. This is why in PyO3 when you acquire
the GIL you get a Python
marker token that carries the lifetime of holding the GIL and all
borrowed references to Python objects carry this lifetime as well. This will statically ensure
that you can never use Python objects after dropping the lock - if you mess this up it will be
caught at compile time and your program will fail to compile.
It also supports this pattern that many extension modules employ:
- Drop the GIL, so that other Python threads can acquire it and make progress themselves
- Do something independently of the Python interpreter, like IO, a long running calculation or awaiting a future
- Once that is done, reacquire the GIL
That API is provided by Python::allow_threads
and enforced via the Ungil
bound on the
closure and the return type. This is done by relying on the Send
auto trait. Ungil
is
defined as the following:
pub unsafe trait Ungil {}
unsafe impl<T: Send> Ungil for T {}
We piggy-back off the Send
auto trait because it is not possible to implement custom auto
traits on stable Rust. This is the solution which enables it for as many types as possible while
making the API usable.
In practice this API works quite well, but it comes with some drawbacks:
§Drawbacks
There is no reason to prevent !Send
types like Rc
from crossing the closure. After all,
Python::allow_threads
just lets other Python threads run - it does not itself launch a new
thread.
use pyo3::prelude::*;
use std::rc::Rc;
fn main() {
Python::with_gil(|py| {
let rc = Rc::new(5);
py.allow_threads(|| {
// This would actually be fine...
println!("{:?}", *rc);
});
});
}
Because we are using Send
for something it’s not quite meant for, other code that
(correctly) upholds the invariants of Send
can cause problems.
SendWrapper
is one of those. Per its documentation:
A wrapper which allows you to move around non-Send-types between threads, as long as you access the contained value only from within the original thread and make sure that it is dropped from within the original thread.
This will “work” to smuggle Python references across the closure, because we’re not actually doing anything with threads:
use pyo3::prelude::*;
use pyo3::types::PyString;
use send_wrapper::SendWrapper;
Python::with_gil(|py| {
let string = PyString::new(py, "foo");
let wrapped = SendWrapper::new(string);
py.allow_threads(|| {
// 💥 Unsound! 💥
let smuggled: &Bound<'_, PyString> = &*wrapped;
println!("{:?}", smuggled);
});
});
For now the answer to that is “don’t do that”.
§A proper implementation using an auto trait
However on nightly Rust and when PyO3’s nightly
feature is
enabled, Ungil
is defined as the following:
#![feature(auto_traits, negative_impls)]
pub unsafe auto trait Ungil {}
// It is unimplemented for the `Python` struct and Python objects.
impl !Ungil for Python<'_> {}
impl !Ungil for ffi::PyObject {}
// `Py` wraps it in a safe api, so this is OK
unsafe impl<T> Ungil for Py<T> {}
With this feature enabled, the above two examples will start working and not working, respectively.
Structs§
- A marker token that represents holding the GIL.
Traits§
- Types that are safe to access while the GIL is not held.