PyO3
Rust bindings for Python. This includes running and interacting with Python code from a Rust binary, as well as writing native Python modules.
A comparison with rust-cpython can be found in the guide.
Usage
PyO3 supports Python 3.5 and up. The minimum required Rust version is 1.42.0-nightly 2019-01-21.
If you have never used nightly Rust, the official guide has a great section about installing it.
PyPy is also supported (via cpyext) for Python 3.5 only, targeted PyPy version is 7.0.0. Please refer to the pypy section.
You can either write a native Python module in Rust, or use Python from a Rust binary.
However, on some OSs, you need some additional packages. E.g. if you are on Ubuntu 18.04, please run
sudo apt install python3-dev python-dev
Using Rust from Python
PyO3 can be used to generate a native Python module.
Cargo.toml
[package]
name = "string-sum"
version = "0.1.0"
edition = "2018"
[lib]
name = "string_sum"
crate-type = ["cdylib"]
[dependencies.pyo3]
version = "0.9.1"
features = ["extension-module"]
src/lib.rs
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::wrap_pyfunction; #[pyfunction] /// Formats the sum of two numbers as string. fn sum_as_string(a: usize, b: usize) -> PyResult<String> { Ok((a + b).to_string()) } #[pymodule] /// A Python module implemented in Rust. fn string_sum(py: Python, m: &PyModule) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(sum_as_string))?; Ok(()) } }
On Windows and Linux, you can build normally with cargo build --release
. On macOS, you need to set additional linker arguments. One option is to compile with cargo rustc --release -- -C link-arg=-undefined -C link-arg=dynamic_lookup
, the other is to create a .cargo/config
with the following content:
[target.x86_64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
While developing, you can symlink (or copy) and rename the shared library from the target folder: On MacOS, rename libstring_sum.dylib
to string_sum.so
, on Windows libstring_sum.dll
to string_sum.pyd
, and on Linux libstring_sum.so
to string_sum.so
. Then open a Python shell in the same folder and you'll be able to import string_sum
.
To build, test and publish your crate as a Python module, you can use maturin or setuptools-rust. You can find an example for setuptools-rust in examples/word-count, while maturin should work on your crate without any configuration.
Using Python from Rust
If you want your Rust application to create a Python interpreter internally and
use it to run Python code, add pyo3
to your Cargo.toml
like this:
[dependencies]
pyo3 = "0.9.1"
Example program displaying the value of sys.version
and the current user name:
use pyo3::prelude::*; use pyo3::types::IntoPyDict; fn main() -> Result<(), ()> { let gil = Python::acquire_gil(); let py = gil.python(); main_(py).map_err(|e| { // We can't display Python exceptions via std::fmt::Display, // so print the error here manually. e.print_and_set_sys_last_vars(py); }) } fn main_(py: Python) -> PyResult<()> { let sys = py.import("sys")?; let version: String = sys.get("version")?.extract()?; let locals = [("os", py.import("os")?)].into_py_dict(py); let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'"; let user: String = py.eval(code, None, Some(&locals))?.extract()?; println!("Hello {}, I'm Python {}", user, version); Ok(()) }
Our guide has a section with lots of examples about this topic.
Examples and tooling
- examples/word-count Counting the occurrences of a word in a text file
- hyperjson A hyper-fast Python module for reading/writing JSON data using Rust's serde-json
- rust-numpy Rust binding of NumPy C-API
- html-py-ever Using html5ever through kuchiki to speed up html parsing and css-selecting.
- pyo3-built Simple macro to expose metadata obtained with the
built
crate as aPyDict
- point-process High level API for pointprocesses as a Python library
- autopy A simple, cross-platform GUI automation library for Python and Rust.
- Contains an example of building wheels on TravisCI and appveyor using cibuildwheel
- orjson Fast Python JSON library
- inline-python Inline Python code directly in your Rust code
- Rogue-Gym Customizable rogue-like game for AI experiments
- Contains an example of building wheels on Azure Pipelines
- fastuuid Python bindings to Rust's UUID library
- python-ext-wasm Python library to run WebAssembly binaries
- dict-derive Derive FromPyObject to automatically transform Python dicts into Rust structs
Python Modules
As shown in the Getting Started chapter, you can create a module as follows:
use pyo3::prelude::*; // add bindings to the generated Python module // N.B: "rust2py" must be the name of the `.so` or `.pyd` file. /// This module is implemented in Rust. #[pymodule] fn rust2py(py: Python, m: &PyModule) -> PyResult<()> { // PyO3 aware function. All of our Python interfaces could be declared in a separate module. // Note that the `#[pyfn()]` annotation automatically converts the arguments from // Python objects to Rust values, and the Rust return value back into a Python object. // The `_py` argument represents that we're holding the GIL. #[pyfn(m, "sum_as_string")] fn sum_as_string_py(_py: Python, a: i64, b: i64) -> PyResult<String> { let out = sum_as_string(a, b); Ok(out) } Ok(()) } // logic implemented as a normal Rust function fn sum_as_string(a: i64, b: i64) -> String { format!("{}", a + b) } fn main() {}
The #[pymodule]
procedural macro attribute takes care of exporting the initialization function of your module to Python. It can take as an argument the name of your module, which must be the name of the .so
or .pyd
file; the default is the Rust function's name.
To import the module, either copy the shared library as described in Get Started or use a tool, e.g. maturin develop
with maturin or python setup.py develop
with setuptools-rust.
Documentation
The Rust doc comments of the module initialization function will be applied automatically as the Python docstring of your module.
import rust2py
print(rust2py.__doc__)
Which means that the above Python code will print This module is implemented in Rust.
.
Modules as objects
In Python, modules are first class objects. This means that you can store them as values or add them to dicts or other modules:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::{wrap_pyfunction, wrap_pymodule}; use pyo3::types::IntoPyDict; #[pyfunction] fn subfunction() -> String { "Subfunction".to_string() } #[pymodule] fn submodule(_py: Python, module: &PyModule) -> PyResult<()> { module.add_wrapped(wrap_pyfunction!(subfunction))?; Ok(()) } #[pymodule] fn supermodule(_py: Python, module: &PyModule) -> PyResult<()> { module.add_wrapped(wrap_pymodule!(submodule))?; Ok(()) } fn nested_call() { let gil = GILGuard::acquire(); let py = gil.python(); let supermodule = wrap_pymodule!(supermodule)(py); let ctx = [("supermodule", supermodule)].into_py_dict(py); py.run("assert supermodule.submodule.subfunction() == 'Subfunction'", None, Some(&ctx)).unwrap(); } }
This way, you can create a module hierarchy within a single extension module.
Python Functions
PyO3 supports two ways to define a free function in Python. Both require registering the function to a module.
One way is defining the function in the module definition, annotated with #[pyfn]
.
use pyo3::prelude::*; #[pymodule] fn rust2py(py: Python, m: &PyModule) -> PyResult<()> { #[pyfn(m, "sum_as_string")] fn sum_as_string_py(_py: Python, a:i64, b:i64) -> PyResult<String> { Ok(format!("{}", a + b)) } Ok(()) } fn main() {}
The other is annotating a function with #[pyfunction]
and then adding it
to the module using the wrap_pyfunction!
macro.
use pyo3::prelude::*; use pyo3::wrap_pyfunction; #[pyfunction] fn double(x: usize) -> usize { x * 2 } #[pymodule] fn module_with_functions(py: Python, m: &PyModule) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(double)).unwrap(); Ok(()) } fn main() {}
Argument parsing
Both the #[pyfunction]
and #[pyfn]
attributes support specifying details of
argument parsing. The details are given in the section "Method arguments" in
the Classes chapter. Here is an example for a function that accepts
arbitrary keyword arguments (**kwargs
in Python syntax) and returns the number
that was passed:
extern crate pyo3; use pyo3::prelude::*; use pyo3::wrap_pyfunction; use pyo3::types::PyDict; #[pyfunction(kwds="**")] fn num_kwds(kwds: Option<&PyDict>) -> usize { kwds.map_or(0, |dict| dict.len()) } #[pymodule] fn module_with_functions(py: Python, m: &PyModule) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(num_kwds)).unwrap(); Ok(()) } fn main() {}
Making the function signature available to Python
In order to make the function signature available to Python to be retrieved via
inspect.signature
, use the #[text_signature]
annotation as in the example
below. The /
signifies the end of positional-only arguments. (This
is not a feature of this library in particular, but the general format used by
CPython for annotating signatures of built-in functions.)
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; /// This function adds two unsigned 64-bit integers. #[pyfunction] #[text_signature = "(a, b, /)"] fn add(a: u64, b: u64) -> u64 { a + b } }
This also works for classes and methods:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::types::PyType; // it works even if the item is not documented: #[pyclass] #[text_signature = "(c, d, /)"] struct MyClass {} #[pymethods] impl MyClass { // the signature for the constructor is attached // to the struct definition instead. #[new] fn new(c: i32, d: &str) -> Self { Self {} } // the self argument should be written $self #[text_signature = "($self, e, f)"] fn my_method(&self, e: i32, f: i32) -> i32 { e + f } #[classmethod] #[text_signature = "(cls, e, f)"] fn my_class_method(cls: &PyType, e: i32, f: i32) -> i32 { e + f } #[staticmethod] #[text_signature = "(e, f)"] fn my_static_method(e: i32, f: i32) -> i32 { e + f } } }
Making the function signature available to Python (old method)
Alternatively, simply make sure the first line of your docstring is
formatted like in the following example. Please note that the newline after the
--
is mandatory. The /
signifies the end of positional-only arguments.
#[text_signature]
should be preferred, since it will override automatically
generated signatures when those are added in a future version of PyO3.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; /// add(a, b, /) /// -- /// /// This function adds two unsigned 64-bit integers. #[pyfunction] fn add(a: u64, b: u64) -> u64 { a + b } // a function with a signature but without docs. Both blank lines after the `--` are mandatory. /// sub(a, b, /) /// -- /// /// #[pyfunction] fn sub(a: u64, b: u64) -> u64 { a - b } }
When annotated like this, signatures are also correctly displayed in IPython.
>>> pyo3_test.add?
Signature: pyo3_test.add(a, b, /)
Docstring: This function adds two unsigned 64-bit integers.
Type: builtin_function_or_method
Closures
Currently, there are no conversions between Fn
s in Rust and callables in Python. This would definitely be possible and very useful, so contributions are welcome. In the meantime, you can do the following:
Calling Python functions in Rust
You can use ObjectProtocol::is_callable
to check if you have a callable object. is_callable
will return true
for functions (including lambdas), methods and objects with a __call__
method. You can call the object with ObjectProtocol::call
with the args as first parameter and the kwargs (or None
) as second parameter. There are also ObjectProtocol::call0
with no args and ObjectProtocol::call1
with only positional args.
Calling Rust functions in Python
If you have a static function, you can expose it with #[pyfunction]
and use wrap_pyfunction!
to get the corresponding PyObject
. For dynamic functions, e.g. lambdas and functions that were passed as arguments, you must put them in some kind of owned container, e.g. a Box
. (A long-term solution will be a special container similar to wasm-bindgen's Closure
). You can then use a #[pyclass]
struct with that container as a field as a way to pass the function over the FFI barrier. You can even make that class callable with __call__
so it looks like a function in Python code.
Python Classes
Defining a new class
To define a custom Python class, a Rust struct needs to be annotated with the
#[pyclass]
attribute.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, debug: bool, } }
The above example generates implementations for PyTypeInfo
, PyTypeObject
,
and PyClass
for MyClass
.
Specifically, the following implementation is generated:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; /// Class for demonstration struct MyClass { num: i32, debug: bool, } impl pyo3::pyclass::PyClassAlloc for MyClass {} unsafe impl pyo3::PyTypeInfo for MyClass { type Type = MyClass; type BaseType = PyAny; type BaseLayout = pyo3::pycell::PyCellBase<PyAny>; type Layout = PyCell<Self>; type Initializer = PyClassInitializer<Self>; type AsRefTarget = PyCell<Self>; const NAME: &'static str = "MyClass"; const MODULE: Option<&'static str> = None; const DESCRIPTION: &'static str = "Class for demonstration"; const FLAGS: usize = 0; #[inline] fn type_object() -> &'static pyo3::ffi::PyTypeObject { use pyo3::type_object::LazyStaticType; static TYPE_OBJECT: LazyStaticType = LazyStaticType::new(); TYPE_OBJECT.get_or_init::<Self>() } } impl pyo3::pyclass::PyClass for MyClass { type Dict = pyo3::pyclass_slots::PyClassDummySlot; type WeakRef = pyo3::pyclass_slots::PyClassDummySlot; type BaseNativeType = PyAny; } impl pyo3::IntoPy<PyObject> for MyClass { fn into_py(self, py: pyo3::Python) -> pyo3::PyObject { pyo3::IntoPy::into_py(pyo3::Py::new(py, self).unwrap(), py) } } pub struct MyClassGeneratedPyo3Inventory { methods: &'static [pyo3::class::PyMethodDefType], } impl pyo3::class::methods::PyMethodsInventory for MyClassGeneratedPyo3Inventory { fn new(methods: &'static [pyo3::class::PyMethodDefType]) -> Self { Self { methods } } fn get_methods(&self) -> &'static [pyo3::class::PyMethodDefType] { self.methods } } impl pyo3::class::methods::PyMethodsInventoryDispatch for MyClass { type InventoryType = MyClassGeneratedPyo3Inventory; } pyo3::inventory::collect!(MyClassGeneratedPyo3Inventory); let gil = Python::acquire_gil(); let py = gil.python(); let cls = py.get_type::<MyClass>(); pyo3::py_run!(py, cls, "assert cls.__name__ == 'MyClass'") }
Adding the class to a module
Custom Python classes can then be added to a module using add_class()
.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, debug: bool, } #[pymodule] fn mymodule(_py: Python, m: &PyModule) -> PyResult<()> { m.add_class::<MyClass>()?; Ok(()) } }
PyCell and interior mutability
You sometimes need to convert your pyclass
into a Python object and access it
from Rust code (e.g., for testing it).
PyCell
is the primary interface for that.
PyCell<T: PyClass>
is always allocated in the Python heap, so Rust doesn't have ownership of it.
In other words, Rust code can only extract a &PyCell<T>
, not a PyCell<T>
.
Thus, to mutate data behind &PyCell
safely, PyO3 employs the
Interior Mutability Pattern
like RefCell
.
Users who are familiar with RefCell
can use PyCell
just like RefCell
.
For users who are not very familiar with RefCell
, here is a reminder of Rust's rules of borrowing:
- At any given time, you can have either (but not both of) one mutable reference or any number of immutable references.
- References must always be valid.
PyCell
, like RefCell
, ensures these borrowing rules by tracking references at runtime.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::types::PyDict; #[pyclass] struct MyClass { #[pyo3(get)] num: i32, debug: bool, } let gil = Python::acquire_gil(); let py = gil.python(); let obj = PyCell::new(py, MyClass { num: 3, debug: true }).unwrap(); { let obj_ref = obj.borrow(); // Get PyRef assert_eq!(obj_ref.num, 3); // You cannot get PyRefMut unless all PyRefs are dropped assert!(obj.try_borrow_mut().is_err()); } { let mut obj_mut = obj.borrow_mut(); // Get PyRefMut obj_mut.num = 5; // You cannot get any other refs until the PyRefMut is dropped assert!(obj.try_borrow().is_err()); assert!(obj.try_borrow_mut().is_err()); } // You can convert `&PyCell` to a Python object pyo3::py_run!(py, obj, "assert obj.num == 5") }
&PyCell<T>
is bounded by the same lifetime as a GILGuard
.
To make the object longer lived (for example, to store it in a struct on the
Rust side), you can use Py<T>
, which stores an object longer than the GIL
lifetime, and therefore needs a Python<'_>
token to access.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, } fn return_myclass() -> Py<MyClass> { let gil = Python::acquire_gil(); let py = gil.python(); Py::new(py, MyClass { num: 1 }).unwrap() } let gil = Python::acquire_gil(); let obj = return_myclass(); let cell = obj.as_ref(gil.python()); // AsPyRef::as_ref returns &PyCell let obj_ref = cell.borrow(); // Get PyRef<T> assert_eq!(obj_ref.num, 1); }
Customizing the class
The #[pyclass]
macro accepts the following parameters:
name=XXX
- Set the class name shown in Python code. By default, the struct name is used as the class name.freelist=XXX
- Thefreelist
parameter adds support of free allocation list to custom class. The performance improvement applies to types that are often created and deleted in a row, so that they can benefit from a freelist.XXX
is a number of items for the free list.gc
- Classes with thegc
parameter participate in Python garbage collection. If a custom class contains references to other Python objects that can be collected, thePyGCProtocol
trait has to be implemented.weakref
- Adds support for Python weak references.extends=BaseType
- Use a custom base class. The baseBaseType
must implementPyTypeInfo
.subclass
- Allows Python classes to inherit from this class.dict
- Adds__dict__
support, so that the instances of this type have a dictionary containing arbitrary instance variables.module="XXX"
- Set the name of the module the class will be shown as defined in. If not given, the class will be a virtual member of thebuiltins
module.
Constructor
By default it is not possible to create an instance of a custom class from Python code.
To declare a constructor, you need to define a method and annotate it with the #[new]
attribute. Only Python's __new__
method can be specified, __init__
is not available.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, } #[pymethods] impl MyClass { #[new] fn new(num: i32) -> Self { MyClass { num } } } }
If no method marked with #[new]
is declared, object instances can only be
created from Rust, but not from Python.
For arguments, see the Method arguments
section below.
Return type
Generally, #[new]
method have to return T: Into<PyClassInitializer<Self>>
or
PyResult<T> where T: Into<PyClassInitializer<Self>>
.
For constructors that may fail, you should wrap the return type in a PyResult as well. Consult the table below to determine which type your constructor should return:
Cannot fail | May fail | |
---|---|---|
No inheritance | T | PyResult<T> |
Inheritance(T Inherits U) | (T, U) | PyResult<(T, U)> |
Inheritance(General Case) | PyClassInitializer<T> | PyResult<PyClassInitializer<T>> |
Inheritance
By default, PyAny
is used as the base class. To override this default,
use the extends
parameter for pyclass
with the full path to the base class.
For convenience, (T, U)
implements Into<PyClassInitializer<T>>
where U
is the
baseclass of T
.
But for more deeply nested inheritance, you have to return PyClassInitializer<T>
explicitly.
To get a parent class from a child, use PyRef
instead of &self
for methods,
or PyRefMut
instead of &mut self
.
Then you can access a parent class by self_.as_ref()
as &Self::BaseClass
,
or by self_.into_super()
as PyRef<Self::BaseClass>
.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct BaseClass { val1: usize, } #[pymethods] impl BaseClass { #[new] fn new() -> Self { BaseClass { val1: 10 } } pub fn method(&self) -> PyResult<usize> { Ok(self.val1) } } #[pyclass(extends=BaseClass)] struct SubClass { val2: usize, } #[pymethods] impl SubClass { #[new] fn new() -> (Self, BaseClass) { (SubClass { val2: 15 }, BaseClass::new()) } fn method2(self_: PyRef<Self>) -> PyResult<usize> { let super_ = self_.as_ref(); // Get &BaseClass super_.method().map(|x| x * self_.val2) } } #[pyclass(extends=SubClass)] struct SubSubClass { val3: usize, } #[pymethods] impl SubSubClass { #[new] fn new() -> PyClassInitializer<Self> { PyClassInitializer::from(SubClass::new()) .add_subclass(SubSubClass{val3: 20}) } fn method3(self_: PyRef<Self>) -> PyResult<usize> { let v = self_.val3; let super_ = self_.into_super(); // Get PyRef<SubClass> SubClass::method2(super_).map(|x| x * v) } } let gil = Python::acquire_gil(); let py = gil.python(); let subsub = pyo3::PyCell::new(py, SubSubClass::new()).unwrap(); pyo3::py_run!(py, subsub, "assert subsub.method3() == 3000") }
You can also inherit native types such as PyDict
, if they implement
PySizedLayout
.
However, because of some technical problems, we don't currently provide safe upcasting methods for types that inherit native types. Even in such cases, you can unsafely get a base class by raw pointer conversion.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::types::PyDict; use pyo3::{AsPyPointer, PyNativeType}; use std::collections::HashMap; #[pyclass(extends=PyDict)] #[derive(Default)] struct DictWithCounter { counter: HashMap<String, usize>, } #[pymethods] impl DictWithCounter { #[new] fn new() -> Self { Self::default() } fn set(mut self_: PyRefMut<Self>, key: String, value: &PyAny) -> PyResult<()> { self_.counter.entry(key.clone()).or_insert(0); let py = self_.py(); let dict: &PyDict = unsafe { py.from_borrowed_ptr_or_err(self_.as_ptr())? }; dict.set_item(key, value) } } let gil = Python::acquire_gil(); let py = gil.python(); let cnt = pyo3::PyCell::new(py, DictWithCounter::new()).unwrap(); pyo3::py_run!(py, cnt, "cnt.set('abc', 10); assert cnt['abc'] == 10") }
If SubClass
does not provide a baseclass initialization, the compilation fails.
# use pyo3::prelude::*;
#[pyclass]
struct BaseClass {
val1: usize,
}
#[pyclass(extends=BaseClass)]
struct SubClass {
val2: usize,
}
#[pymethods]
impl SubClass {
#[new]
fn new() -> Self {
SubClass { val2: 15 }
}
}
Object properties
Property descriptor methods can be defined in a #[pymethods]
impl
block only and have to be
annotated with #[getter]
and #[setter]
attributes. For example:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, } #[pymethods] impl MyClass { #[getter] fn num(&self) -> PyResult<i32> { Ok(self.num) } } }
A getter or setter's function name is used as the property name by default. There are several ways how to override the name.
If a function name starts with get_
or set_
for getter or setter respectively,
the descriptor name becomes the function name with this prefix removed. This is also useful in case of
Rust keywords like type
(raw identifiers
can be used since Rust 2018).
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, } #[pymethods] impl MyClass { #[getter] fn get_num(&self) -> PyResult<i32> { Ok(self.num) } #[setter] fn set_num(&mut self, value: i32) -> PyResult<()> { self.num = value; Ok(()) } } }
In this case, a property num
is defined and available from Python code as self.num
.
Both the #[getter]
and #[setter]
attributes accept one parameter.
If this parameter is specified, it is used as the property name, i.e.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, } #[pymethods] impl MyClass { #[getter(number)] fn num(&self) -> PyResult<i32> { Ok(self.num) } #[setter(number)] fn set_num(&mut self, value: i32) -> PyResult<()> { self.num = value; Ok(()) } } }
In this case, the property number
is defined and available from Python code as self.number
.
For simple cases where a member variable is just read and written with no side effects, you can also declare getters and setters in your Rust struct field definition, for example:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { #[pyo3(get, set)] num: i32 } }
Then it is available from Python code as self.num
.
Instance methods
To define a Python compatible method, an impl
block for your struct has to be annotated with the
#[pymethods]
attribute. PyO3 generates Python compatible wrappers for all functions in this
block with some variations, like descriptors, class method static methods, etc.
Since Rust allows any number of impl
blocks, you can easily split methods
between those accessible to Python (and Rust) and those accessible only to Rust.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, } #[pymethods] impl MyClass { fn method1(&self) -> PyResult<i32> { Ok(10) } fn set_method(&mut self, value: i32) -> PyResult<()> { self.num = value; Ok(()) } } }
Calls to these methods are protected by the GIL, so both &self
and &mut self
can be used.
The return type must be PyResult<T>
or T
for some T
that implements IntoPy<PyObject>
;
the latter is allowed if the method cannot raise Python exceptions.
A Python
parameter can be specified as part of method signature, in this case the py
argument
gets injected by the method wrapper, e.g.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, debug: bool, } #[pymethods] impl MyClass { fn method2(&self, py: Python) -> PyResult<i32> { Ok(10) } } }
From the Python perspective, the method2
in this example does not accept any arguments.
Class methods
To create a class method for a custom class, the method needs to be annotated
with the #[classmethod]
attribute.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::types::PyType; #[pyclass] struct MyClass { num: i32, debug: bool, } #[pymethods] impl MyClass { #[classmethod] fn cls_method(cls: &PyType) -> PyResult<i32> { Ok(10) } } }
Declares a class method callable from Python.
- The first parameter is the type object of the class on which the method is called. This may be the type object of a derived class.
- The first parameter implicitly has type
&PyType
. - For details on
parameter-list
, see the documentation ofMethod arguments
section. - The return type must be
PyResult<T>
orT
for someT
that implementsIntoPy<PyObject>
.
Static methods
To create a static method for a custom class, the method needs to be annotated with the
#[staticmethod]
attribute. The return type must be T
or PyResult<T>
for some T
that implements
IntoPy<PyObject>
.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: i32, debug: bool, } #[pymethods] impl MyClass { #[staticmethod] fn static_method(param1: i32, param2: &str) -> PyResult<i32> { Ok(10) } } }
Callable objects
To specify a custom __call__
method for a custom class, the method needs to be annotated with
the #[call]
attribute. Arguments of the method are specified as for instance methods.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::types::PyTuple; #[pyclass] struct MyClass { num: i32, debug: bool, } #[pymethods] impl MyClass { #[call] #[args(args="*")] fn __call__(&self, args: &PyTuple) -> PyResult<i32> { println!("MyClass has been called"); Ok(self.num) } } }
Method arguments
By default, PyO3 uses function signatures to determine which arguments are required. Then it scans
the incoming args
and kwargs
parameters. If it can not find all required
parameters, it raises a TypeError
exception. It is possible to override the default behavior
with the #[args(...)]
attribute. This attribute accepts a comma separated list of parameters in
the form of attr_name="default value"
. Each parameter has to match the method parameter by name.
Each parameter can be one of the following types:
"*"
: var arguments separator, each parameter defined after"*"
is a keyword-only parameter. Corresponds to python'sdef meth(*, arg1.., arg2=..)
.args="*"
: "args" is var args, corresponds to Python'sdef meth(*args)
. Type of theargs
parameter has to be&PyTuple
.kwargs="**"
: "kwargs" receives keyword arguments, corresponds to Python'sdef meth(**kwargs)
. The type of thekwargs
parameter has to beOption<&PyDict>
.arg="Value"
: arguments with default value. Corresponds to Python'sdef meth(arg=Value)
. If thearg
argument is defined after var arguments, it is treated as a keyword-only argument. Note thatValue
has to be valid rust code, PyO3 just inserts it into the generated code unmodified.
Example:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::types::{PyDict, PyTuple}; #[pyclass] struct MyClass { num: i32, debug: bool, } #[pymethods] impl MyClass { #[new] #[args(num = "-1", debug = "true")] fn new(num: i32, debug: bool) -> Self { MyClass { num, debug } } #[args( num = "10", debug = "true", py_args = "*", name = "\"Hello\"", py_kwargs = "**" )] fn method( &mut self, num: i32, debug: bool, name: &str, py_args: &PyTuple, py_kwargs: Option<&PyDict>, ) -> PyResult<String> { self.debug = debug; self.num = num; Ok(format!( "py_args={:?}, py_kwargs={:?}, name={}, num={}, debug={}", py_args, py_kwargs, name, self.num, self.debug )) } fn make_change(&mut self, num: i32, debug: bool) -> PyResult<String> { self.num = num; self.debug = debug; Ok(format!("num={}, debug={}", self.num, self.debug)) } } }
N.B. the position of the "*"
argument (if included) controls the system of handling positional and keyword arguments. In Python:
import mymodule
mc = mymodule.MyClass()
print(mc.method(44, False, "World", 666, x=44, y=55))
print(mc.method(num=-1, name="World"))
print(mc.make_change(44, False))
print(mc.make_change(debug=False, num=-1))
Produces output:
py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44, debug=false
py_args=(), py_kwargs=None, name=World, num=-1, debug=true
num=44, debug=false
num=-1, debug=false
Class customizations
Python's object model defines several protocols for different object behavior, like sequence,
mapping or number protocols. PyO3 defines separate traits for each of them. To provide specific
Python object behavior, you need to implement the specific trait for your struct. Important note,
each protocol implementation block has to be annotated with the #[pyproto]
attribute.
Basic object customization
The PyObjectProtocol
trait provides several basic customizations.
Attribute access
To customize object attribute access, define the following methods:
fn __getattr__(&self, name: FromPyObject) -> PyResult<impl IntoPy<PyObject>>
fn __setattr__(&mut self, name: FromPyObject, value: FromPyObject) -> PyResult<()>
fn __delattr__(&mut self, name: FromPyObject) -> PyResult<()>
Each method corresponds to Python's self.attr
, self.attr = value
and del self.attr
code.
String Conversions
-
fn __repr__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>
-
fn __str__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>
Possible return types for
__str__
and__repr__
arePyResult<String>
orPyResult<PyString>
. -
fn __bytes__(&self) -> PyResult<PyBytes>
Provides the conversion to
bytes
. -
fn __format__(&self, format_spec: &str) -> PyResult<impl ToPyObject<ObjectType=PyString>>
Special method that is used by the
format()
builtin and thestr.format()
method. Possible return types arePyResult<String>
orPyResult<PyString>
.
Comparison operators
-
fn __richcmp__(&self, other: impl FromPyObject, op: CompareOp) -> PyResult<impl ToPyObject>
Overloads Python comparison operations (
==
,!=
,<
,<=
,>
, and>=
). Theop
argument indicates the comparison operation being performed. The return type will normally bePyResult<bool>
, but any Python object can be returned. Ifother
is not of the type specified in the signature, the generated code will automaticallyreturn NotImplemented
. -
fn __hash__(&self) -> PyResult<impl PrimInt>
Objects that compare equal must have the same hash value. The return type must be
PyResult<T>
whereT
is one of Rust's primitive integer types.
Other methods
-
fn __bool__(&self) -> PyResult<bool>
Determines the "truthyness" of the object.
Garbage Collector Integration
If your type owns references to other Python objects, you will need to
integrate with Python's garbage collector so that the GC is aware of
those references.
To do this, implement the PyGCProtocol
trait for your struct.
It includes two methods __traverse__
and __clear__
.
These correspond to the slots tp_traverse
and tp_clear
in the Python C API.
__traverse__
must call visit.call()
for each reference to another Python object.
__clear__
must clear out any mutable references to other Python objects
(thus breaking reference cycles). Immutable references do not have to be cleared,
as every cycle must contain at least one mutable reference.
Example:
#![allow(unused_variables)] fn main() { extern crate pyo3; use pyo3::prelude::*; use pyo3::PyTraverseError; use pyo3::gc::{PyGCProtocol, PyVisit}; #[pyclass] struct ClassWithGCSupport { obj: Option<PyObject>, } #[pyproto] impl PyGCProtocol for ClassWithGCSupport { fn __traverse__(&self, visit: PyVisit) -> Result<(), PyTraverseError> { if let Some(ref obj) = self.obj { visit.call(obj)? } Ok(()) } fn __clear__(&mut self) { if let Some(obj) = self.obj.take() { // Release reference, this decrements ref counter. let gil = GILGuard::acquire(); let py = gil.python(); py.release(obj); } } } }
Special protocol trait implementations have to be annotated with the #[pyproto]
attribute.
It is also possible to enable GC for custom classes using the gc
parameter of the pyclass
attribute.
i.e. #[pyclass(gc)]
. In that case instances of custom class participate in Python garbage
collection, and it is possible to track them with gc
module methods. When using the gc
parameter,
it is required to implement the PyGCProtocol
trait, failure to do so will result in an error
at compile time:
#[pyclass(gc)]
struct GCTracked {} // Fails because it does not implement PyGCProtocol
Iterator Types
Iterators can be defined using the
PyIterProtocol
trait.
It includes two methods __iter__
and __next__
:
fn __iter__(slf: PyRefMut<Self>) -> PyResult<impl IntoPy<PyObject>>
fn __next__(slf: PyRefMut<Self>) -> PyResult<Option<impl IntoPy<PyObject>>>
Returning Ok(None)
from __next__
indicates that that there are no further items.
Example:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::PyIterProtocol; #[pyclass] struct MyIterator { iter: Box<Iterator<Item = PyObject> + Send>, } #[pyproto] impl PyIterProtocol for MyIterator { fn __iter__(mut slf: PyRefMut<Self>) -> PyResult<Py<MyIterator>> { Ok(slf.into()) } fn __next__(mut slf: PyRefMut<Self>) -> PyResult<Option<PyObject>> { Ok(slf.iter.next()) } } }
How methods are implemented
Users should be able to define a #[pyclass]
with or without #[pymethods]
, while PyO3 needs a
trait with a function that returns all methods. Since it's impossible to make the code generation in
pyclass dependent on whether there is an impl block, we'd need to implement the trait on
#[pyclass]
and override the implementation in #[pymethods]
, which is to the best of my knowledge
only possible with the specialization feature, which can't be used on stable.
To escape this we use inventory, which allows us to collect impl
s from arbitrary source code by exploiting some binary trick. See inventory: how it works and pyo3_derive_backend::py_class::impl_inventory
for more details.
Type Conversions
PyO3 provides some handy traits to convert between Python types and Rust types.
.extract()
and the FromPyObject
trait
The easiest way to convert a Python object to a Rust value is using
.extract()
. It returns a PyResult
with a type error if the conversion
fails, so usually you will use something like
let v: Vec<i32> = obj.extract()?;
This method is available for many Python object types, and can produce a wide
variety of Rust types, which you can check out in the implementor list of
FromPyObject
.
FromPyObject
is also implemented for your own Rust types wrapped as Python
objects (see the chapter about classes). There, in order to both be
able to operate on mutable references and satisfy Rust's rules of non-aliasing
mutable references, you have to extract the PyO3 reference wrappers PyRef
and PyRefMut
. They work like the reference wrappers of
std::cell::RefCell
and ensure (at runtime) that Rust borrows are allowed.
The ToPyObject
trait
ToPyObject
is a conversion trait that allows various objects to be
converted into PyObject
. IntoPy<PyObject>
serves the
same purpose, except that it consumes self
.
*args
and **kwargs
for Python object calls
There are several ways how to pass positional and keyword arguments to a Python object call.
The ObjectProtocol
trait provides two methods:
call
- call any callable Python object.call_method
- call a specific method on the object, shorthand forget_attr
thencall
.
Both methods need args
and kwargs
arguments, but there are variants for less
complex calls, such as call1
for only args
and call0
for no arguments at all.
use pyo3::prelude::*; use pyo3::types::{PyDict, PyTuple}; struct SomeObject; impl SomeObject { fn new(py: Python) -> PyObject { PyDict::new(py).to_object(py) } } fn main() { let arg1 = "arg1"; let arg2 = "arg2"; let arg3 = "arg3"; let gil = Python::acquire_gil(); let py = gil.python(); let obj = SomeObject::new(py); // call object without empty arguments obj.call0(py); // call object with PyTuple let args = PyTuple::new(py, &[arg1, arg2, arg3]); obj.call1(py, args); // pass arguments as rust tuple let args = (arg1, arg2, arg3); obj.call1(py, args); }
kwargs
can be None
or Some(&PyDict)
. You can use the
IntoPyDict
trait to convert other dict-like containers,
e.g. HashMap
or BTreeMap
, as well as tuples with up to 10 elements and
Vec
s where each element is a two-element tuple.
use pyo3::prelude::*; use pyo3::types::{IntoPyDict, PyDict}; use std::collections::HashMap; struct SomeObject; impl SomeObject { fn new(py: Python) -> PyObject { PyDict::new(py).to_object(py) } } fn main() { let key1 = "key1"; let val1 = 1; let key2 = "key2"; let val2 = 2; let gil = Python::acquire_gil(); let py = gil.python(); let obj = SomeObject::new(py); // call object with PyDict let kwargs = [(key1, val1)].into_py_dict(py); obj.call(py, (), Some(kwargs)); // pass arguments as Vec let kwargs = vec![(key1, val1), (key2, val2)]; obj.call(py, (), Some(kwargs.into_py_dict(py))); // pass arguments as HashMap let mut kwargs = HashMap::<&str, i32>::new(); kwargs.insert(key1, 1); obj.call(py, (), Some(kwargs.into_py_dict(py))); }
FromPy<T>
and IntoPy<T>
Many conversions in PyO3 can't use std::convert::From
because they need a GIL token.
The FromPy
trait offers an from_py
method that works just like from
, except for taking a Python<'_>
argument.
I.e. FromPy<T>
could be converting a Rust object into a Python object even though it is called FromPy
- it doesn't say anything about which side of the conversion is a Python object.
Just like From<T>
, if you implement FromPy<T>
you gain a blanket implementation of IntoPy
for free.
Eventually, traits such as ToPyObject
will be replaced by this trait and a FromPy
trait will be added that will implement
IntoPy
, just like with From
and Into
.
Python Exceptions
Defining a new exception
You can use the create_exception!
macro to define a new exception type:
#![allow(unused_variables)] fn main() { use pyo3::create_exception; create_exception!(module, MyError, pyo3::exceptions::Exception); }
module
is the name of the containing module.MyError
is the name of the new exception type.
For example:
use pyo3::prelude::*; use pyo3::create_exception; use pyo3::types::IntoPyDict; use pyo3::exceptions::Exception; create_exception!(mymodule, CustomError, Exception); fn main() { let gil = Python::acquire_gil(); let py = gil.python(); let ctx = [("CustomError", py.get_type::<CustomError>())].into_py_dict(py); py.run("assert str(CustomError) == \"<class 'mymodule.CustomError'>\"", None, Some(&ctx)).unwrap(); py.run("assert CustomError('oops').args == ('oops',)", None, Some(&ctx)).unwrap(); }
Raising an exception
To raise an exception, first you need to obtain an exception type and construct a new PyErr
, then call the PyErr::restore
method to write the exception back to the Python interpreter's global state.
use pyo3::{Python, PyErr}; use pyo3::exceptions; fn main() { let gil = Python::acquire_gil(); let py = gil.python(); PyErr::new::<exceptions::TypeError, _>("Error").restore(py); assert!(PyErr::occurred(py)); drop(PyErr::fetch(py)); }
From pyfunction
s and pyclass
methods, returning an Err(PyErr)
is enough;
PyO3 will handle restoring the exception on the Python interpreter side.
If you already have a Python exception instance, you can simply call PyErr::from_instance
.
PyErr::from_instance(py, err).restore(py);
If a Rust type exists for the exception, then it is possible to use the py_err
method.
For example, each standard exception defined in the pyo3::exceptions
module
has a corresponding Rust type, exceptions defined by create_exception!
and import_exception!
macro
have Rust types as well.
#![allow(unused_variables)] fn main() { use pyo3::exceptions; use pyo3::prelude::*; fn check_for_error() -> bool {false} fn my_func(arg: PyObject) -> PyResult<()> { if check_for_error() { Err(exceptions::ValueError::py_err("argument is wrong")) } else { Ok(()) } } }
Checking exception types
Python has an isinstance
method to check an object's type,
in PyO3 there is a Python::is_instance
method which does the same thing.
use pyo3::Python; use pyo3::types::{PyBool, PyList}; fn main() { let gil = Python::acquire_gil(); let py = gil.python(); assert!(py.is_instance::<PyBool, _>(PyBool::new(py, true)).unwrap()); let list = PyList::new(py, &[1, 2, 3, 4]); assert!(!py.is_instance::<PyBool, _>(list.as_ref()).unwrap()); assert!(py.is_instance::<PyList, _>(list.as_ref()).unwrap()); }
Python::is_instance
calls the underlying PyType::is_instance
method to do the actual work.
To check the type of an exception, you can simply do:
use pyo3::exceptions; use pyo3::prelude::*; fn main() { let gil = Python::acquire_gil(); let py = gil.python(); let err = exceptions::TypeError::py_err(()); err.is_instance::<exceptions::TypeError>(py); }
Handling Rust errors
The vast majority of operations in this library will return PyResult<T>
,
which is an alias for the type Result<T, PyErr>
.
A PyErr
represents a Python exception.
Errors within the PyO3 library are also exposed as Python exceptions.
The PyO3 library handles Python exceptions in two stages. During the first stage, a PyErr
instance is
created. At this stage, holding Python's GIL is not required. During the second stage, an actual Python
exception instance is created and set active in the Python interpreter.
In simple cases, for custom errors adding an implementation of std::convert::From<T>
trait
for this custom error is enough. PyErr::new
accepts an argument in the form
of ToPyObject + 'static
. If the 'static
constraint can not be satisfied or
more complex arguments are required, the
PyErrArguments
trait can be implemented. In that case, actual exception argument creation is delayed
until a Python
object is available.
#![allow(unused_variables)] fn main() { use pyo3::{exceptions, PyErr, PyResult}; use std::error::Error; use std::fmt; #[derive(Debug)] struct CustomIOError; impl Error for CustomIOError {} impl fmt::Display for CustomIOError { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "Oh no!") } } fn bind(_addr: &str) -> Result<(), CustomIOError> { Err(CustomIOError) } impl std::convert::From<CustomIOError> for PyErr { fn from(err: CustomIOError) -> PyErr { exceptions::OSError::py_err(err.to_string()) } } fn connect(s: String) -> PyResult<bool> { bind("127.0.0.1:80")?; Ok(true) } }
The code snippet above will raise an OSError
in Python if bind()
returns a CustomIOError
.
The std::convert::From<T>
trait is implemented for most of the Rust standard library's error
types so the ?
operator can be used.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; fn parse_int(s: String) -> PyResult<usize> { Ok(s.parse::<usize>()?) } }
The code snippet above will raise a ValueError
in Python if String::parse()
returns an error.
Using exceptions defined in Python code
It is possible to use an exception defined in Python code as a native Rust type.
The import_exception!
macro allows importing a specific exception class and defines a zero-sized Rust type
for that exception.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::import_exception; import_exception!(io, UnsupportedOperation); fn tell(file: PyObject) -> PyResult<u64> { use pyo3::exceptions::*; let gil = Python::acquire_gil(); let py = gil.python(); match file.call_method0(py, "tell") { Err(_) => Err(UnsupportedOperation::py_err("not supported: tell")), Ok(x) => x.extract::<u64>(py), } } }
pyo3::exceptions
defines exceptions for several standard library modules.
Calling Python in Rust code
These APIs work from Rust whenever you have a Python
object handy, whether
PyO3 is built for an extension module or not.
Want to run just an expression? Then use eval
.
Python::eval
is
a method to execute a Python expression
and return the evaluated value as a &PyAny
object.
use pyo3::prelude::*; use pyo3::types::IntoPyDict; fn main() -> Result<(), ()> { let gil = Python::acquire_gil(); let py = gil.python(); let result = py.eval("[i * 10 for i in range(5)]", None, None).map_err(|e| { e.print_and_set_sys_last_vars(py); })?; let res: Vec<i64> = result.extract().unwrap(); assert_eq!(res, vec![0, 10, 20, 30, 40]); Ok(()) }
Want to run statements? Then use run
.
Python::run
is a method to execute one or more
Python statements.
This method returns nothing (like any Python statement), but you can get
access to manipulated objects via the locals
dict.
You can also use the py_run!
macro, which is a shorthand for Python::run
.
Since py_run!
panics on exceptions, we recommend you use this macro only for
quickly testing your Python extensions.
use pyo3::prelude::*; use pyo3::{PyCell, PyObjectProtocol, py_run}; fn main() { #[pyclass] struct UserData { id: u32, name: String, } #[pymethods] impl UserData { fn as_tuple(&self) -> (u32, String) { (self.id, self.name.clone()) } } #[pyproto] impl PyObjectProtocol for UserData { fn __repr__(&self) -> PyResult<String> { Ok(format!("User {}(id: {})", self.name, self.id)) } } let gil = Python::acquire_gil(); let py = gil.python(); let userdata = UserData { id: 34, name: "Yu".to_string(), }; let userdata = PyCell::new(py, userdata).unwrap(); let userdata_as_tuple = (34, "Yu"); py_run!(py, userdata userdata_as_tuple, r#" assert repr(userdata) == "User Yu(id: 34)" assert userdata.as_tuple() == userdata_as_tuple "#); }
You have a Python file or Python function? Then use PyModule
.
PyModule also can execute Python code by calling its methods.
use pyo3::{prelude::*, types::{IntoPyDict, PyModule}}; fn main() -> PyResult<()> { let gil = Python::acquire_gil(); let py = gil.python(); let activators = PyModule::from_code(py, r#" def relu(x): """see https://en.wikipedia.org/wiki/Rectifier_(neural_networks)""" return max(0.0, x) def leaky_relu(x, slope=0.01): return x if x >= 0 else x * slope "#, "activators.py", "activators")?; let relu_result: f64 = activators.call1("relu", (-1.0,))?.extract()?; assert_eq!(relu_result, 0.0); let kwargs = [("slope", 0.2)].into_py_dict(py); let lrelu_result: f64 = activators .call("leaky_relu", (-1.0,), Some(kwargs))? .extract()?; assert_eq!(lrelu_result, -0.2); Ok(()) }
GIL lifetimes, mutability and Python object types
On first glance, PyO3 provides a huge number of different types that can be used to wrap or refer to Python objects. This page delves into the details and gives an overview of their intended meaning, with examples when each type is best used.
Mutability and Rust types
Since Python has no concept of ownership, and works solely with boxed objects, any Python object can be referenced any number of times, and mutation is allowed from any reference.
The situation is helped a little by the Global Interpreter Lock (GIL), which ensures that only one thread can use the Python interpreter and its API at the same time, while non-Python operations (system calls and extension code) can unlock the GIL. (See the section on parallelism for how to do that in PyO3.)
In PyO3, holding the GIL is modeled by acquiring a token of the type
Python<'py>
, which serves three purposes:
- It provides some global API for the Python interpreter, such as
eval
. - It can be passed to functions that require a proof of holding the GIL,
such as
PyObject::clone_ref
. - Its lifetime can be used to create Rust references that implicitly guarantee
holding the GIL, such as
&'py PyAny
.
The latter two points are the reason why some APIs in PyO3 require the py: Python
argument, while others don't.
The PyO3 API for Python objects is written such that instead of requiring a
mutable Rust reference for mutating operations such as
PyList::append
, a shared reference (which, in turn, can only
be created through Python<'_>
with a GIL lifetime) is sufficient.
However, Rust structs wrapped as Python objects (called pyclass
types) usually
do need &mut
access. Due to the GIL, PyO3 can guarantee thread-safe acces
to them, but it cannot statically guarantee uniqueness of &mut
references once
an object's ownership has been passed to the Python interpreter, ensuring
references is done at runtime using PyCell
, a scheme very similar to
std::cell::RefCell
.
Object types
PyObject
Represents: a GIL independent reference to a Python object of unspecified type.
Used: Whenever you want to carry around references to "some" Python object, without caring about a GIL lifetime. For example, storing Python object references in a Rust struct that outlives the Python-Rust FFI boundary, or returning objects from functions implemented in Rust back to Python.
Can be cloned using Python reference counts with .clone_ref()
.
Conversions:
- To
&PyAny
:obj.as_ref(py)
- To
Py<ConcreteType>
:obj.as_ref(py).extract::<Py<ConcreteType>>
- To
&ConcreteType
(which must be a Python native type):obj.cast_as(py)
Py<SomeType>
Represents: a GIL independent reference to a Python object of known type.
This can be a Python native type (like PyTuple
), or a pyclass
type
implemented in Rust.
Used: Like PyObject
, but with a known inner type.
Conversions:
- To
PyObject
:obj.to_object(py)
- To
&SomeType
or&PyCell<SomeType>
:obj.as_ref(py)
. Forpyclass
types implemented in Rust, you get aPyCell
(see below). For Python native types, mutating operations through PyO3's API don't require&mut
access.
Note: PyObject
is semantically equivalent to Py<PyAny>
and might be
merged with it in the future.
PyAny
Represents: a Python object of unspecified type, restricted to a GIL
lifetime. Currently, PyAny
can only ever occur as a reference, usually
&PyAny
.
Used: Whenever you want to refer to some Python object only as long as
holding the GIL. For example, intermediate values and arguments to
pyfunction
s or pymethod
s implemented in Rust where any type is allowed.
Conversions:
- To
PyObject
:obj.to_object(py)
PyTuple
, PyDict
, and many more
Represents: a native Python object of known type, restricted to a GIL
lifetime just like PyAny
.
Used: Whenever you want to operate with native Python types while holding
the GIL. Like PyAny
, this is the most convenient form to use for function
arguments and intermediate values.
Conversions:
- To
PyAny
:obj.as_ref()
- To
Py<T>
:Py::from(obj)
PyCell<SomeType>
Represents: a reference to a Rust object (instance of PyClass
) which is
wrapped in a Python object. The cell part is an analog to stdlib's
RefCell
to allow access to &mut
references.
Used: for accessing pure-Rust API of the instance (members and functions
taking &SomeType
or &mut SomeType
) while maintaining the aliasing rules of
Rust references.
Conversions:
- From
PyAny
:.downcast()
PyRef<SomeType>
and PyRefMut<SomeType>
Represents: reference wrapper types employed by PyCell
to keep track of
borrows, analog to Ref
and RefMut
used by RefCell
.
Used: while borrowing a PyCell
. They can also be used with .extract()
on types like Py<T>
and PyAny
to get a reference quickly.
Related traits and types
PyClass
This trait marks structs defined in Rust that are also usable as Python classes,
usually defined using the #[pyclass]
macro.
PyNativeType
This trait marks structs that mirror native Python types, such as PyList
.
Parallelism
CPython has the infamous GIL (Global Interpreter Lock), which prevents developers from getting true parallelism when running pure Python code. While PyO3 needs to hold the GIL by default when called from Python, in order to allow manipulation of Python objects, you can release the GIL when executing Rust-only code to achieve true parallelism.
The Python::allow_threads
method temporarily releases the GIL, thus allowing other Python threads to run.
impl Python {
pub fn allow_threads<T, F>(self, f: F) -> T where F: Send + FnOnce() -> T {}
}
Let's take a look at our word-count example,
where we have a wc_parallel
function that utilizes the rayon crate to count words in parallel.
fn wc_parallel(lines: &str, search: &str) -> i32 {
lines.par_lines()
.map(|line| wc_line(line, search))
.sum()
}
Then in the Python bridge, we have a function search
exposed to the Python runtime which calls
wc_parallel
inside a closure passed to Python::allow_threads
to enable true parallelism:
#[pymodule]
fn word_count(py: Python, m: &PyModule) -> PyResult<()> {
#[pyfn(m, "search")]
fn search(py: Python, path: String, search: String) -> PyResult<i32> {
let mut file = File::open(path)?;
let mut contents = String::new();
file.read_to_string(&mut contents)?;
let count = py.allow_threads(move || wc_parallel(&contents, &search));
Ok(count)
}
Ok(())
}
Benchmark
Let's benchmark the word-count
example to verify that we did unlock true parallelism with PyO3.
We are using pytest-benchmark
to benchmark three word count functions:
The benchmark script can be found here,
then we can run pytest tests
to benchmark them.
On MacBook Pro (Retina, 15-inch, Mid 2015) the benchmark gives:
Debugging
Macros
PyO3's attributes (#[pyclass]
, #[pymodule]
, etc.) are procedural macros, which means that they rewrite the source of the annotated item. You can view the generated source with the following command, which also expands a few other things:
cargo rustc --profile=check -- -Z unstable-options --pretty=expanded > expanded.rs; rustfmt expanded.rs
(You might need to install rustfmt if you don't already have it.)
You can also debug classic !
-macros by adding -Z trace-macros
:
cargo rustc --profile=check -- -Z unstable-options --pretty=expanded -Z trace-macros > expanded.rs; rustfmt expanded.rs
See cargo expand for a more elaborate version of those commands.
Running with Valgrind
Valgrind is a tool to detect memory management bugs such as memory leaks.
You first need to install a debug build of Python, otherwise Valgrind won't produce usable results. In Ubuntu there's e.g. a python3-dbg
package.
Activate an environment with the debug interpreter and recompile. If you're on Linux, use ldd
with the name of your binary and check that you're linking e.g. libpython3.6dm.so.1.0
instead of libpython3.6m.so.1.0
.
Download the suppressions file for cpython.
Run Valgrind with valgrind --suppressions=valgrind-python.supp ./my-command --with-options
Getting a stacktrace
The best start to investigate a crash such as an segmentation fault is a backtrace.
- Link against a debug build of python as described in the previous chapter
- Run
gdb <my-binary>
- Enter
r
to run - After the crash occurred, enter
bt
orbt full
to print the stacktrace
Advanced topics
FFI
PyO3 exposes much of Python's C API through the ffi
module.
The C API is naturally unsafe and requires you to manage reference counts, errors and specific invariants yourself. Please refer to the C API Reference Manual and The Rustonomicon before using any function from that API.
Testing
Currently, #341 causes cargo test
to fail with weird linking errors when the extension-module
feature is activated. For now you can work around this by making the extension-module
feature optional and running the tests with cargo test --no-default-features
:
[dependencies.pyo3]
version = "0.8.1"
[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]
Building and Distribution
Python version
PyO3 uses a build script to determine the Python version and set the correct linker arguments. By default it uses the python3
executable. You can override the Python interpreter by setting PYTHON_SYS_EXECUTABLE
, e.g., PYTHON_SYS_EXECUTABLE=python3.6
.
Linking
Different linker arguments must be set for libraries/extension modules and binaries, which includes both standalone binaries and tests. (More specifically, binaries must be told where to find libpython and libraries must not link to libpython for manylinux compliance).
Since PyO3's build script can't know whether you're building a binary or a library, you have to activate the extension-module
feature to get the build options for a library, or it'll default to binary.
If you have e.g. a library crate and a profiling crate alongside, you need to use optional features. E.g. you put the following in the library crate:
[dependencies]
pyo3 = "0.6"
[lib]
name = "hyperjson"
crate-type = ["rlib", "cdylib"]
[features]
default = ["pyo3/extension-module"]
And this in the profiling crate:
[dependencies]
my_main_crate = { path = "..", default-features = false }
pyo3 = "0.6"
On Linux/macOS you might have to change LD_LIBRARY_PATH
to include libpython, while on windows you might need to set LIB
to include pythonxy.lib
(where x and y are major and minor version), which is normally either in the libs
or Lib
folder of a Python installation.
Distribution
There are two ways to distribute your module as a Python package: the old, setuptools-rust, and the new, maturin. setuptools-rust needs some configuration files (setup.py
, MANIFEST.in
, build-wheels.sh
, etc.) and external tools (docker, twine). maturin doesn't need any configuration files. It can not yet build sdist though (pyo3/maturin#2).
Cross Compiling
Cross compiling PyO3 modules is relatively straightforward and requires a few pieces of software:
- A toolchain for your target.
- The appropriate options in your Cargo
.config
for the platform you're targeting and the toolchain you are using. - A Python interpreter that's already been compiled for your target.
- The headers that match the above interpreter.
See https://github.com/japaric/rust-cross for a primer on cross compiling Rust in general.
After you've obtained the above, you can build a cross compiled PyO3 module by setting a few extra environment variables:
PYO3_CROSS_INCLUDE_DIR
: This variable must be set to the directory containing the headers for the target's Python interpreter.PYO3_CROSS_LIB_DIR
: This variable must be set to the directory containing the target's libpython DSO.
An example might look like the following (assuming your target's sysroot is at /home/pyo3/cross/sysroot
and that your target is armv7
):
export PYO3_CROSS_INCLUDE_DIR="/home/pyo3/cross/sysroot/usr/include"
export PYO3_CROSS_LIB_DIR="/home/pyo3/cross/sysroot/usr/lib"
cargo build --target armv7-unknown-linux-gnueabihf
PyPy Support
Using PyPy is supported via cpyext.
Support is only provided for building Rust extension for code running under PyPy. This means that PyPy cannot be called from rust via cpyext. Note that there some differences in the ffi module between PyPy and CPython.
This is a limitation of cpyext and support for embedding cpyext is not planned.
Compilation against PyPy is done by exporting the PYTHON_SYS_EXECUTABLE
to point to a PyPy binary or by compiling in a PyPy virtualenv.
For example, PYTHON_SYS_EXECUTABLE="/path/to/pypy3" /path/to/pypy3 setup.py install
Unsupported features
These are features currently supported by PyO3, but not yet implemented in cpyext.
- Complex number functions (
_Py_c_sum
,_Py_c_sum
..) - Conversion to rust's i128, u128 types.
PySequence_Count
(which is used to count number of element in array)PyDict_MergeFromSeq2
(used inPyDict::from_sequence
)
Appendix A: PyO3 and rust-cpython
PyO3 began as fork of rust-cpython when rust-cpython wasn't maintained. Over the time PyO3 has become fundamentally different from rust-cpython.
This chapter is based on the discussion in PyO3/pyo3#55.
Macros
While rust-cpython has a macro based dsl for declaring modules and classes, PyO3 uses proc macros and specialization. PyO3 also doesn't change your struct and functions so you can still use them as normal Rust functions. The disadvantage is that specialization currently only works on nightly.
rust-cpython
py_class!(class MyClass |py| {
data number: i32;
def __new__(_cls, arg: i32) -> PyResult<MyClass> {
MyClass::create_instance(py, arg)
}
def half(&self) -> PyResult<i32> {
Ok(self.number(py) / 2)
}
});
pyo3
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass { num: u32, } #[pymethods] impl MyClass { #[new] fn new(num: u32) -> Self { MyClass { num } } fn half(&self) -> PyResult<u32> { Ok(self.num / 2) } } }
Ownership and lifetimes
All objects are owned by the PyO3 library and all APIs available with references, while in rust-cpython, you own python objects.
Here is an example of the PyList API:
rust-cpython
impl PyList {
fn new(py: Python) -> PyList {...}
fn get_item(&self, py: Python, index: isize) -> PyObject {...}
}
pyo3
impl PyList {
fn new(py: Python) -> &PyList {...}
fn get_item(&self, index: isize) -> &PyAny {...}
}
Because PyO3 allows only references to Python objects, all references have the GIL lifetime. So the owned Python object is not required, and it is safe to have functions like fn py<'p>(&'p self) -> Python<'p> {}
.
Error handling
rust-cpython requires a Python
parameter for constructing a PyErr
, so error handling ergonomics is pretty bad. It is not possible to use ?
with Rust errors.
PyO3 on other hand does not require Python
for constructing a PyErr
, it is only required if you want to raise an exception in Python with the PyErr::restore()
method. Due to various std::convert::From<E> for PyErr
implementations for Rust standard error types E
, propagating ?
is supported automatically.
Appendix B: Migrating from older PyO3 versions
This guide can help you upgrade code through breaking changes from one PyO3 version to the next. For a detailed list of all changes, see CHANGELOG.md
from 0.8.* to 0.9
#[new]
interface
PyRawObject
is now removed and our syntax for constructors has changed.
Before:
#![allow(unused_variables)] fn main() { #[pyclass] struct MyClass {} #[pymethods] impl MyClass { #[new] fn new(obj: &PyRawObject) { obj.init(MyClass { }) } } }
After:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass {} #[pymethods] impl MyClass { #[new] fn new() -> Self { MyClass {} } } }
Basically you can return Self
or Result<Self>
directly.
For more, see the constructor section of this guide.
PyCell
PyO3 0.9 introduces PyCell
, which is a RefCell
-like object wrapper
for ensuring Rust's rules regarding aliasing of references are upheld.
For more detail, see the
Rust Book's section on Rust's rules of references
For #[pymethods]
or #[pyfunction]
s, your existing code should continue to work without any change.
Python exceptions will automatically be raised when your functions are used in a way which breaks Rust's
rules of references.
Here is an example.
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct Names { names: Vec<String> } #[pymethods] impl Names { #[new] fn new() -> Self { Names { names: vec![] } } fn merge(&mut self, other: &mut Names) { self.names.append(&mut other.names) } } let gil = Python::acquire_gil(); let py = gil.python(); let names = PyCell::new(py, Names::new()).unwrap(); let borrow_mut_err = py.get_type::<pyo3::pycell::PyBorrowMutError>(); pyo3::py_run!(py, names borrow_mut_err, r" try: names.merge(names) assert False, 'Unreachable' except RuntimeError as e: isinstance(e, borrow_mut_err) "); }
Names
has a merge
method, which takes &mut self
and another argument of type &mut Self
.
Given this #[pyclass]
, calling names.merge(names)
in Python raises
a PyBorrowMutError
exception, since it requires two mutable borrows of names
.
However, for #[pyproto]
and some functions, you need to manually fix the code.
Object creation
In 0.8 object creation was done with PyRef::new
and PyRefMut::new
.
In 0.9 these have both been removed.
To upgrade code, please use
PyCell::new
instead.
If you need PyRef
or PyRefMut
, just call .borrow()
or .borrow_mut()
on the newly-created PyCell
.
Before:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass {} let gil = Python::acquire_gil(); let py = gil.python(); let obj_ref = PyRef::new(py, MyClass {}).unwrap(); }
After:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; #[pyclass] struct MyClass {} let gil = Python::acquire_gil(); let py = gil.python(); let obj = PyCell::new(py, MyClass {}).unwrap(); let obj_ref = obj.borrow(); }
Object extraction
For PyClass
types T
, &T
and &mut T
no longer have FromPyObject
implementations.
Instead you should extract PyRef<T>
or PyRefMut<T>
, respectively.
If T
implements Clone
, you can extract T
itself.
In addition, you can also extract &PyCell<T>
, though you rarely need it.
Before:
let obj: &PyAny = create_obj();
let obj_ref: &MyClass = obj.extract().unwrap();
let obj_ref_mut: &mut MyClass = obj.extract().unwrap();
After:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::types::IntoPyDict; #[pyclass] #[derive(Clone)] struct MyClass {} #[pymethods] impl MyClass { #[new]fn new() -> Self { MyClass {} }} let gil = Python::acquire_gil(); let py = gil.python(); let typeobj = py.get_type::<MyClass>(); let d = [("c", typeobj)].into_py_dict(py); let create_obj = || py.eval("c()", None, Some(d)).unwrap(); let obj: &PyAny = create_obj(); let obj_cell: &PyCell<MyClass> = obj.extract().unwrap(); let obj_cloned: MyClass = obj.extract().unwrap(); // extracted by cloning the object { let obj_ref: PyRef<MyClass> = obj.extract().unwrap(); // we need to drop obj_ref before we can extract a PyRefMut due to Rust's rules of references } let obj_ref_mut: PyRefMut<MyClass> = obj.extract().unwrap(); }
#[pyproto]
Most of the arguments to methods in #[pyproto]
impls require a
FromPyObject
implementation.
So if your protocol methods take &T
or &mut T
(where T: PyClass
),
please use PyRef
or PyRefMut
instead.
Before:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::class::PySequenceProtocol; #[pyclass] struct ByteSequence { elements: Vec<u8>, } #[pyproto] impl PySequenceProtocol for ByteSequence { fn __concat__(&self, other: &Self) -> PyResult<Self> { let mut elements = self.elements.clone(); elements.extend_from_slice(&other.elements); Ok(Self { elements }) } } }
After:
#![allow(unused_variables)] fn main() { use pyo3::prelude::*; use pyo3::class::PySequenceProtocol; #[pyclass] struct ByteSequence { elements: Vec<u8>, } #[pyproto] impl PySequenceProtocol for ByteSequence { fn __concat__(&self, other: PyRef<'p, Self>) -> PyResult<Self> { let mut elements = self.elements.clone(); elements.extend_from_slice(&other.elements); Ok(Self { elements }) } } }