Python Modules

You can create a module as follows:

use pyo3::prelude::*;

// add bindings to the generated Python module
// N.B: "rust2py" must be the name of the `.so` or `.pyd` file.

/// This module is implemented in Rust.
#[pymodule]
fn rust2py(py: Python, m: &PyModule) -> PyResult<()> {
    // PyO3 aware function. All of our Python interfaces could be declared in a separate module.
    // Note that the `#[pyfn()]` annotation automatically converts the arguments from
    // Python objects to Rust values, and the Rust return value back into a Python object.
    // The `_py` argument represents that we're holding the GIL.
    #[pyfn(m, "sum_as_string")]
    fn sum_as_string_py(_py: Python, a: i64, b: i64) -> PyResult<String> {
        let out = sum_as_string(a, b);
        Ok(out)
    }

    Ok(())
}

// logic implemented as a normal Rust function
fn sum_as_string(a: i64, b: i64) -> String {
    format!("{}", a + b)
}

fn main() {}

The #[pymodule] procedural macro attribute takes care of exporting the initialization function of your module to Python. It can take as an argument the name of your module, which must be the name of the .so or .pyd file; the default is the Rust function's name.

If the name of the module (the default being the function name) does not match the name of the .so or .pyd file, you will get an import error in Python with the following message: ImportError: dynamic module does not define module export function (PyInit_name_of_your_module)

To import the module, either copy the shared library as described in the README or use a tool, e.g. maturin develop with maturin or python setup.py develop with setuptools-rust.

Documentation

The Rust doc comments of the module initialization function will be applied automatically as the Python docstring of your module.

import rust2py

print(rust2py.__doc__)

Which means that the above Python code will print This module is implemented in Rust..

Modules as objects

In Python, modules are first class objects. This means that you can store them as values or add them to dicts or other modules:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::{wrap_pyfunction, wrap_pymodule};
use pyo3::types::IntoPyDict;

#[pyfunction]
fn subfunction() -> String {
    "Subfunction".to_string()
}

#[pymodule]
fn submodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pyfunction!(subfunction))?;
    Ok(())
}

#[pymodule]
fn supermodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;
    Ok(())
}

fn nested_call() {
    let gil = GILGuard::acquire();
    let py = gil.python();
    let supermodule = wrap_pymodule!(supermodule)(py);
    let ctx = [("supermodule", supermodule)].into_py_dict(py);

    py.run("assert supermodule.submodule.subfunction() == 'Subfunction'", None, Some(&ctx)).unwrap();
}
}

This way, you can create a module hierarchy within a single extension module.

Python Functions

PyO3 supports two ways to define a free function in Python. Both require registering the function to a module.

One way is defining the function in the module definition, annotated with #[pyfn].

use pyo3::prelude::*;

#[pymodule]
fn rust2py(py: Python, m: &PyModule) -> PyResult<()> {

    #[pyfn(m, "sum_as_string")]
    fn sum_as_string_py(_py: Python, a:i64, b:i64) -> PyResult<String> {
        Ok(format!("{}", a + b))
    }

    Ok(())
}

fn main() {}

The other is annotating a function with #[pyfunction] and then adding it to the module using the wrap_pyfunction! macro.

use pyo3::prelude::*;
use pyo3::wrap_pyfunction;

#[pyfunction]
fn double(x: usize) -> usize {
    x * 2
}

#[pymodule]
fn module_with_functions(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(double)).unwrap();

    Ok(())
}

fn main() {}

Argument parsing

Both the #[pyfunction] and #[pyfn] attributes support specifying details of argument parsing. The details are given in the section "Method arguments" in the Classes chapter. Here is an example for a function that accepts arbitrary keyword arguments (**kwargs in Python syntax) and returns the number that was passed:

extern crate pyo3;
use pyo3::prelude::*;
use pyo3::wrap_pyfunction;
use pyo3::types::PyDict;

#[pyfunction(kwds="**")]
fn num_kwds(kwds: Option<&PyDict>) -> usize {
    kwds.map_or(0, |dict| dict.len())
}

#[pymodule]
fn module_with_functions(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(num_kwds)).unwrap();
    Ok(())
}

fn main() {}

Making the function signature available to Python

In order to make the function signature available to Python to be retrieved via inspect.signature, use the #[text_signature] annotation as in the example below. The / signifies the end of positional-only arguments. (This is not a feature of this library in particular, but the general format used by CPython for annotating signatures of built-in functions.)

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

/// This function adds two unsigned 64-bit integers.
#[pyfunction]
#[text_signature = "(a, b, /)"]
fn add(a: u64, b: u64) -> u64 {
    a + b
}
}

This also works for classes and methods:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyType;

// it works even if the item is not documented:

#[pyclass]
#[text_signature = "(c, d, /)"]
struct MyClass {}

#[pymethods]
impl MyClass {
    // the signature for the constructor is attached
    // to the struct definition instead.
    #[new]
    fn new(c: i32, d: &str) -> Self {
        Self {}
    }
    // the self argument should be written $self
    #[text_signature = "($self, e, f)"]
    fn my_method(&self, e: i32, f: i32) -> i32 {
        e + f
    }
    #[classmethod]
    #[text_signature = "(cls, e, f)"]
    fn my_class_method(cls: &PyType, e: i32, f: i32) -> i32 {
        e + f
    }
    #[staticmethod]
    #[text_signature = "(e, f)"]
    fn my_static_method(e: i32, f: i32) -> i32 {
        e + f
    }
}
}

Making the function signature available to Python (old method)

Alternatively, simply make sure the first line of your docstring is formatted like in the following example. Please note that the newline after the -- is mandatory. The / signifies the end of positional-only arguments.

#[text_signature] should be preferred, since it will override automatically generated signatures when those are added in a future version of PyO3.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

/// add(a, b, /)
/// --
///
/// This function adds two unsigned 64-bit integers.
#[pyfunction]
fn add(a: u64, b: u64) -> u64 {
    a + b
}

// a function with a signature but without docs. Both blank lines after the `--` are mandatory.

/// sub(a, b, /)
/// --
///
///
#[pyfunction]
fn sub(a: u64, b: u64) -> u64 {
    a - b
}
}

When annotated like this, signatures are also correctly displayed in IPython.

>>> pyo3_test.add?
Signature: pyo3_test.add(a, b, /)
Docstring: This function adds two unsigned 64-bit integers.
Type:      builtin_function_or_method

Closures

Currently, there are no conversions between Fns in Rust and callables in Python. This would definitely be possible and very useful, so contributions are welcome. In the meantime, you can do the following:

Calling Python functions in Rust

You can use PyAny::is_callable to check if you have a callable object. is_callable will return true for functions (including lambdas), methods and objects with a __call__ method. You can call the object with PyAny::call with the args as first parameter and the kwargs (or None) as second parameter. There are also PyAny::call0 with no args and PyAny::call1 with only positional args.

Calling Rust functions in Python

If you have a static function, you can expose it with #[pyfunction] and use wrap_pyfunction! to get the corresponding PyObject. For dynamic functions, e.g. lambdas and functions that were passed as arguments, you must put them in some kind of owned container, e.g. a Box. (A long-term solution will be a special container similar to wasm-bindgen's Closure). You can then use a #[pyclass] struct with that container as a field as a way to pass the function over the FFI barrier. You can even make that class callable with __call__ so it looks like a function in Python code.

Python Classes

Defining a new class

To define a custom Python class, a Rust struct needs to be annotated with the #[pyclass] attribute.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    num: i32,
    debug: bool,
}
}

Because Python objects are freely shared between threads by the Python interpreter, all structs annotated with #[pyclass] must implement Send.

The above example generates implementations for PyTypeInfo, PyTypeObject, and PyClass for MyClass. To see these generated implementations, refer to the section How methods are implemented at the end of this chapter.

Adding the class to a module

Custom Python classes can then be added to a module using add_class().

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   debug: bool,
}
#[pymodule]
fn mymodule(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<MyClass>()?;
    Ok(())
}
}

PyCell and interior mutability

You sometimes need to convert your pyclass into a Python object and access it from Rust code (e.g., for testing it). PyCell is the primary interface for that.

PyCell<T: PyClass> is always allocated in the Python heap, so Rust doesn't have ownership of it. In other words, Rust code can only extract a &PyCell<T>, not a PyCell<T>.

Thus, to mutate data behind &PyCell safely, PyO3 employs the Interior Mutability Pattern like RefCell.

Users who are familiar with RefCell can use PyCell just like RefCell.

For users who are not very familiar with RefCell, here is a reminder of Rust's rules of borrowing:

• At any given time, you can have either (but not both of) one mutable reference or any number of immutable references.
• References must always be valid.

PyCell, like RefCell, ensures these borrowing rules by tracking references at runtime.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyDict;
#[pyclass]
struct MyClass {
    #[pyo3(get)]
    num: i32,
    debug: bool,
}
let gil = Python::acquire_gil();
let py = gil.python();
let obj = PyCell::new(py, MyClass { num: 3, debug: true }).unwrap();
{
    let obj_ref = obj.borrow(); // Get PyRef
    assert_eq!(obj_ref.num, 3);
    // You cannot get PyRefMut unless all PyRefs are dropped
    assert!(obj.try_borrow_mut().is_err());
}
{
    let mut obj_mut = obj.borrow_mut(); // Get PyRefMut
    obj_mut.num = 5;
    // You cannot get any other refs until the PyRefMut is dropped
    assert!(obj.try_borrow().is_err());
    assert!(obj.try_borrow_mut().is_err());
}

// You can convert `&PyCell` to a Python object
pyo3::py_run!(py, obj, "assert obj.num == 5")
}

&PyCell<T> is bounded by the same lifetime as a GILGuard. To make the object longer lived (for example, to store it in a struct on the Rust side), you can use Py<T>, which stores an object longer than the GIL lifetime, and therefore needs a Python<'_> token to access.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    num: i32,
}
fn return_myclass() -> Py<MyClass> {
    let gil = Python::acquire_gil();
    let py = gil.python();
    Py::new(py, MyClass { num: 1 }).unwrap()
}
let gil = Python::acquire_gil();
let obj = return_myclass();
let cell = obj.as_ref(gil.python()); // AsPyRef::as_ref returns &PyCell
let obj_ref = cell.borrow(); // Get PyRef<T>
assert_eq!(obj_ref.num, 1);
}

Customizing the class

The #[pyclass] macro accepts the following parameters:

• name=XXX - Set the class name shown in Python code. By default, the struct name is used as the class name.
• freelist=XXX - The freelist parameter adds support of free allocation list to custom class. The performance improvement applies to types that are often created and deleted in a row, so that they can benefit from a freelist. XXX is a number of items for the free list.
• gc - Classes with the gc parameter participate in Python garbage collection. If a custom class contains references to other Python objects that can be collected, the PyGCProtocol trait has to be implemented.
• weakref - Adds support for Python weak references.
• extends=BaseType - Use a custom base class. The base BaseType must implement PyTypeInfo.
• subclass - Allows Python classes to inherit from this class.
• dict - Adds __dict__ support, so that the instances of this type have a dictionary containing arbitrary instance variables.
• unsendable - Making it safe to expose !Send structs to Python, where all object can be accessed by multiple threads. A class marked with unsendable panics when accessed by another thread.
• module="XXX" - Set the name of the module the class will be shown as defined in. If not given, the class will be a virtual member of the builtins module.

Constructor

By default it is not possible to create an instance of a custom class from Python code. To declare a constructor, you need to define a method and annotate it with the #[new] attribute. Only Python's __new__ method can be specified, __init__ is not available.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    num: i32,
}

#[pymethods]
impl MyClass {
    #[new]
    fn new(num: i32) -> Self {
        MyClass { num }
    }
}
}

If no method marked with #[new] is declared, object instances can only be created from Rust, but not from Python.

For arguments, see the Method arguments section below.

Return type

Generally, #[new] method have to return T: Into<PyClassInitializer<Self>> or PyResult<T> where T: Into<PyClassInitializer<Self>>.

For constructors that may fail, you should wrap the return type in a PyResult as well. Consult the table below to determine which type your constructor should return:

Inheritance

By default, PyAny is used as the base class. To override this default, use the extends parameter for pyclass with the full path to the base class.

For convenience, (T, U) implements Into<PyClassInitializer<T>> where U is the baseclass of T. But for more deeply nested inheritance, you have to return PyClassInitializer<T> explicitly.

To get a parent class from a child, use PyRef instead of &self for methods, or PyRefMut instead of &mut self. Then you can access a parent class by self_.as_ref() as &Self::BaseClass, or by self_.into_super() as PyRef<Self::BaseClass>.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct BaseClass {
    val1: usize,
}

#[pymethods]
impl BaseClass {
    #[new]
    fn new() -> Self {
        BaseClass { val1: 10 }
    }

    pub fn method(&self) -> PyResult<usize> {
        Ok(self.val1)
    }
}

#[pyclass(extends=BaseClass)]
struct SubClass {
    val2: usize,
}

#[pymethods]
impl SubClass {
    #[new]
    fn new() -> (Self, BaseClass) {
        (SubClass { val2: 15 }, BaseClass::new())
    }

    fn method2(self_: PyRef<Self>) -> PyResult<usize> {
        let super_ = self_.as_ref();  // Get &BaseClass
        super_.method().map(|x| x * self_.val2)
    }
}

#[pyclass(extends=SubClass)]
struct SubSubClass {
    val3: usize,
}

#[pymethods]
impl SubSubClass {
    #[new]
    fn new() -> PyClassInitializer<Self> {
        PyClassInitializer::from(SubClass::new())
            .add_subclass(SubSubClass{val3: 20})
    }

    fn method3(self_: PyRef<Self>) -> PyResult<usize> {
        let v = self_.val3;
        let super_ = self_.into_super();  // Get PyRef<SubClass>
        SubClass::method2(super_).map(|x| x * v)
    }
}
let gil = Python::acquire_gil();
let py = gil.python();
let subsub = pyo3::PyCell::new(py, SubSubClass::new()).unwrap();
pyo3::py_run!(py, subsub, "assert subsub.method3() == 3000")
}

You can also inherit native types such as PyDict, if they implement PySizedLayout.

However, because of some technical problems, we don't currently provide safe upcasting methods for types that inherit native types. Even in such cases, you can unsafely get a base class by raw pointer conversion.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyDict;
use pyo3::{AsPyPointer, PyNativeType};
use std::collections::HashMap;

#[pyclass(extends=PyDict)]
#[derive(Default)]
struct DictWithCounter {
    counter: HashMap<String, usize>,
}

#[pymethods]
impl DictWithCounter {
    #[new]
    fn new() -> Self {
        Self::default()
    }
    fn set(mut self_: PyRefMut<Self>, key: String, value: &PyAny) -> PyResult<()> {
        self_.counter.entry(key.clone()).or_insert(0);
        let py = self_.py();
        let dict: &PyDict = unsafe { py.from_borrowed_ptr_or_err(self_.as_ptr())? };
        dict.set_item(key, value)
    }
}
let gil = Python::acquire_gil();
let py = gil.python();
let cnt = pyo3::PyCell::new(py, DictWithCounter::new()).unwrap();
pyo3::py_run!(py, cnt, "cnt.set('abc', 10); assert cnt['abc'] == 10")
}

If SubClass does not provide a baseclass initialization, the compilation fails.

# use pyo3::prelude::*;

#[pyclass]
struct BaseClass {
    val1: usize,
}

#[pyclass(extends=BaseClass)]
struct SubClass {
    val2: usize,
}

#[pymethods]
impl SubClass {
    #[new]
    fn new() -> Self {
        SubClass { val2: 15 }
    }
}

Object properties

PyO3 supports two ways to add properties to your #[pyclass]:

• For simple fields with no side effects, a #[pyo3(get, set)] attribute can be added directly to the field definition in the #[pyclass].
• For properties which require computation you can define #[getter] and #[setter] functions in the #[pymethods] block.

We'll cover each of these in the following sections.

Object properties using #[pyo3(get, set)]

For simple cases where a member variable is just read and written with no side effects, you can declare getters and setters in your #[pyclass] field definition using the pyo3 attribute, like in the example below:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    #[pyo3(get, set)]
    num: i32
}
}

The above would make the num property available for reading and writing from Python code as self.num.

Properties can be readonly or writeonly by using just #[pyo3(get)] or #[pyo3(set)] respectively.

To use these annotations, your field type must implement some conversion traits:

• For get the field type must implement both IntoPy<PyObject> and Clone.
• For set the field type must implement FromPyObject.

Object properties using #[getter] and #[setter]

For cases which don't satisfy the #[pyo3(get, set)] trait requirements, or need side effects, descriptor methods can be defined in a #[pymethods] impl block.

This is done using the #[getter] and #[setter] attributes, like in the example below:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    num: i32,
}

#[pymethods]
impl MyClass {
    #[getter]
    fn num(&self) -> PyResult<i32> {
        Ok(self.num)
    }
}
}

A getter or setter's function name is used as the property name by default. There are several ways how to override the name.

If a function name starts with get_ or set_ for getter or setter respectively, the descriptor name becomes the function name with this prefix removed. This is also useful in case of Rust keywords like type (raw identifiers can be used since Rust 2018).

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    num: i32,
}
#[pymethods]
impl MyClass {
    #[getter]
    fn get_num(&self) -> PyResult<i32> {
        Ok(self.num)
    }

    #[setter]
    fn set_num(&mut self, value: i32) -> PyResult<()> {
        self.num = value;
        Ok(())
    }
}
}

In this case, a property num is defined and available from Python code as self.num.

Both the #[getter] and #[setter] attributes accept one parameter. If this parameter is specified, it is used as the property name, i.e.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
}
#[pymethods]
impl MyClass {
    #[getter(number)]
    fn num(&self) -> PyResult<i32> {
        Ok(self.num)
    }

    #[setter(number)]
    fn set_num(&mut self, value: i32) -> PyResult<()> {
        self.num = value;
        Ok(())
    }
}
}

In this case, the property number is defined and available from Python code as self.number.

Instance methods

To define a Python compatible method, an impl block for your struct has to be annotated with the #[pymethods] attribute. PyO3 generates Python compatible wrappers for all functions in this block with some variations, like descriptors, class method static methods, etc.

Since Rust allows any number of impl blocks, you can easily split methods between those accessible to Python (and Rust) and those accessible only to Rust.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    num: i32,
}
#[pymethods]
impl MyClass {
    fn method1(&self) -> PyResult<i32> {
        Ok(10)
    }

    fn set_method(&mut self, value: i32) -> PyResult<()> {
        self.num = value;
        Ok(())
    }
}
}

Calls to these methods are protected by the GIL, so both &self and &mut self can be used. The return type must be PyResult<T> or T for some T that implements IntoPy<PyObject>; the latter is allowed if the method cannot raise Python exceptions.

A Python parameter can be specified as part of method signature, in this case the py argument gets injected by the method wrapper, e.g.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    num: i32,
    debug: bool,
}
#[pymethods]
impl MyClass {
    fn method2(&self, py: Python) -> PyResult<i32> {
        Ok(10)
    }
}
}

From the Python perspective, the method2 in this example does not accept any arguments.

Class methods

To create a class method for a custom class, the method needs to be annotated with the #[classmethod] attribute. This is the equivalent of the Python decorator @classmethod.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyType;
#[pyclass]
struct MyClass {
    num: i32,
    debug: bool,
}
#[pymethods]
impl MyClass {
    #[classmethod]
    fn cls_method(cls: &PyType) -> PyResult<i32> {
        Ok(10)
    }
}
}

Declares a class method callable from Python.

• The first parameter is the type object of the class on which the method is called. This may be the type object of a derived class.
• The first parameter implicitly has type &PyType.
• For details on parameter-list, see the documentation of Method arguments section.
• The return type must be PyResult<T> or T for some T that implements IntoPy<PyObject>.

Static methods

To create a static method for a custom class, the method needs to be annotated with the #[staticmethod] attribute. The return type must be T or PyResult<T> for some T that implements IntoPy<PyObject>.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
    num: i32,
    debug: bool,
}
#[pymethods]
impl MyClass {
    #[staticmethod]
    fn static_method(param1: i32, param2: &str) -> PyResult<i32> {
        Ok(10)
    }
}
}

Class attributes

To create a class attribute (also called class variable), a method without any arguments can be annotated with the #[classattr] attribute. The return type must be T for some T that implements IntoPy<PyObject>.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
    #[classattr]
    fn my_attribute() -> String {
        "hello".to_string()
    }
}

let gil = Python::acquire_gil();
let py = gil.python();
let my_class = py.get_type::<MyClass>();
pyo3::py_run!(py, my_class, "assert my_class.my_attribute == 'hello'")
}

Note that unlike class variables defined in Python code, class attributes defined in Rust cannot be mutated at all:

// Would raise a `TypeError: can't set attributes of built-in/extension type 'MyClass'`
pyo3::py_run!(py, my_class, "my_class.my_attribute = 'foo'")

If the class attribute is defined with const code only, one can also annotate associated constants:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
    #[classattr]
    const MY_CONST_ATTRIBUTE: &'static str = "foobar";
}
}

Callable objects

To specify a custom __call__ method for a custom class, the method needs to be annotated with the #[call] attribute. Arguments of the method are specified as for instance methods.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyTuple;
#[pyclass]
struct MyClass {
    num: i32,
    debug: bool,
}
#[pymethods]
impl MyClass {
    #[call]
    #[args(args="*")]
    fn __call__(&self, args: &PyTuple) -> PyResult<i32> {
        println!("MyClass has been called");
        Ok(self.num)
    }
}
}

Method arguments

By default, PyO3 uses function signatures to determine which arguments are required. Then it scans the incoming args and kwargs parameters. If it can not find all required parameters, it raises a TypeError exception. It is possible to override the default behavior with the #[args(...)] attribute. This attribute accepts a comma separated list of parameters in the form of attr_name="default value". Each parameter has to match the method parameter by name.

Each parameter can be one of the following types:

• "*": var arguments separator, each parameter defined after "*" is a keyword-only parameter. Corresponds to python's def meth(*, arg1.., arg2=..).
• args="*": "args" is var args, corresponds to Python's def meth(*args). Type of the args parameter has to be &PyTuple.
• kwargs="**": "kwargs" receives keyword arguments, corresponds to Python's def meth(**kwargs). The type of the kwargs parameter has to be Option<&PyDict>.
• arg="Value": arguments with default value. Corresponds to Python's def meth(arg=Value). If the arg argument is defined after var arguments, it is treated as a keyword-only argument. Note that Value has to be valid rust code, PyO3 just inserts it into the generated code unmodified.

Example:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyTuple};

#[pyclass]
struct MyClass {
    num: i32,
    debug: bool,
}
#[pymethods]
impl MyClass {
    #[new]
    #[args(num = "-1", debug = "true")]
    fn new(num: i32, debug: bool) -> Self {
        MyClass { num, debug }
    }

    #[args(
        num = "10",
        debug = "true",
        py_args = "*",
        name = "\"Hello\"",
        py_kwargs = "**"
    )]
    fn method(
        &mut self,
        num: i32,
        debug: bool,
        name: &str,
        py_args: &PyTuple,
        py_kwargs: Option<&PyDict>,
    ) -> PyResult<String> {
        self.debug = debug;
        self.num = num;
        Ok(format!(
            "py_args={:?}, py_kwargs={:?}, name={}, num={}, debug={}",
            py_args, py_kwargs, name, self.num, self.debug
        ))
    }

    fn make_change(&mut self, num: i32, debug: bool) -> PyResult<String> {
        self.num = num;
        self.debug = debug;
        Ok(format!("num={}, debug={}", self.num, self.debug))
    }
}
}

N.B. the position of the "*" argument (if included) controls the system of handling positional and keyword arguments. In Python:

import mymodule

mc = mymodule.MyClass()
print(mc.method(44, False, "World", 666, x=44, y=55))
print(mc.method(num=-1, name="World"))
print(mc.make_change(44, False))
print(mc.make_change(debug=False, num=-1))

Produces output:

py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44, debug=false
py_args=(), py_kwargs=None, name=World, num=-1, debug=true
num=44, debug=false
num=-1, debug=false

Class customizations

Python's object model defines several protocols for different object behavior, like sequence, mapping or number protocols. PyO3 defines separate traits for each of them. To provide specific Python object behavior, you need to implement the specific trait for your struct. Important note, each protocol implementation block has to be annotated with the #[pyproto] attribute.

All #[pyproto] methods which can be defined below can return T instead of PyResult<T> if the method implementation is infallible. In addition, if the return type is (), it can be omitted altogether.

Basic object customization

The PyObjectProtocol trait provides several basic customizations.

Attribute access

To customize object attribute access, define the following methods:

• fn __getattr__(&self, name: FromPyObject) -> PyResult<impl IntoPy<PyObject>>
• fn __setattr__(&mut self, name: FromPyObject, value: FromPyObject) -> PyResult<()>
• fn __delattr__(&mut self, name: FromPyObject) -> PyResult<()>

Each method corresponds to Python's self.attr, self.attr = value and del self.attr code.

String Conversions

• fn __repr__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>

• fn __str__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>

  Possible return types for __str__ and __repr__ are PyResult<String> or PyResult<PyString>.

• fn __bytes__(&self) -> PyResult<PyBytes>

  Provides the conversion to bytes.

• fn __format__(&self, format_spec: &str) -> PyResult<impl ToPyObject<ObjectType=PyString>>

  Special method that is used by the format() builtin and the str.format() method. Possible return types are PyResult<String> or PyResult<PyString>.

Comparison operators

• fn __richcmp__(&self, other: impl FromPyObject, op: CompareOp) -> PyResult<impl ToPyObject>

  Overloads Python comparison operations (==, !=, <, <=, >, and >=). The op argument indicates the comparison operation being performed. The return type will normally be PyResult<bool>, but any Python object can be returned. If other is not of the type specified in the signature, the generated code will automatically return NotImplemented.

• fn __hash__(&self) -> PyResult<impl PrimInt>

  Objects that compare equal must have the same hash value. The return type must be PyResult<T> where T is one of Rust's primitive integer types.

Other methods

• fn __bool__(&self) -> PyResult<bool>

  Determines the "truthyness" of the object.

Garbage Collector Integration

If your type owns references to other Python objects, you will need to integrate with Python's garbage collector so that the GC is aware of those references. To do this, implement the PyGCProtocol trait for your struct. It includes two methods __traverse__ and __clear__. These correspond to the slots tp_traverse and tp_clear in the Python C API. __traverse__ must call visit.call() for each reference to another Python object. __clear__ must clear out any mutable references to other Python objects (thus breaking reference cycles). Immutable references do not have to be cleared, as every cycle must contain at least one mutable reference. Example:

#![allow(unused)]
fn main() {
extern crate pyo3;

use pyo3::prelude::*;
use pyo3::PyTraverseError;
use pyo3::gc::{PyGCProtocol, PyVisit};

#[pyclass]
struct ClassWithGCSupport {
    obj: Option<PyObject>,
}

#[pyproto]
impl PyGCProtocol for ClassWithGCSupport {
    fn __traverse__(&self, visit: PyVisit) -> Result<(), PyTraverseError> {
        if let Some(ref obj) = self.obj {
            visit.call(obj)?
        }
        Ok(())
    }

    fn __clear__(&mut self) {
        if let Some(obj) = self.obj.take() {
            // Release reference, this decrements ref counter.
            let gil = GILGuard::acquire();
            let py = gil.python();
            py.release(obj);
        }
    }
}
}

Special protocol trait implementations have to be annotated with the #[pyproto] attribute.

It is also possible to enable GC for custom classes using the gc parameter of the pyclass attribute. i.e. #[pyclass(gc)]. In that case instances of custom class participate in Python garbage collection, and it is possible to track them with gc module methods. When using the gc parameter, it is required to implement the PyGCProtocol trait, failure to do so will result in an error at compile time:

#[pyclass(gc)]
struct GCTracked {} // Fails because it does not implement PyGCProtocol

Iterator Types

Iterators can be defined using the PyIterProtocol trait. It includes two methods __iter__ and __next__:

• fn __iter__(slf: PyRefMut<Self>) -> PyResult<impl IntoPy<PyObject>>
• fn __next__(slf: PyRefMut<Self>) -> PyResult<Option<impl IntoPy<PyObject>>>

Returning None from __next__ indicates that that there are no further items. These two methods can be take either PyRef<Self> or PyRefMut<Self> as their first argument, so that mutable borrow can be avoided if needed.

Example:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::PyIterProtocol;

#[pyclass]
struct MyIterator {
    iter: Box<Iterator<Item = PyObject> + Send>,
}

#[pyproto]
impl PyIterProtocol for MyIterator {
    fn __iter__(slf: PyRef<Self>) -> Py<MyIterator> {
        slf.into()
    }
    fn __next__(mut slf: PyRefMut<Self>) -> Option<PyObject> {
        slf.iter.next()
    }
}
}

In many cases you'll have a distinction between the type being iterated over (i.e. the iterable) and the iterator it provides. In this case, you should implement PyIterProtocol for both the iterable and the iterator, but the iterable only needs to support __iter__() while the iterator must support both __iter__() and __next__(). The default implementations in PyIterProtocol will ensure that the objects behave correctly in Python. For example:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::PyIterProtocol;

#[pyclass]
struct Iter {
    inner: std::vec::IntoIter<usize>,
}

#[pyproto]
impl PyIterProtocol for Iter {
    fn __iter__(slf: PyRefMut<Self>) -> Py<Iter> {
        slf.into()
    }

    fn __next__(mut slf: PyRefMut<Self>) -> Option<usize> {
        slf.inner.next()
    }
}

#[pyclass]
struct Container {
    iter: Vec<usize>,
}

#[pyproto]
impl PyIterProtocol for Container {
    fn __iter__(slf: PyRefMut<Self>) -> PyResult<Py<Iter>> {
        let iter = Iter {
            inner: slf.iter.clone().into_iter(),
        };
        Py::new(slf.py(), iter)
    }
}

let gil = Python::acquire_gil();
let py = gil.python();
let inst = pyo3::PyCell::new(
    py,
    Container {
        iter: vec![1, 2, 3, 4],
    },
)
.unwrap();
pyo3::py_run!(py, inst, "assert list(inst) == [1, 2, 3, 4]");
pyo3::py_run!(py, inst, "assert list(iter(iter(inst))) == [1, 2, 3, 4]");
}

For more details on Python's iteration protocols, check out the "Iterator Types" section of the library documentation.

Returning a value from iteration

This guide has so far shown how to use Option<T> to implement yielding values during iteration. In Python a generator can also return a value. To express this in Rust, PyO3 provides the IterNextOutput enum to both Yield values and Return a final value - see its docs for further details and an example.

How methods are implemented

Users should be able to define a #[pyclass] with or without #[pymethods], while PyO3 needs a trait with a function that returns all methods. Since it's impossible to make the code generation in pyclass dependent on whether there is an impl block, we'd need to implement the trait on #[pyclass] and override the implementation in #[pymethods]. To enable this, we use a static registry type provided by inventory, which allows us to collect impls from arbitrary source code by exploiting some binary trick. See inventory: how it works and pyo3_derive_backend::py_class for more details. Also for #[pyproto], we use a similar, but more task-specific registry and initialize it using the ctor crate.

Specifically, the following implementation is generated:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

/// Class for demonstration
struct MyClass {
    num: i32,
    debug: bool,
}

impl pyo3::pyclass::PyClassAlloc for MyClass {}

unsafe impl pyo3::PyTypeInfo for MyClass {
    type Type = MyClass;
    type BaseType = PyAny;
    type BaseLayout = pyo3::pycell::PyCellBase<PyAny>;
    type Layout = PyCell<Self>;
    type Initializer = PyClassInitializer<Self>;
    type AsRefTarget = PyCell<Self>;

    const NAME: &'static str = "MyClass";
    const MODULE: Option<&'static str> = None;
    const DESCRIPTION: &'static str = "Class for demonstration";
    const FLAGS: usize = 0;

    #[inline]
    fn type_object_raw(py: pyo3::Python) -> *mut pyo3::ffi::PyTypeObject {
        use pyo3::type_object::LazyStaticType;
        static TYPE_OBJECT: LazyStaticType = LazyStaticType::new();
        TYPE_OBJECT.get_or_init::<Self>(py)
    }
}

impl pyo3::pyclass::PyClass for MyClass {
    type Dict = pyo3::pyclass_slots::PyClassDummySlot;
    type WeakRef = pyo3::pyclass_slots::PyClassDummySlot;
    type BaseNativeType = PyAny;
}

impl pyo3::IntoPy<PyObject> for MyClass {
    fn into_py(self, py: pyo3::Python) -> pyo3::PyObject {
        pyo3::IntoPy::into_py(pyo3::Py::new(py, self).unwrap(), py)
    }
}

pub struct Pyo3MethodsInventoryForMyClass {
    methods: &'static [pyo3::class::PyMethodDefType],
}
impl pyo3::class::methods::PyMethodsInventory for Pyo3MethodsInventoryForMyClass {
    fn new(methods: &'static [pyo3::class::PyMethodDefType]) -> Self {
        Self { methods }
    }
    fn get(&self) -> &'static [pyo3::class::PyMethodDefType] {
        self.methods
    }
}
impl pyo3::class::methods::HasMethodsInventory for MyClass {
    type Methods = Pyo3MethodsInventoryForMyClass;
}
pyo3::inventory::collect!(Pyo3MethodsInventoryForMyClass);

impl pyo3::class::proto_methods::HasProtoRegistry for MyClass {
    fn registry() -> &'static pyo3::class::proto_methods::PyProtoRegistry {
        static REGISTRY: pyo3::class::proto_methods::PyProtoRegistry
            = pyo3::class::proto_methods::PyProtoRegistry::new();
        &REGISTRY
    }
}

impl pyo3::pyclass::PyClassSend for MyClass {
    type ThreadChecker = pyo3::pyclass::ThreadCheckerStub<MyClass>;
}
let gil = Python::acquire_gil();
let py = gil.python();
let cls = py.get_type::<MyClass>();
pyo3::py_run!(py, cls, "assert cls.__name__ == 'MyClass'")
}

Type Conversions

In this portion of the guide we'll talk about the mapping of Python types to Rust types offered by PyO3, as well as the traits available to perform conversions between them.

Mapping of Rust types to Python types

When writing functions callable from Python (such as a #[pyfunction] or in a #[pymethods] block), the trait FromPyObject is required for function arguments, and IntoPy<PyObject> is required for function return values.

Consult the tables in the following section to find the Rust types provided by PyO3 which implement these traits.

Argument Types

When accepting a function argument, it is possible to either use Rust library types or PyO3's Python-native types. (See the next section for discussion on when to use each.)

The table below contains the Python type and the corresponding function argument types that will accept them:

There are also a few special types related to the GIL and Rust-defined #[pyclass]es which may come in useful:

For more detail on accepting #[pyclass] values as function arguments, see the section of this guide on Python Classes.

Using Rust library types vs Python-native types

Using Rust library types as function arguments will incur a conversion cost compared to using the Python-native types. Using the Python-native types is almost zero-cost (they just require a type check similar to the Python builtin function isinstance()).

However, once that conversion cost has been paid, the Rust standard library types offer a number of benefits:

• You can write functionality in native-speed Rust code (free of Python's runtime costs).
• You get better interoperability with the rest of the Rust ecosystem.
• You can use Python::allow_threads to release the Python GIL and let other Python threads make progress while your Rust code is executing.
• You also benefit from stricter type checking. For example you can specify Vec<i32>, which will only accept a Python list containing integers. The Python-native equivalent, &PyList, would accept a Python list containing Python objects of any type.

For most PyO3 usage the conversion cost is worth paying to get these benefits. As always, if you're not sure it's worth it in your case, benchmark it!

Returning Rust values to Python

When returning values from functions callable from Python, Python-native types (&PyAny, &PyDict etc.) can be used with zero cost.

Because these types are references, in some situations the Rust compiler may ask for lifetime annotations. If this is the case, you should use Py<PyAny>, Py<PyDict> etc. instead - which are also zero-cost. For all of these Python-native types T, Py<T> can be created from T with an .into() conversion.

If your function is fallible, it should return PyResult<T>, which will raise a Python exception if the Err variant is returned.

Finally, the following Rust types are also able to convert to Python as return values:

Traits

PyO3 provides some handy traits to convert between Python types and Rust types.

.extract() and the FromPyObject trait

The easiest way to convert a Python object to a Rust value is using .extract(). It returns a PyResult with a type error if the conversion fails, so usually you will use something like

let v: Vec<i32> = obj.extract()?;

This method is available for many Python object types, and can produce a wide variety of Rust types, which you can check out in the implementor list of FromPyObject.

FromPyObject is also implemented for your own Rust types wrapped as Python objects (see the chapter about classes). There, in order to both be able to operate on mutable references and satisfy Rust's rules of non-aliasing mutable references, you have to extract the PyO3 reference wrappers PyRef and PyRefMut. They work like the reference wrappers of std::cell::RefCell and ensure (at runtime) that Rust borrows are allowed.

The ToPyObject trait

ToPyObject is a conversion trait that allows various objects to be converted into PyObject. IntoPy<PyObject> serves the same purpose, except that it consumes self.

*args and **kwargs for Python object calls

There are several ways how to pass positional and keyword arguments to a Python object call. PyAny provides two methods:

• call - call any callable Python object.
• call_method - call a specific method on the object, shorthand for get_attr then call.

Both methods need args and kwargs arguments, but there are variants for less complex calls, such as call1 for only args and call0 for no arguments at all.

use pyo3::prelude::*;
use pyo3::types::{PyDict, PyTuple};

struct SomeObject;
impl SomeObject {
    fn new(py: Python) -> PyObject {
        PyDict::new(py).to_object(py)
    }
}

fn main() {
    let arg1 = "arg1";
    let arg2 = "arg2";
    let arg3 = "arg3";

    let gil = Python::acquire_gil();
    let py = gil.python();

    let obj = SomeObject::new(py);

    // call object without empty arguments
    obj.call0(py);

    // call object with PyTuple
    let args = PyTuple::new(py, &[arg1, arg2, arg3]);
    obj.call1(py, args);

    // pass arguments as rust tuple
    let args = (arg1, arg2, arg3);
    obj.call1(py, args);
}

kwargs can be None or Some(&PyDict). You can use the IntoPyDict trait to convert other dict-like containers, e.g. HashMap or BTreeMap, as well as tuples with up to 10 elements and Vecs where each element is a two-element tuple.

use pyo3::prelude::*;
use pyo3::types::{IntoPyDict, PyDict};
use std::collections::HashMap;

struct SomeObject;

impl SomeObject {
    fn new(py: Python) -> PyObject {
        PyDict::new(py).to_object(py)
    }
}

fn main() {
    let key1 = "key1";
    let val1 = 1;
    let key2 = "key2";
    let val2 = 2;

    let gil = Python::acquire_gil();
    let py = gil.python();

    let obj = SomeObject::new(py);

    // call object with PyDict
    let kwargs = [(key1, val1)].into_py_dict(py);
    obj.call(py, (), Some(kwargs));

    // pass arguments as Vec
    let kwargs = vec![(key1, val1), (key2, val2)];
    obj.call(py, (), Some(kwargs.into_py_dict(py)));

    // pass arguments as HashMap
    let mut kwargs = HashMap::<&str, i32>::new();
    kwargs.insert(key1, 1);
    obj.call(py, (), Some(kwargs.into_py_dict(py)));
}

FromPy<T> and IntoPy<T>

Many conversions in PyO3 can't use std::convert::From because they need a GIL token. The FromPy trait offers an from_py method that works just like from, except for taking a Python<'_> argument. I.e. FromPy<T> could be converting a Rust object into a Python object even though it is called FromPy - it doesn't say anything about which side of the conversion is a Python object.

Just like From<T>, if you implement FromPy<T> you gain a blanket implementation of IntoPy for free.

Eventually, traits such as ToPyObject will be replaced by this trait and a FromPy trait will be added that will implement IntoPy, just like with From and Into.

Python Exceptions

Defining a new exception

You can use the create_exception! macro to define a new exception type:

#![allow(unused)]
fn main() {
use pyo3::create_exception;

create_exception!(module, MyError, pyo3::exceptions::Exception);
}

• module is the name of the containing module.
• MyError is the name of the new exception type.

For example:

use pyo3::prelude::*;
use pyo3::create_exception;
use pyo3::types::IntoPyDict;
use pyo3::exceptions::Exception;

create_exception!(mymodule, CustomError, Exception);

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let ctx = [("CustomError", py.get_type::<CustomError>())].into_py_dict(py);

    py.run("assert str(CustomError) == \"<class 'mymodule.CustomError'>\"", None, Some(&ctx)).unwrap();
    py.run("assert CustomError('oops').args == ('oops',)", None, Some(&ctx)).unwrap();
}

Raising an exception

To raise an exception, first you need to obtain an exception type and construct a new PyErr, then call the PyErr::restore method to write the exception back to the Python interpreter's global state.

use pyo3::{Python, PyErr};
use pyo3::exceptions;

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    PyErr::new::<exceptions::TypeError, _>("Error").restore(py);
    assert!(PyErr::occurred(py));
    drop(PyErr::fetch(py));
}

From pyfunctions and pyclass methods, returning an Err(PyErr) is enough; PyO3 will handle restoring the exception on the Python interpreter side.

If you already have a Python exception instance, you can simply call PyErr::from_instance.

PyErr::from_instance(py, err).restore(py);

If a Rust type exists for the exception, then it is possible to use the py_err method. For example, each standard exception defined in the pyo3::exceptions module has a corresponding Rust type, exceptions defined by create_exception! and import_exception! macro have Rust types as well.

#![allow(unused)]
fn main() {
use pyo3::exceptions;
use pyo3::prelude::*;
fn check_for_error() -> bool {false}
fn my_func(arg: PyObject) -> PyResult<()> {
    if check_for_error() {
        Err(exceptions::ValueError::py_err("argument is wrong"))
    } else {
        Ok(())
    }
}
}

Checking exception types

Python has an isinstance method to check an object's type, in PyO3 there is a Python::is_instance method which does the same thing.

use pyo3::Python;
use pyo3::types::{PyBool, PyList};

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    assert!(py.is_instance::<PyBool, _>(PyBool::new(py, true)).unwrap());
    let list = PyList::new(py, &[1, 2, 3, 4]);
    assert!(!py.is_instance::<PyBool, _>(list.as_ref()).unwrap());
    assert!(py.is_instance::<PyList, _>(list.as_ref()).unwrap());
}

Python::is_instance calls the underlying PyType::is_instance method to do the actual work.

To check the type of an exception, you can simply do:

use pyo3::exceptions;
use pyo3::prelude::*;
fn main() {
let gil = Python::acquire_gil();
let py = gil.python();
let err = exceptions::TypeError::py_err(());
err.is_instance::<exceptions::TypeError>(py);
}

Handling Rust errors

The vast majority of operations in this library will return PyResult<T>, which is an alias for the type Result<T, PyErr>.

A PyErr represents a Python exception. Errors within the PyO3 library are also exposed as Python exceptions.

The PyO3 library handles Python exceptions in two stages. During the first stage, a PyErr instance is created. At this stage, holding Python's GIL is not required. During the second stage, an actual Python exception instance is created and set active in the Python interpreter.

In simple cases, for custom errors adding an implementation of std::convert::From<T> trait for this custom error is enough. PyErr::new accepts an argument in the form of ToPyObject + 'static. If the 'static constraint can not be satisfied or more complex arguments are required, the PyErrArguments trait can be implemented. In that case, actual exception argument creation is delayed until a Python object is available.

#![allow(unused)]
fn main() {
use pyo3::{exceptions, PyErr, PyResult};
use std::error::Error;
use std::fmt;

#[derive(Debug)]
struct CustomIOError;

impl Error for CustomIOError {}

impl fmt::Display for CustomIOError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "Oh no!")
    }
}

fn bind(_addr: &str) -> Result<(), CustomIOError> {
    Err(CustomIOError)
}
impl std::convert::From<CustomIOError> for PyErr {
    fn from(err: CustomIOError) -> PyErr {
        exceptions::OSError::py_err(err.to_string())
    }
}

fn connect(s: String) -> PyResult<bool> {
    bind("127.0.0.1:80")?;
    Ok(true)
}
}

The code snippet above will raise an OSError in Python if bind() returns a CustomIOError.

The std::convert::From<T> trait is implemented for most of the Rust standard library's error types so the ? operator can be used.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

fn parse_int(s: String) -> PyResult<usize> {
    Ok(s.parse::<usize>()?)
}
}

The code snippet above will raise a ValueError in Python if String::parse() returns an error.

Using exceptions defined in Python code

It is possible to use an exception defined in Python code as a native Rust type. The import_exception! macro allows importing a specific exception class and defines a zero-sized Rust type for that exception.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::import_exception;

import_exception!(io, UnsupportedOperation);

fn tell(file: PyObject) -> PyResult<u64> {
    use pyo3::exceptions::*;

    let gil = Python::acquire_gil();
    let py = gil.python();

    match file.call_method0(py, "tell") {
        Err(_) => Err(UnsupportedOperation::py_err("not supported: tell")),
        Ok(x) => x.extract::<u64>(py),
    }
}

}

pyo3::exceptions defines exceptions for several standard library modules.

Calling Python in Rust code

These APIs work from Rust whenever you have a Python object handy, whether PyO3 is built for an extension module or not.

Want to access Python APIs? Then use PyModule::import.

Pymodule::import can be used to get handle to a Python module from Rust. You can use this to import and use any Python module available in your environment.

use pyo3::prelude::*;

fn main() -> PyResult<()> {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let builtins = PyModule::import(py, "builtins")?;
    let total: i32 = builtins.call1("sum", (vec![1, 2, 3],))?.extract()?;
    assert_eq!(total, 6);
    Ok(())
}

Want to run just an expression? Then use eval.

Python::eval is a method to execute a Python expression and return the evaluated value as a &PyAny object.

use pyo3::prelude::*;
use pyo3::types::IntoPyDict;

fn main() -> Result<(), ()> {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let result = py.eval("[i * 10 for i in range(5)]", None, None).map_err(|e| {
        e.print_and_set_sys_last_vars(py);
    })?;
    let res: Vec<i64> = result.extract().unwrap();
    assert_eq!(res, vec![0, 10, 20, 30, 40]);
    Ok(())
}

Want to run statements? Then use run.

Python::run is a method to execute one or more Python statements. This method returns nothing (like any Python statement), but you can get access to manipulated objects via the locals dict.

You can also use the py_run! macro, which is a shorthand for Python::run. Since py_run! panics on exceptions, we recommend you use this macro only for quickly testing your Python extensions.

use pyo3::prelude::*;
use pyo3::{PyCell, PyObjectProtocol, py_run};
 fn main() {
#[pyclass]
struct UserData {
    id: u32,
    name: String,
}
#[pymethods]
impl UserData {
    fn as_tuple(&self) -> (u32, String) {
        (self.id, self.name.clone())
    }
}
#[pyproto]
impl PyObjectProtocol for UserData {
    fn __repr__(&self) -> PyResult<String> {
        Ok(format!("User {}(id: {})", self.name, self.id))
    }
}
let gil = Python::acquire_gil();
let py = gil.python();
let userdata = UserData {
    id: 34,
    name: "Yu".to_string(),
};
let userdata = PyCell::new(py, userdata).unwrap();
let userdata_as_tuple = (34, "Yu");
py_run!(py, userdata userdata_as_tuple, r#"
assert repr(userdata) == "User Yu(id: 34)"
assert userdata.as_tuple() == userdata_as_tuple
"#);
}

You have a Python file or Python function? Then use PyModule::from_code.

PyModule::from_code can be used to generate a Python module which can then be used just as if it was imported with PyModule::import.

use pyo3::{prelude::*, types::{IntoPyDict, PyModule}};
 fn main() -> PyResult<()> {
let gil = Python::acquire_gil();
let py = gil.python();
let activators = PyModule::from_code(py, r#"
def relu(x):
    """see https://en.wikipedia.org/wiki/Rectifier_(neural_networks)"""
    return max(0.0, x)

def leaky_relu(x, slope=0.01):
    return x if x >= 0 else x * slope
"#, "activators.py", "activators")?;

let relu_result: f64 = activators.call1("relu", (-1.0,))?.extract()?;
assert_eq!(relu_result, 0.0);

let kwargs = [("slope", 0.2)].into_py_dict(py);
let lrelu_result: f64 = activators
    .call("leaky_relu", (-1.0,), Some(kwargs))?
    .extract()?;
assert_eq!(lrelu_result, -0.2);
Ok(()) }

GIL lifetimes, mutability and Python object types

On first glance, PyO3 provides a huge number of different types that can be used to wrap or refer to Python objects. This page delves into the details and gives an overview of their intended meaning, with examples when each type is best used.

Mutability and Rust types

Since Python has no concept of ownership, and works solely with boxed objects, any Python object can be referenced any number of times, and mutation is allowed from any reference.

The situation is helped a little by the Global Interpreter Lock (GIL), which ensures that only one thread can use the Python interpreter and its API at the same time, while non-Python operations (system calls and extension code) can unlock the GIL. (See the section on parallelism for how to do that in PyO3.)

In PyO3, holding the GIL is modeled by acquiring a token of the type Python<'py>, which serves three purposes:

• It provides some global API for the Python interpreter, such as eval.
• It can be passed to functions that require a proof of holding the GIL, such as PyObject::clone_ref.
• Its lifetime can be used to create Rust references that implicitly guarantee holding the GIL, such as &'py PyAny.

The latter two points are the reason why some APIs in PyO3 require the py: Python argument, while others don't.

The PyO3 API for Python objects is written such that instead of requiring a mutable Rust reference for mutating operations such as PyList::append, a shared reference (which, in turn, can only be created through Python<'_> with a GIL lifetime) is sufficient.

However, Rust structs wrapped as Python objects (called pyclass types) usually do need &mut access. Due to the GIL, PyO3 can guarantee thread-safe acces to them, but it cannot statically guarantee uniqueness of &mut references once an object's ownership has been passed to the Python interpreter, ensuring references is done at runtime using PyCell, a scheme very similar to std::cell::RefCell.

Object types

PyAny

Represents: a Python object of unspecified type, restricted to a GIL lifetime. Currently, PyAny can only ever occur as a reference, &PyAny.

Used: Whenever you want to refer to some Python object and will have the GIL for the whole duration you need to access that object. For example, intermediate values and arguments to pyfunctions or pymethods implemented in Rust where any type is allowed.

Many general methods for interacting with Python objects are on the PyAny struct, such as getattr, setattr, and .call.

Conversions:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyList;
let gil = Python::acquire_gil();
let py = gil.python();
let obj: &PyAny = PyList::empty(py);

// Convert to &ConcreteType using PyAny::downcast
let _: &PyList = obj.downcast().unwrap();

// Convert to PyObject using .into() or .to_object(py)
let _: PyObject = obj.into();

// Convert to Py<PyAny> using .into() or Py::from
let _: Py<PyAny> = obj.into();

// Convert to Py<ConcreteType> using PyAny::extract
let _: Py<PyList> = obj.extract().unwrap();
}

PyTuple, PyDict, and many more

Represents: a native Python object of known type, restricted to a GIL lifetime just like PyAny.

Used: Whenever you want to operate with native Python types while holding the GIL. Like PyAny, this is the most convenient form to use for function arguments and intermediate values.

These types all implement Deref<Target = PyAny>, so they all expose the same methods which can be found on PyAny.

To see all Python types exposed by PyO3 you should consult the pyo3::types module.

Conversions:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyList;
let gil = Python::acquire_gil();
let py = gil.python();
let list = PyList::empty(py);

// Can use methods from PyAny on all Python types due to Deref implementation
let _ = list.repr();

// Rust will convert &PyList etc. to &PyAny automatically due to Deref implementation
let _: &PyAny = list;

// For more explicit &PyAny conversion, use .as_ref()
let _: &PyAny = list.as_ref();

// To convert to PyObject use .into() or .to_object(py)
let _: PyObject = list.into();

// To convert to Py<T> use .into() or Py::from()
let _: Py<PyList> = list.into();
}

PyObject

Represents: a GIL independent reference to a Python object of unspecified type.

Used: Whenever you want to carry around references to "some" Python object, without caring about a GIL lifetime. For example, storing Python object references in a Rust struct that outlives the Python-Rust FFI boundary, or returning objects from functions implemented in Rust back to Python.

Can be cloned using Python reference counts with .clone_ref().

Conversions:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyList;
let gil = Python::acquire_gil();
let py = gil.python();
let obj: PyObject = PyList::empty(py).into();

// Convert to &PyAny using AsPyRef::as_ref
let _: &PyAny = obj.as_ref(py);

// Convert to &ConcreteType using PyObject::cast_as
let _: &PyList = obj.cast_as(py).unwrap();

// Convert to Py<ConcreteType> using PyObject::extract
let _: Py<PyList> = obj.extract(py).unwrap();
}

Py<SomeType>

Represents: a GIL independent reference to a Python object of known type. This can be a Python native type (like PyTuple), or a pyclass type implemented in Rust.

Used: Like PyObject, but with a known inner type.

Conversions:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyList;
let gil = Python::acquire_gil();
let py = gil.python();
let list: Py<PyList> = PyList::empty(py).into();

// Access the native type using AsPyRef::as_ref(py)
// (For #[pyclass] types, as_ref() will return &PyCell<T>)
let _: &PyList = list.as_ref(py);

// Convert to PyObject with .into()
let _: PyObject = list.into();
}

Note: PyObject is semantically equivalent to Py<PyAny> and might be merged with it in the future.

PyCell<SomeType>

Represents: a reference to a Rust object (instance of PyClass) which is wrapped in a Python object. The cell part is an analog to stdlib's RefCell to allow access to &mut references.

Used: for accessing pure-Rust API of the instance (members and functions taking &SomeType or &mut SomeType) while maintaining the aliasing rules of Rust references.

Like pyo3's Python native types, PyCell<T> implements Deref<Target = PyAny>, so it also exposes all of the methods on PyAny.

Conversions:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyList;
#[pyclass] struct MyClass { }
let gil = Python::acquire_gil();
let py = gil.python();
let cell: &PyCell<MyClass> = PyCell::new(py, MyClass { }).unwrap();

// Obtain PyRef<T> with .try_borrow()
let pr: PyRef<MyClass> = cell.try_borrow().unwrap();
drop(pr);

// Obtain PyRefMut<T> with .try_borrow_mut()
let prm: PyRefMut<MyClass> = cell.try_borrow_mut().unwrap();
drop(prm);

// Can use methods from PyAny on PyCell<T> due to Deref implementation
let _ = cell.repr();

// Rust will convert &PyCell<T> to &PyAny automatically due to Deref implementation
let _: &PyAny = cell;

// For more explicit &PyAny conversion, use .as_ref()
let any: &PyAny = cell.as_ref();

// To obtain a PyCell<T> from PyAny, use PyAny::downcast
let _: &PyCell<MyClass> = any.downcast().unwrap();
}

PyRef<SomeType> and PyRefMut<SomeType>

Represents: reference wrapper types employed by PyCell to keep track of borrows, analog to Ref and RefMut used by RefCell.

Used: while borrowing a PyCell. They can also be used with .extract() on types like Py<T> and PyAny to get a reference quickly.

Related traits and types

PyClass

This trait marks structs defined in Rust that are also usable as Python classes, usually defined using the #[pyclass] macro.

PyNativeType

This trait marks structs that mirror native Python types, such as PyList.

Parallelism

CPython has the infamous Global Interpreter Lock, which prevents several threads from executing Python bytecode in parallel. This makes threading in Python a bad fit for CPU-bound tasks and often forces developers to accept the overhead of multiprocessing.

In PyO3 parallelism can be easily achieved in Rust-only code. Let's take a look at our word-count example, where we have a search function that utilizes the rayon crate to count words in parallel.

#[pyfunction]
fn search(contents: &str, needle: &str) -> usize {
    contents
        .par_lines()
        .map(|line| count_line(line, needle))
        .sum()
}

But let's assume you have a long running Rust function which you would like to execute several times in parallel. For the sake of example let's take a sequential version of the word count:

fn search_sequential(contents: &str, needle: &str) -> usize {
    contents.lines().map(|line| count_line(line, needle)).sum()
}

To enable parallel execution of this function, the Python::allow_threads method can be used to temporarily release the GIL, thus allowing other Python threads to run. We then have a function exposed to the Python runtime which calls search_sequential inside a closure passed to Python::allow_threads to enable true parallelism:

#[pyfunction]
fn search_sequential_allow_threads(py: Python, contents: &str, needle: &str) -> usize {
    py.allow_threads(|| search_sequential(contents, needle))
}

Now Python threads can use more than one CPU core, resolving the limitation which usually makes multi-threading in Python only good for IO-bound tasks:

from concurrent.futures import ThreadPoolExecutor
from word_count import search_sequential_allow_threads

executor = ThreadPoolExecutor(max_workers=2)

future_1 = executor.submit(
    word_count.search_sequential_allow_threads, contents, needle
)
future_2 = executor.submit(
    word_count.search_sequential_allow_threads, contents, needle
)
result_1 = future_1.result()
result_2 = future_2.result()

Benchmark

Let's benchmark the word-count example to verify that we really did unlock parallelism with PyO3.

We are using pytest-benchmark to benchmark four word count functions:

1. Pure Python version
2. Rust parallel version
3. Rust sequential version
4. Rust sequential version executed twice with two Python threads

The benchmark script can be found here, and we can run tox in the word-count folder to benchmark these functions.

While the results of the benchmark of course depend on your machine, the relative results should be similar to this (mid 2020):

-------------------------------------------------------------------------------------------------- benchmark: 4 tests -------------------------------------------------------------------------------------------------
Name (time in ms)                                          Min                Max               Mean            StdDev             Median               IQR            Outliers       OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_word_count_rust_parallel                           1.7315 (1.0)       4.6495 (1.0)       1.9972 (1.0)      0.4299 (1.0)       1.8142 (1.0)      0.2049 (1.0)         40;46  500.6943 (1.0)         375           1
test_word_count_rust_sequential                         7.3348 (4.24)     10.3556 (2.23)      8.0035 (4.01)     0.7785 (1.81)      7.5597 (4.17)     0.8641 (4.22)         26;5  124.9457 (0.25)        121           1
test_word_count_rust_sequential_twice_with_threads      7.9839 (4.61)     10.3065 (2.22)      8.4511 (4.23)     0.4709 (1.10)      8.2457 (4.55)     0.3927 (1.92)        17;17  118.3274 (0.24)        114           1
test_word_count_python_sequential                      27.3985 (15.82)    45.4527 (9.78)     28.9604 (14.50)    4.1449 (9.64)     27.5781 (15.20)    0.4638 (2.26)          3;5   34.5299 (0.07)         35           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

You can see that the Python threaded version is not much slower than the Rust sequential version, which means compared to an execution on a single CPU core the speed has doubled.

Debugging

Macros

PyO3's attributes (#[pyclass], #[pymodule], etc.) are procedural macros, which means that they rewrite the source of the annotated item. You can view the generated source with the following command, which also expands a few other things:

cargo rustc --profile=check -- -Z unstable-options --pretty=expanded > expanded.rs; rustfmt expanded.rs

(You might need to install rustfmt if you don't already have it.)

You can also debug classic !-macros by adding -Z trace-macros:

cargo rustc --profile=check -- -Z unstable-options --pretty=expanded -Z trace-macros > expanded.rs; rustfmt expanded.rs

See cargo expand for a more elaborate version of those commands.

Running with Valgrind

Valgrind is a tool to detect memory management bugs such as memory leaks.

You first need to install a debug build of Python, otherwise Valgrind won't produce usable results. In Ubuntu there's e.g. a python3-dbg package.

Activate an environment with the debug interpreter and recompile. If you're on Linux, use ldd with the name of your binary and check that you're linking e.g. libpython3.6dm.so.1.0 instead of libpython3.6m.so.1.0.

Download the suppressions file for cpython.

Run Valgrind with valgrind --suppressions=valgrind-python.supp ./my-command --with-options

Getting a stacktrace

The best start to investigate a crash such as an segmentation fault is a backtrace.

• Link against a debug build of python as described in the previous chapter
• Run gdb <my-binary>
• Enter r to run
• After the crash occurred, enter bt or bt full to print the stacktrace

Advanced topics

FFI

PyO3 exposes much of Python's C API through the ffi module.

The C API is naturally unsafe and requires you to manage reference counts, errors and specific invariants yourself. Please refer to the C API Reference Manual and The Rustonomicon before using any function from that API.

Memory Management

PyO3's "owned references" (&PyAny etc.) make PyO3 more ergonomic to use by ensuring that their lifetime can never be longer than the duration the Python GIL is held. This means that most of PyO3's API can assume the GIL is held. (If PyO3 could not assume this, every PyO3 API would need to take a Python GIL token to prove that the GIL is held.)

The caveat to these "owned references" is that Rust references do not normally convey ownership (they are always Copy, and cannot implement Drop). Whenever a PyO3 API returns an owned reference, PyO3 stores it internally, so that PyO3 can decrease the reference count just before PyO3 releases the GIL.

For most use cases this behaviour is invisible. Occasionally, however, users may need to clear memory usage sooner than PyO3 usually does. PyO3 exposes this functionality with the the GILPool struct. When a GILPool is dropped, all owned references created after the GILPool was created will be cleared.

The unsafe function Python::new_pool allows you to create a new GILPool. When doing this, you must be very careful to ensure that once the GILPool is dropped you do not retain access any owned references created after the GILPool was created.

The nightly feature

The pyo3/nightly feature needs the nightly Rust compiler. This allows PyO3 to use Rust's unstable specialization feature to apply the following optimizations:

• FromPyObject for Vec and [T;N] can perform a memcpy when the object is a PyBuffer
• ToBorrowedObject can skip a reference count increase when the provided object is a Python native type.

Building and Distribution

Python version

PyO3 uses a build script to determine the Python version and set the correct linker arguments. By default it uses the python3 executable. You can override the Python interpreter by setting PYTHON_SYS_EXECUTABLE, e.g., PYTHON_SYS_EXECUTABLE=python3.6.

Linking

Different linker arguments must be set for libraries/extension modules and binaries, which includes both standalone binaries and tests. (More specifically, binaries must be told where to find libpython and libraries must not link to libpython for manylinux compliance).

Since PyO3's build script can't know whether you're building a binary or a library, you have to activate the extension-module feature to get the build options for a library, or it'll default to binary.

If you have e.g. a library crate and a profiling crate alongside, you need to use optional features. E.g. you put the following in the library crate:

[dependencies]
pyo3 = "0.6"

[lib]
name = "hyperjson"
crate-type = ["rlib", "cdylib"]

[features]
default = ["pyo3/extension-module"]

And this in the profiling crate:

[dependencies]
my_main_crate = { path = "..", default-features = false }
pyo3 = "0.6"

On Linux/macOS you might have to change LD_LIBRARY_PATH to include libpython, while on windows you might need to set LIB to include pythonxy.lib (where x and y are major and minor version), which is normally either in the libs or Lib folder of a Python installation.

Distribution

There are two ways to distribute your module as a Python package: the old, setuptools-rust, and the new, maturin. setuptools-rust needs some configuration files (setup.py, MANIFEST.in, build-wheels.sh, etc.) and external tools (docker, twine). maturin doesn't need any configuration files, however it does not support some functionality of setuptools such as package data (pyo3/maturin#258).

Cross Compiling

Cross compiling PyO3 modules is relatively straightforward and requires a few pieces of software:

• A toolchain for your target.
• The appropriate options in your Cargo .config for the platform you're targeting and the toolchain you are using.
• A Python interpreter that's already been compiled for your target.
• The headers that match the above interpreter.

See https://github.com/japaric/rust-cross for a primer on cross compiling Rust in general.

After you've obtained the above, you can build a cross compiled PyO3 module by setting a few extra environment variables:

• PYO3_CROSS_INCLUDE_DIR: This variable must be set to the directory containing the headers for the target's Python interpreter.
• PYO3_CROSS_LIB_DIR: This variable must be set to the directory containing the target's libpython DSO.

An example might look like the following (assuming your target's sysroot is at /home/pyo3/cross/sysroot and that your target is armv7):

export PYO3_CROSS_INCLUDE_DIR="/home/pyo3/cross/sysroot/usr/include"
export PYO3_CROSS_LIB_DIR="/home/pyo3/cross/sysroot/usr/lib"

cargo build --target armv7-unknown-linux-gnueabihf

PyPy Support

Using PyPy is supported via cpyext.

Support is only provided for building Rust extension for code running under PyPy. This means that PyPy cannot be called from rust via cpyext. Note that there some differences in the ffi module between PyPy and CPython.

This is a limitation of cpyext and support for embedding cpyext is not planned.

Compilation against PyPy is done by exporting the PYTHON_SYS_EXECUTABLE to point to a PyPy binary or by compiling in a PyPy virtualenv.

For example, PYTHON_SYS_EXECUTABLE="/path/to/pypy3" /path/to/pypy3 setup.py install

Unsupported features

These are features currently supported by PyO3, but not yet implemented in cpyext.

• Complex number functions (_Py_c_sum, _Py_c_sum ..)
• Conversion to rust's i128, u128 types.
• PySequence_Count (which is used to count number of element in array)
• PyDict_MergeFromSeq2 (used in PyDict::from_sequence)

Frequently Asked Questions / Troubleshooting

I'm experiencing deadlocks using PyO3 with lazy_static or once_cell!

lazy_static and once_cell::sync both use locks to ensure that initialization is performed only by a single thread. Because the Python GIL is an additional lock this can lead to deadlocks in the following way:

1. A thread (thread A) which has acquired the Python GIL starts initialization of a lazy_static value.
2. The initialization code calls some Python API which temporarily releases the GIL e.g. Python::import.
3. Another thread (thread B) acquires the Python GIL and attempts to access the same lazy_static value.
4. Thread B is blocked, because it waits for lazy_static's initialization to lock to release.
5. Thread A is blocked, because it waits to re-aquire the GIL which thread B still holds.
6. Deadlock.

PyO3 provides a struct GILOnceCell which works equivalently to OnceCell but relies solely on the Python GIL for thread safety. This means it can be used in place of lazy_static or once_cell where you are experiencing the deadlock described above. See the documentation for GILOnceCell for an example how to use it.

I can't run cargo test: I'm having linker issues like "Symbol not found" or "Undefined reference to _PyExc_SystemError"!

Currently, #341 causes cargo test to fail with linking errors when the extension-module feature is activated. For now you can work around this by making the extension-module feature optional and running the tests with cargo test --no-default-features:

[dependencies.pyo3]
version = "*"

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

Appendix A: PyO3 and rust-cpython

PyO3 began as fork of rust-cpython when rust-cpython wasn't maintained. Over the time PyO3 has become fundamentally different from rust-cpython.

This chapter is based on the discussion in PyO3/pyo3#55.

Macros

While rust-cpython has a macro_rules! based dsl for declaring modules and classes, PyO3 uses proc macros. PyO3 also doesn't change your struct and functions so you can still use them as normal Rust functions.

rust-cpython

py_class!(class MyClass |py| {
    data number: i32;
    def __new__(_cls, arg: i32) -> PyResult<MyClass> {
        MyClass::create_instance(py, arg)
    }
    def half(&self) -> PyResult<i32> {
        Ok(self.number(py) / 2)
    }
});

pyo3

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct MyClass {
   num: u32,
}

#[pymethods]
impl MyClass {
    #[new]
    fn new(num: u32) -> Self {
        MyClass { num }
    }

    fn half(&self) -> PyResult<u32> {
        Ok(self.num / 2)
    }
}
}

Ownership and lifetimes

While in rust-cpython you always own python objects, PyO3 allows efficient borrowed objects and most APIs are available with references.

Here is an example of the PyList API:

rust-cpython

impl PyList {

   fn new(py: Python) -> PyList {...}

   fn get_item(&self, py: Python, index: isize) -> PyObject {...}
}

pyo3

impl PyList {

   fn new(py: Python) -> &PyList {...}

   fn get_item(&self, index: isize) -> &PyAny {...}
}

In PyO3, all object references are bounded by the GIL lifetime. So the owned Python object is not required, and it is safe to have functions like fn py<'p>(&'p self) -> Python<'p> {}.

Error handling

rust-cpython requires a Python parameter for constructing a PyErr, so error handling ergonomics is pretty bad. It is not possible to use ? with Rust errors.

PyO3 on other hand does not require Python for constructing a PyErr, it is only required if you want to raise an exception in Python with the PyErr::restore() method. Due to various std::convert::From<E> for PyErr implementations for Rust standard error types E, propagating ? is supported automatically.

Appendix B: Migrating from older PyO3 versions

This guide can help you upgrade code through breaking changes from one PyO3 version to the next. For a detailed list of all changes, see CHANGELOG.md

from 0.10.* to 0.11

Stable Rust

PyO3 now supports the stable Rust toolchain. The minimum required version is 1.39.0.

#[pyclass] structs must now be Send or unsendable

Because #[pyclass] structs can be sent between threads by the Python interpreter, they must implement Send or declared as unsendable (by #[pyclass(unsendable)]). Note that unsendable is added in PyO3 0.11.1 and Send is always required in PyO3 0.11.0.

This may "break" some code which previously was accepted, even though it could be unsound. There can be two fixes:

1. If you think that your #[pyclass] actually must be Sendable, then let's implement Send. A common, safer way is using thread-safe types. E.g., Arc instead of Rc, Mutex instead of RefCell, and Box<dyn Send + T> instead of Box<dyn T>.

   Before:

   
   #![allow(unused)]
   fn main() {
   use pyo3::prelude::*;
   use std::rc::Rc;
   use std::cell::RefCell;
   
   #[pyclass]
   struct NotThreadSafe {
       shared_bools: Rc<RefCell<Vec<bool>>>,
       closure: Box<dyn Fn()>
   }
   }
   

   After:

   
   #![allow(unused)]
   fn main() {
   use pyo3::prelude::*;
   use std::sync::{Arc, Mutex};
   
   #[pyclass]
   struct ThreadSafe {
       shared_bools: Arc<Mutex<Vec<bool>>>,
       closure: Box<dyn Fn() + Send>
   }
   }
   

   In situations where you cannot change your #[pyclass] to automatically implement Send (e.g., when it contains a raw pointer), you can use unsafe impl Send. In such cases, care should be taken to ensure the struct is actually thread safe. See the Rustnomicon for more.

2. If you think that your #[pyclass] should not be accessed by another thread, you can use unsendable flag. A class marked with unsendable panics when accessed by another thread, making it thread-safe to expose an unsendable object to the Python interpreter.

   Before:

   
   #![allow(unused)]
   fn main() {
   use pyo3::prelude::*;
   
   #[pyclass]
   struct Unsendable {
       pointers: Vec<*mut std::os::raw::c_char>,
   }
   }
   

   After:

   
   #![allow(unused)]
   fn main() {
   use pyo3::prelude::*;
   
   #[pyclass(unsendable)]
   struct Unsendable {
       pointers: Vec<*mut std::os::raw::c_char>,
   }
   }
   

All PyObject and Py<T> methods now take Python as an argument

Previously, a few methods such as Object::get_refcnt did not take Python as an argument (to ensure that the Python GIL was held by the current thread). Technically, this was not sound. To migrate, just pass a py argument to any calls to these methods.

Before:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

let gil = Python::acquire_gil();
let py = gil.python();

py.None().get_refcnt();
}

After:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

let gil = Python::acquire_gil();
let py = gil.python();

py.None().get_refcnt(py);
}

from 0.9.* to 0.10

ObjectProtocol is removed

All methods are moved to PyAny. And since now all native types (e.g., PyList) implements Deref<Target=PyAny>, all you need to do is remove ObjectProtocol from your code. Or if you use ObjectProtocol by use pyo3::prelude::*, you have to do nothing.

Before:

#![allow(unused)]
fn main() {
use pyo3::ObjectProtocol;

let gil = pyo3::Python::acquire_gil();
let obj = gil.python().eval("lambda: 'Hi :)'", None, None).unwrap();
let hi: &pyo3::types::PyString = obj.call0().unwrap().downcast().unwrap();
assert_eq!(hi.len().unwrap(), 5);
}

After:

#![allow(unused)]
fn main() {
let gil = pyo3::Python::acquire_gil();
let obj = gil.python().eval("lambda: 'Hi :)'", None, None).unwrap();
let hi: &pyo3::types::PyString = obj.call0().unwrap().downcast().unwrap();
assert_eq!(hi.len().unwrap(), 5);
}

No #![feature(specialization)] in user code

While PyO3 itself still requires specialization and nightly Rust, now you don't have to use #![feature(specialization)] in your crate.

from 0.8.* to 0.9

#[new] interface

PyRawObject is now removed and our syntax for constructors has changed.

Before:

#![allow(unused)]
fn main() {
#[pyclass]
struct MyClass {}

#[pymethods]
impl MyClass {
   #[new]
   fn new(obj: &PyRawObject) {
       obj.init(MyClass { })
   }
}
}

After:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}

#[pymethods]
impl MyClass {
   #[new]
   fn new() -> Self {
       MyClass {}
   }
}
}

Basically you can return Self or Result<Self> directly. For more, see the constructor section of this guide.

PyCell

PyO3 0.9 introduces PyCell, which is a RefCell-like object wrapper for ensuring Rust's rules regarding aliasing of references are upheld. For more detail, see the Rust Book's section on Rust's rules of references

For #[pymethods] or #[pyfunction]s, your existing code should continue to work without any change. Python exceptions will automatically be raised when your functions are used in a way which breaks Rust's rules of references.

Here is an example.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct Names {
    names: Vec<String>
}

#[pymethods]
impl Names {
    #[new]
    fn new() -> Self {
        Names { names: vec![] }
    }
    fn merge(&mut self, other: &mut Names) {
        self.names.append(&mut other.names)
    }
}
let gil = Python::acquire_gil();
let py = gil.python();
let names = PyCell::new(py, Names::new()).unwrap();
let borrow_mut_err = py.get_type::<pyo3::pycell::PyBorrowMutError>();
pyo3::py_run!(py, names borrow_mut_err, r"
try:
   names.merge(names)
   assert False, 'Unreachable'
except RuntimeError as e:
   isinstance(e, borrow_mut_err)
");
}

Names has a merge method, which takes &mut self and another argument of type &mut Self. Given this #[pyclass], calling names.merge(names) in Python raises a PyBorrowMutError exception, since it requires two mutable borrows of names.

However, for #[pyproto] and some functions, you need to manually fix the code.

Object creation

In 0.8 object creation was done with PyRef::new and PyRefMut::new. In 0.9 these have both been removed. To upgrade code, please use PyCell::new instead. If you need PyRef or PyRefMut, just call .borrow() or .borrow_mut() on the newly-created PyCell.

Before:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
let gil = Python::acquire_gil();
let py = gil.python();
let obj_ref = PyRef::new(py, MyClass {}).unwrap();
}

After:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
let gil = Python::acquire_gil();
let py = gil.python();
let obj = PyCell::new(py, MyClass {}).unwrap();
let obj_ref = obj.borrow();
}

Object extraction

For PyClass types T, &T and &mut T no longer have FromPyObject implementations. Instead you should extract PyRef<T> or PyRefMut<T>, respectively. If T implements Clone, you can extract T itself. In addition, you can also extract &PyCell<T>, though you rarely need it.

Before:

let obj: &PyAny = create_obj();
let obj_ref: &MyClass = obj.extract().unwrap();
let obj_ref_mut: &mut MyClass = obj.extract().unwrap();

After:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
#[pyclass] #[derive(Clone)] struct MyClass {}
#[pymethods] impl MyClass { #[new]fn new() -> Self { MyClass {} }}
let gil = Python::acquire_gil();
let py = gil.python();
let typeobj = py.get_type::<MyClass>();
let d = [("c", typeobj)].into_py_dict(py);
let create_obj = || py.eval("c()", None, Some(d)).unwrap();
let obj: &PyAny = create_obj();
let obj_cell: &PyCell<MyClass> = obj.extract().unwrap();
let obj_cloned: MyClass = obj.extract().unwrap(); // extracted by cloning the object
{
    let obj_ref: PyRef<MyClass> = obj.extract().unwrap();
    // we need to drop obj_ref before we can extract a PyRefMut due to Rust's rules of references
}
let obj_ref_mut: PyRefMut<MyClass> = obj.extract().unwrap();
}

#[pyproto]

Most of the arguments to methods in #[pyproto] impls require a FromPyObject implementation. So if your protocol methods take &T or &mut T (where T: PyClass), please use PyRef or PyRefMut instead.

Before:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::class::PySequenceProtocol;
#[pyclass]
struct ByteSequence {
    elements: Vec<u8>,
}
#[pyproto]
impl PySequenceProtocol for ByteSequence {
    fn __concat__(&self, other: &Self) -> PyResult<Self> {
        let mut elements = self.elements.clone();
        elements.extend_from_slice(&other.elements);
        Ok(Self { elements })
    }
}
}

After:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::class::PySequenceProtocol;
#[pyclass]
struct ByteSequence {
    elements: Vec<u8>,
}
#[pyproto]
impl PySequenceProtocol for ByteSequence {
    fn __concat__(&self, other: PyRef<'p, Self>) -> PyResult<Self> {
        let mut elements = self.elements.clone();
        elements.extend_from_slice(&other.elements);
        Ok(Self { elements })
    }
}
}

Using in Python a Rust function with trait bounds

PyO3 allows for easy conversion from Rust to Python for certain functions and classes (see the conversion table). However, it is not always straightforward to convert Rust code that requires a given trait implementation as an argument.

This tutorial explains how to convert a Rust function that takes a trait as argument for use in Python with classes implementing the same methods as the trait.

Why is this useful?

Pros

• Make your Rust code available to Python users
• Code complex algorithms in Rust with the help of the borrow checker

Cons

• Not as fast as native Rust (type conversion has to be performed and one part of the code runs in Python)
• You need to adapt your code to expose it

Example

Let's work with the following basic example of an implementation of a optimization solver operating on a given model.

Let's say we have a function solve that operates on a model and mutates its state. The argument of the function can be any model that implements the Model trait :

#![allow(unused)]
fn main() {
pub trait Model {
  fn set_variables(&mut self, inputs: &Vec<f64>);
  fn compute(&mut self);
  fn get_results(&self) -> Vec<f64>;
}

pub fn solve<T: Model>(model: &mut T) {
  println!("Magic solver that mutates the model into a resolved state");
}
}

Let's assume we have the following constraints:

• We cannot change that code as it runs on many Rust models.
• We also have many Python models that cannot be solved as this solver is not available in that language. Rewriting it in Python would be cumbersome and error-prone, as everything is already available in Rust.

How could we expose this solver to Python thanks to PyO3 ?

Implementation of the trait bounds for the Python class

If a Python class implements the same three methods as the Model trait, it seems logical it could be adapted to use the solver. However, it is not possible to pass a PyObject to it as it does not implement the Rust trait (even if the Python model has the required methods).

In order to implement the trait, we must write a wrapper around the calls in Rust to the Python model. The method signatures must be the same as the trait, keeping in mind that the Rust trait cannot be changed for the purpose of making the code available in Python.

The Python model we want to expose is the following one, which already contains all the required methods:

class Model:
    def set_variables(self, inputs):
        self.inputs = inputs
    def compute(self):
        self.results = [elt**2 - 3 for elt in self.inputs]
    def get_results(self):
        return self.results

The following wrapper will call the Python model from Rust, using a struct to hold the model as a PyAny object:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyAny;

pub trait Model {
  fn set_variables(&mut self, inputs: &Vec<f64>);
  fn compute(&mut self);
  fn get_results(&self) -> Vec<f64>;
}

struct UserModel {
    model: Py<PyAny>,
}

impl Model for UserModel {
    fn set_variables(&mut self, var: &Vec<f64>) {
        println!("Rust calling Python to set the variables");
        let gil = Python::acquire_gil();
        let py = gil.python();
        let values: Vec<f64> = var.clone();
        let list: PyObject = values.into_py(py);
        let py_model = self.model.as_ref(py);
        py_model
            .call_method("set_variables", (list,), None)
            .unwrap();
    }

    fn get_results(&self) -> Vec<f64> {
        println!("Rust calling Python to get the results");
        let gil = Python::acquire_gil();
        let py = gil.python();
        self
            .model
            .as_ref(py)
            .call_method("get_results", (), None)
            .unwrap()
            .extract()
            .unwrap()
    }

    fn compute(&mut self) {
        println!("Rust calling Python to perform the computation");
        let gil = Python::acquire_gil();
        let py = gil.python();
        self.model
            .as_ref(py)
            .call_method("compute", (), None)
            .unwrap();
    }
}
}

Now that this bit is implemented, let's expose the model wrapper to Python. Let's add the PyO3 annotations and add a constructor:

#![allow(unused)]
fn main() {
pub trait Model {
  fn set_variables(&mut self, inputs: &Vec<f64>);
  fn compute(&mut self);
  fn get_results(&self) -> Vec<f64>;
}
use pyo3::prelude::*;
use pyo3::types::PyAny;

#[pyclass]
struct UserModel {
    model: Py<PyAny>,
}

#[pymodule]
fn trait_exposure(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<UserModel>()?;
    Ok(())
}

#[pymethods]
impl UserModel {
    #[new]
    pub fn new(model: Py<PyAny>) -> Self {
        UserModel { model }
    }
}
}

Now we add the PyO3 annotations to the trait implementation:

#[pymethods]
impl Model for UserModel {
  // the previous trait implementation
}

However, the previous code will not compile. The compilation error is the following one: error: #[pymethods] cannot be used on trait impl blocks

That's a bummer! However, we can write a second wrapper around these functions to call them directly. This wrapper will also perform the type conversions between Python and Rust.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyAny;

pub trait Model {
  fn set_variables(&mut self, inputs: &Vec<f64>);
  fn compute(&mut self);
  fn get_results(&self) -> Vec<f64>;
}

#[pyclass]
struct UserModel {
    model: Py<PyAny>,
}

impl Model for UserModel {
 fn set_variables(&mut self, var: &Vec<f64>) {
     println!("Rust calling Python to set the variables");
     let gil = Python::acquire_gil();
     let py = gil.python();
     let values: Vec<f64> = var.clone();
     let list: PyObject = values.into_py(py);
     let py_model = self.model.as_ref(py);
     py_model
         .call_method("set_variables", (list,), None)
         .unwrap();
 }

 fn get_results(&self) -> Vec<f64> {
     println!("Rust calling Python to get the results");
     let gil = Python::acquire_gil();
     let py = gil.python();
     self
         .model
         .as_ref(py)
         .call_method("get_results", (), None)
         .unwrap()
         .extract()
         .unwrap()
 }

 fn compute(&mut self) {
     println!("Rust calling Python to perform the computation");
     let gil = Python::acquire_gil();
     let py = gil.python();
     self.model
         .as_ref(py)
         .call_method("compute", (), None)
         .unwrap();
 }
}

#[pymethods]
impl UserModel {
    pub fn set_variables(&mut self, var: Vec<f64>) -> PyResult<()> {
        println!("Set variables from Python calling Rust");
        Model::set_variables(self, &var);
        Ok(())
    }

    pub fn get_results(&mut self) -> PyResult<Vec<f64>> {
        println!("Get results from Python calling Rust");
        let results = Model::get_results(self);
        let gil = Python::acquire_gil();
        let py = gil.python();
        let py_results = results.into_py(py);
        Ok(py_results)
    }

    pub fn compute(&mut self) -> PyResult<()> {
        println!("Compute from Python calling Rust");
        Model::compute(self);
        Ok(())
    }
}
}

This wrapper handles the type conversion between the PyO3 requirements and the trait. In order to meet PyO3 requirements, this wrapper must:

• return an object of type PyResult
• use only values, not references in the method signatures

Let's run the file python file:

class Model:
    def set_variables(self, inputs):
        self.inputs = inputs
    def compute(self):
        self.results = [elt**2 - 3 for elt in self.inputs]
    def get_results(self):
        return self.results

if __name__=="__main__":
  import trait_exposure

  myModel = Model()
  my_rust_model = trait_exposure.UserModel(myModel)
  my_rust_model.set_variables([2.0])
  print("Print value from Python: ", myModel.inputs)
  my_rust_model.compute()
  print("Print value from Python through Rust: ", my_rust_model.get_results())
  print("Print value directly from Python: ", myModel.get_results())

This outputs:

Set variables from Python calling Rust
Set variables from Rust calling Python
Print value from Python:  [2.0]
Compute from Python calling Rust
Compute from Rust calling Python
Get results from Python calling Rust
Get results from Rust calling Python
Print value from Python through Rust:  [1.0]
Print value directly from Python:  [1.0]

We have now successfully exposed a Rust model that implements the Model trait to Python!

We will now expose the solve function, but before, let's talk about types errors.

Type errors in Python

What happens if you have type errors when using Python and how can you improve the error messages?

Wrong types in Python function arguments

Let's assume in the first case that you will use in your Python file my_rust_model.set_variables(2.0) instead of my_rust_model.set_variables([2.0]).

The Rust signature expects a vector, which corresponds to a list in Python. What happens if instead of a vector, we pass a single value ?

At the execution of Python, we get :

File "main.py", line 15, in <module>
   my_rust_model.set_variables(2)
TypeError

It is a type error and Python points to it, so it's easy to identify and solve.

Wrong types in Python method signatures

Let's assume now that the return type of one of the methods of our Model class is wrong, for example the get_results method that is expected to return a Vec<f64> in Rust, a list in Python.

class Model:
    def set_variables(self, inputs):
        self.inputs = inputs
    def compute(self):
        self.results = [elt**2 -3 for elt in self.inputs]
    def get_results(self):
        return self.results[0]
        #return self.results <-- this is the expected output

This call results in the following panic:

pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: PyErr { type: Py(0x10dcf79f0, PhantomData) }

This error code is not helpful for a Python user that does not know anything about Rust, or someone that does not know PyO3 was used to interface the Rust code.

However, as we are responsible for making the Rust code available to Python, we can do something about it.

The issue is that we called unwrap anywhere we could, and therefore any panic from PyO3 will be directly forwarded to the end user.

Let's modify the code performing the type conversion to give a helpful error message to the Python user:

We used in our get_results method the following call that performs the type conversion:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyAny;

pub trait Model {
  fn set_variables(&mut self, inputs: &Vec<f64>);
  fn compute(&mut self);
  fn get_results(&self) -> Vec<f64>;
}

#[pyclass]
struct UserModel {
    model: Py<PyAny>,
}

impl Model for UserModel {
  fn get_results(&self) -> Vec<f64> {
    println!("Get results from Rust calling Python");
    let gil = Python::acquire_gil();
    let py = gil.python();
    self
        .model
        .as_ref(py)
        .call_method("get_results", (), None)
        .unwrap()
        .extract()
        .unwrap()
    }
 fn set_variables(&mut self, var: &Vec<f64>) {
     println!("Rust calling Python to set the variables");
     let gil = Python::acquire_gil();
     let py = gil.python();
     let values: Vec<f64> = var.clone();
     let list: PyObject = values.into_py(py);
     let py_model = self.model.as_ref(py);
     py_model
         .call_method("set_variables", (list,), None)
         .unwrap();
 }

 fn compute(&mut self) {
     println!("Rust calling Python to perform the computation");
     let gil = Python::acquire_gil();
     let py = gil.python();
     self.model
         .as_ref(py)
         .call_method("compute", (), None)
         .unwrap();
 }
}
}

Let's break it down in order to perform better error handling:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyAny;

pub trait Model {
  fn set_variables(&mut self, inputs: &Vec<f64>);
  fn compute(&mut self);
  fn get_results(&self) -> Vec<f64>;
}

#[pyclass]
struct UserModel {
    model: Py<PyAny>,
}

impl Model for UserModel {
  fn get_results(&self) -> Vec<f64> {
      println!("Get results from Rust calling Python");
      let gil = Python::acquire_gil();
      let py = gil.python();
      let py_result: &PyAny = self
          .model
          .as_ref(py)
          .call_method("get_results", (), None)
          .unwrap();

      if py_result.get_type().name() != "list" {
          panic!("Expected a list for the get_results() method signature, got {}", py_result.get_type().name());
      }
      py_result.extract().unwrap()
  }
 fn set_variables(&mut self, var: &Vec<f64>) {
     println!("Rust calling Python to set the variables");
     let gil = Python::acquire_gil();
     let py = gil.python();
     let values: Vec<f64> = var.clone();
     let list: PyObject = values.into_py(py);
     let py_model = self.model.as_ref(py);
     py_model
         .call_method("set_variables", (list,), None)
         .unwrap();
 }

 fn compute(&mut self) {
     println!("Rust calling Python to perform the computation");
     let gil = Python::acquire_gil();
     let py = gil.python();
     self.model
         .as_ref(py)
         .call_method("compute", (), None)
         .unwrap();
 }
}
}

By doing so, you catch the result of the Python computation and check its type in order to be able to deliver a better error message before performing the unwrapping.

Of course, it does not cover all the possible wrong outputs: the user could return a list of strings instead of a list of floats. In this case, a runtime panic would still occur due to PyO3, but with an error message much more difficult to decipher for non-rust user.

It is up to the developer exposing the rust code to decide how much effort to invest into Python type error handling and improved error messages.

The final code

Now let's expose the solve() function to make it available from Python.

It is not possible to directly expose the solve function to Python, as the type conversion cannot be performed. It requires an object implementing the Model trait as input.

However, the UserModel already implements this trait. Because of this, we can write a function wrapper that takes the UserModel--which has already been exposed to Python--as an argument in order to call the core function solve.

It is also required to make the struct public.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::wrap_pyfunction;
use pyo3::types::PyAny;

pub trait Model {
    fn set_variables(&mut self, var: &Vec<f64>);
    fn get_results(&self) -> Vec<f64>;
    fn compute(&mut self);
}

pub fn solve<T: Model>(model: &mut T) {
  println!("Magic solver that mutates the model into a resolved state");
}

#[pyfunction]
#[name = "solve"]
pub fn solve_wrapper(model: &mut UserModel) {
    solve(model);
}

#[pyclass]
pub struct UserModel {
    model: Py<PyAny>,
}

#[pymodule]
fn trait_exposure(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<UserModel>()?;
    m.add_wrapped(wrap_pyfunction!(solve_wrapper)).unwrap();
    Ok(())
}

#[pymethods]
impl UserModel {
    #[new]
    pub fn new(model: Py<PyAny>) -> Self {
        UserModel { model }
    }

    pub fn set_variables(&mut self, var: Vec<f64>) -> PyResult<()> {
        println!("Set variables from Python calling Rust");
        Model::set_variables(self, &var);
        Ok(())
    }

    pub fn get_results(&mut self) -> PyResult<Vec<f64>> {
        println!("Get results from Python calling Rust");
        let results = Model::get_results(self);
        let gil = Python::acquire_gil();
        let py = gil.python();
        let py_results = results.into_py(py);
        Ok(py_results)
    }

    pub fn compute(&mut self) -> PyResult<()> {
        Model::compute(self);
        Ok(())
    }
}

impl Model for UserModel {
    fn set_variables(&mut self, var: &Vec<f64>) {
        println!("Set variables from Rust calling Python");
        let gil = Python::acquire_gil();
        let py = gil.python();
        let values: Vec<f64> = var.clone();
        let list: PyObject = values.into_py(py);
        let py_model = self.model.as_ref(py);
        py_model
            .call_method("set_variables", (list,), None)
            .unwrap();
    }

    fn get_results(&self) -> Vec<f64> {
        println!("Get results from Rust calling Python");
        let gil = Python::acquire_gil();
        let py = gil.python();
        let py_result: &PyAny = self
            .model
            .as_ref(py)
            .call_method("get_results", (), None)
            .unwrap();

        if py_result.get_type().name() != "list" {
            panic!("Expected a list for the get_results() method signature, got {}", py_result.get_type().name());
        }
        py_result.extract().unwrap()
    }

    fn compute(&mut self) {
        println!("Compute from Rust calling Python");
        let gil = Python::acquire_gil();
        let py = gil.python();
        self.model
            .as_ref(py)
            .call_method("compute", (), None)
            .unwrap();
    }
}
}