The PyO3 user guide
Welcome to the PyO3 user guide! This book is a companion to PyO3's API docs. It contains examples and documentation to explain all of PyO3's use cases in detail.
The rough order of material in this user guide is as follows:
- Getting started
- Wrapping Rust code for use from Python
- How to use Python code from Rust
- Remaining topics which go into advanced concepts in detail
Please choose from the chapters on the left to jump to individual topics, or continue below to start with PyO3's README.
⚠️ Warning: API update in progress 🛠️
PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer Bound<T>
.
While most of this guide has been updated to the new API, it is possible some stray references to the older "GIL Refs" API such as &PyAny
remain.
PyO3
Rust bindings for Python, including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.
Usage
PyO3 supports the following software versions:
- Python 3.7 and up (CPython, PyPy, and GraalPy)
- Rust 1.56 and up
You can use PyO3 to write a native Python module in Rust, or to embed Python in a Rust binary. The following sections explain each of these in turn.
Using Rust from Python
PyO3 can be used to generate a native Python module. The easiest way to try this out for the first time is to use maturin
. maturin
is a tool for building and publishing Rust-based Python packages with minimal configuration. The following steps install maturin
, use it to generate and build a new Python package, and then launch Python to import and execute a function from the package.
First, follow the commands below to create a new directory containing a new Python virtualenv
, and install maturin
into the virtualenv using Python's package manager, pip
:
# (replace string_sum with the desired package name)
$ mkdir string_sum
$ cd string_sum
$ python -m venv .env
$ source .env/bin/activate
$ pip install maturin
Still inside this string_sum
directory, now run maturin init
. This will generate the new package source. When given the choice of bindings to use, select pyo3 bindings:
$ maturin init
✔ 🤷 What kind of bindings to use? · pyo3
✨ Done! New project created string_sum
The most important files generated by this command are Cargo.toml
and lib.rs
, which will look roughly like the following:
Cargo.toml
[package]
name = "string_sum"
version = "0.1.0"
edition = "2021"
[lib]
# The name of the native library. This is the name which will be used in Python to import the
# library (i.e. `import string_sum`). If you change this, you must also change the name of the
# `#[pymodule]` in `src/lib.rs`.
name = "string_sum"
# "cdylib" is necessary to produce a shared library for Python to import from.
#
# Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
# to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
# crate-type = ["cdylib", "rlib"]
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.21.1", features = ["extension-module"] }
src/lib.rs
use pyo3::prelude::*;
/// Formats the sum of two numbers as string.
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
Ok((a + b).to_string())
}
/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn string_sum(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
Ok(())
}
Finally, run maturin develop
. This will build the package and install it into the Python virtualenv previously created and activated. The package is then ready to be used from python
:
$ maturin develop
# lots of progress output as maturin runs the compilation...
$ python
>>> import string_sum
>>> string_sum.sum_as_string(5, 20)
'25'
To make changes to the package, just edit the Rust source code and then re-run maturin develop
to recompile.
To run this all as a single copy-and-paste, use the bash script below (replace string_sum
in the first command with the desired package name):
mkdir string_sum && cd "$_"
python -m venv .env
source .env/bin/activate
pip install maturin
maturin init --bindings pyo3
maturin develop
If you want to be able to run cargo test
or use this project in a Cargo workspace and are running into linker issues, there are some workarounds in the FAQ.
As well as with maturin
, it is possible to build using setuptools-rust
or manually. Both offer more flexibility than maturin
but require more configuration to get started.
Using Python from Rust
To embed Python into a Rust binary, you need to ensure that your Python installation contains a shared library. The following steps demonstrate how to ensure this (for Ubuntu), and then give some example code which runs an embedded Python interpreter.
To install the Python shared library on Ubuntu:
sudo apt install python3-dev
To install the Python shared library on RPM based distributions (e.g. Fedora, Red Hat, SuSE), install the python3-devel
package.
Start a new project with cargo new
and add pyo3
to the Cargo.toml
like this:
[dependencies.pyo3]
version = "0.21.1"
features = ["auto-initialize"]
Example program displaying the value of sys.version
and the current user name:
use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let sys = py.import_bound("sys")?;
let version: String = sys.getattr("version")?.extract()?;
let locals = [("os", py.import_bound("os")?)].into_py_dict_bound(py);
let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
let user: String = py.eval_bound(code, None, Some(&locals))?.extract()?;
println!("Hello {}, I'm Python {}", user, version);
Ok(())
})
}
The guide has a section with lots of examples about this topic.
Tools and libraries
- maturin Build and publish crates with pyo3, rust-cpython or cffi bindings as well as rust binaries as python packages
- setuptools-rust Setuptools plugin for Rust support.
- pyo3-built Simple macro to expose metadata obtained with the
built
crate as aPyDict
- rust-numpy Rust binding of NumPy C-API
- dict-derive Derive FromPyObject to automatically transform Python dicts into Rust structs
- pyo3-log Bridge from Rust to Python logging
- pythonize Serde serializer for converting Rust objects to JSON-compatible Python objects
- pyo3-asyncio Utilities for working with Python's Asyncio library and async functions
- rustimport Directly import Rust files or crates from Python, without manual compilation step. Provides pyo3 integration by default and generates pyo3 binding code automatically.
Examples
- autopy A simple, cross-platform GUI automation library for Python and Rust.
- Contains an example of building wheels on TravisCI and appveyor using cibuildwheel
- ballista-python A Python library that binds to Apache Arrow distributed query engine Ballista.
- bed-reader Read and write the PLINK BED format, simply and efficiently.
- Shows Rayon/ndarray::parallel (including capturing errors, controlling thread num), Python types to Rust generics, Github Actions
- cryptography Python cryptography library with some functionality in Rust.
- css-inline CSS inlining for Python implemented in Rust.
- datafusion-python A Python library that binds to Apache Arrow in-memory query engine DataFusion.
- deltalake-python Native Delta Lake Python binding based on delta-rs with Pandas integration.
- fastbloom A fast bloom filter | counting bloom filter implemented by Rust for Rust and Python!
- fastuuid Python bindings to Rust's UUID library.
- feos Lightning fast thermodynamic modeling in Rust with fully developed Python interface.
- forust A lightweight gradient boosted decision tree library written in Rust.
- greptimedb Support Python scripting in the database
- haem A Python library for working on Bioinformatics problems.
- html-py-ever Using html5ever through kuchiki to speed up html parsing and css-selecting.
- hyperjson A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.
- inline-python Inline Python code directly in your Rust code.
- johnnycanencrypt OpenPGP library with Yubikey support.
- jsonschema-rs Fast JSON Schema validation library.
- mocpy Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere.
- opendal A data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.
- orjson Fast Python JSON library.
- ormsgpack Fast Python msgpack library.
- point-process High level API for pointprocesses as a Python library.
- polaroid Hyper Fast and safe image manipulation library for Python written in Rust.
- polars Fast multi-threaded DataFrame library in Rust | Python | Node.js.
- pydantic-core Core validation logic for pydantic written in Rust.
- pyheck Fast case conversion library, built by wrapping heck.
- Quite easy to follow as there's not much code.
- pyre Fast Python HTTP server written in Rust.
- ril-py A performant and high-level image processing library for Python written in Rust.
- river Online machine learning in python, the computationally heavy statistics algorithms are implemented in Rust.
- rust-python-coverage Example PyO3 project with automated test coverage for Rust and Python.
- tiktoken A fast BPE tokeniser for use with OpenAI's models.
- tokenizers Python bindings to the Hugging Face tokenizers (NLP) written in Rust.
- tzfpy A fast package to convert longitude/latitude to timezone name.
- utiles Fast Python web-map tile utilities
- wasmer-python Python library to run WebAssembly binaries.
Articles and other media
- (Video) Extending Python with Rust using PyO3 - Dec 16, 2023
- A Week of PyO3 + rust-numpy (How to Speed Up Your Data Pipeline X Times) - Jun 6, 2023
- (Podcast) PyO3 with David Hewitt - May 19, 2023
- Making Python 100x faster with less than 100 lines of Rust - Mar 28, 2023
- How Pydantic V2 leverages Rust's Superpowers - Feb 4, 2023
- How we extended the River stats module with Rust using PyO3 - Dec 23, 2022
- Nine Rules for Writing Python Extensions in Rust - Dec 31, 2021
- Calling Rust from Python using PyO3 - Nov 18, 2021
- davidhewitt's 2021 talk at Rust Manchester meetup - Aug 19, 2021
- Incrementally porting a small Python project to Rust - Apr 29, 2021
- Vortexa - Integrating Rust into Python - Apr 12, 2021
- Writing and publishing a Python module in Rust - Aug 2, 2020
Contributing
Everyone is welcomed to contribute to PyO3! There are many ways to support the project, such as:
- help PyO3 users with issues on GitHub and Discord
- improve documentation
- write features and bugfixes
- publish blogs and examples of how to use PyO3
Our contributing notes and architecture guide have more resources if you wish to volunteer time for PyO3 and are searching where to start.
If you don't have time to contribute yourself but still wish to support the project's future success, some of our maintainers have GitHub sponsorship pages:
License
PyO3 is licensed under the Apache-2.0 license or the MIT license, at your option.
Python is licensed under the Python License.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in PyO3 by you, as defined in the Apache License, shall be dual-licensed as above, without any additional terms or conditions.
Installation
To get started using PyO3 you will need three things: a Rust toolchain, a Python environment, and a way to build. We'll cover each of these below.
If you'd like to chat to the PyO3 maintainers and other PyO3 users, consider joining the PyO3 Discord server. We're keen to hear about your experience getting started, so we can make PyO3 as accessible as possible for everyone!
Rust
First, make sure you have Rust installed on your system. If you haven't already done so, try following the instructions here. PyO3 runs on both the stable
and nightly
versions so you can choose whichever one fits you best. The minimum required Rust version is 1.56.
If you can run rustc --version
and the version is new enough you're good to go!
Python
To use PyO3, you need at least Python 3.7. While you can simply use the default Python interpreter on your system, it is recommended to use a virtual environment.
Virtualenvs
While you can use any virtualenv manager you like, we recommend the use of pyenv
in particular if you want to develop or test for multiple different Python versions, so that is what the examples in this book will use. The installation instructions for pyenv
can be found here. (Note: To get the pyenv activate
and pyenv virtualenv
commands, you will also need to install the pyenv-virtualenv
plugin. The pyenv installer will install both together.)
If you intend to run Python from Rust (for example in unit tests) you should set the following environment variable when installing a new Python version using pyenv
:
PYTHON_CONFIGURE_OPTS="--enable-shared"
For example:
env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.12
You can read more about pyenv
's configuration options here.
Building
There are a number of build and Python package management systems such as setuptools-rust
or manually. We recommend the use of maturin
, which you can install here. It is developed to work with PyO3 and provides the most "batteries included" experience, especially if you are aiming to publish to PyPI. maturin
is just a Python package, so you can add it in the same way you already install Python packages.
System Python:
pip install maturin --user
pipx:
pipx install maturin
pyenv:
pyenv activate pyo3
pip install maturin
poetry:
poetry add -G dev maturin
After installation, you can run maturin --version
to check that you have correctly installed it.
Starting a new project
First you should create the folder and virtual environment that are going to contain your new project. Here we will use the recommended pyenv
:
mkdir pyo3-example
cd pyo3-example
pyenv virtualenv pyo3
pyenv local pyo3
After this, you should install your build manager. In this example, we will use maturin
. After you've activated your virtualenv, add maturin
to it:
pip install maturin
Now you can initialize the new project:
maturin init
If maturin
is already installed, you can create a new project using that directly as well:
maturin new -b pyo3 pyo3-example
cd pyo3-example
pyenv virtualenv pyo3
pyenv local pyo3
Adding to an existing project
Sadly, maturin
cannot currently be run in existing projects, so if you want to use Python in an existing project you basically have two options:
- Create a new project as above and move your existing code into that project
- Manually edit your project configuration as necessary
If you opt for the second option, here are the things you need to pay attention to:
Cargo.toml
Make sure that the Rust crate you want to be able to access from Python is compiled into a library. You can have a binary output as well, but the code you want to access from Python has to be in the library part. Also, make sure that the crate type is cdylib
and add PyO3 as a dependency as so:
# If you already have [package] information in `Cargo.toml`, you can ignore
# this section!
[package]
# `name` here is name of the package.
name = "pyo3_start"
# these are good defaults:
version = "0.1.0"
edition = "2021"
[lib]
# The name of the native library. This is the name which will be used in Python to import the
# library (i.e. `import string_sum`). If you change this, you must also change the name of the
# `#[pymodule]` in `src/lib.rs`.
name = "pyo3_example"
# "cdylib" is necessary to produce a shared library for Python to import from.
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.21.1", features = ["extension-module"] }
pyproject.toml
You should also create a pyproject.toml
with the following contents:
[build-system]
requires = ["maturin>=1,<2"]
build-backend = "maturin"
[project]
name = "pyo3_example"
requires-python = ">=3.7"
classifiers = [
"Programming Language :: Rust",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
]
Running code
After this you can setup Rust code to be available in Python as below; for example, you can place this code in src/lib.rs
:
use pyo3::prelude::*;
/// Formats the sum of two numbers as string.
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
Ok((a + b).to_string())
}
/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn pyo3_example(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
Ok(())
}
Now you can run maturin develop
to prepare the Python package, after which you can use it like so:
$ maturin develop
# lots of progress output as maturin runs the compilation...
$ python
>>> import pyo3_example
>>> pyo3_example.sum_as_string(5, 20)
'25'
For more instructions on how to use Python code from Rust, see the Python from Rust page.
Using Rust from Python
This chapter of the guide is dedicated to explaining how to wrap Rust code into Python objects.
PyO3 uses Rust's "procedural macros" to provide a powerful yet simple API to denote what Rust code should map into Python objects.
The three types of Python objects which PyO3 can produce are:
- Python modules, via the
#[pymodule]
macro - Python functions, via the
#[pyfunction]
macro - Python classes, via the
#[pyclass]
macro (plus#[pymethods]
to define methods for those clases)
The following subchapters go through each of these in turn.
Python modules
You can create a module using #[pymodule]
:
use pyo3::prelude::*;
#[pyfunction]
fn double(x: usize) -> usize {
x * 2
}
/// This module is implemented in Rust.
#[pymodule]
fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(double, m)?)?;
Ok(())
}
The #[pymodule]
procedural macro takes care of exporting the initialization function of your
module to Python.
The module's name defaults to the name of the Rust function. You can override the module name by
using #[pyo3(name = "custom_name")]
:
use pyo3::prelude::*;
#[pyfunction]
fn double(x: usize) -> usize {
x * 2
}
#[pymodule]
#[pyo3(name = "custom_name")]
fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(double, m)?)?;
Ok(())
}
The name of the module must match the name of the .so
or .pyd
file. Otherwise, you will get an import error in Python with the following message:
ImportError: dynamic module does not define module export function (PyInit_name_of_your_module)
To import the module, either:
- copy the shared library as described in Manual builds, or
- use a tool, e.g.
maturin develop
with maturin orpython setup.py develop
with setuptools-rust.
Documentation
The Rust doc comments of the module initialization function will be applied automatically as the Python docstring of your module.
For example, building off of the above code, this will print This module is implemented in Rust.
:
import my_extension
print(my_extension.__doc__)
Python submodules
You can create a module hierarchy within a single extension module by using
Bound<'_, PyModule>::add_submodule()
.
For example, you could define the modules parent_module
and parent_module.child_module
.
use pyo3::prelude::*;
#[pymodule]
fn parent_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
register_child_module(m)?;
Ok(())
}
fn register_child_module(parent_module: &Bound<'_, PyModule>) -> PyResult<()> {
let child_module = PyModule::new_bound(parent_module.py(), "child_module")?;
child_module.add_function(wrap_pyfunction!(func, &child_module)?)?;
parent_module.add_submodule(&child_module)?;
Ok(())
}
#[pyfunction]
fn func() -> String {
"func".to_string()
}
Python::with_gil(|py| {
use pyo3::wrap_pymodule;
use pyo3::types::IntoPyDict;
let parent_module = wrap_pymodule!(parent_module)(py);
let ctx = [("parent_module", parent_module)].into_py_dict_bound(py);
py.run_bound("assert parent_module.child_module.func() == 'func'", None, Some(&ctx)).unwrap();
})
Note that this does not define a package, so this won’t allow Python code to directly import
submodules by using from parent_module import child_module
. For more information, see
#759 and
#1517.
It is not necessary to add #[pymodule]
on nested modules, which is only required on the top-level module.
Declarative modules (experimental)
Another syntax based on Rust inline modules is also available to declare modules.
The experimental-declarative-modules
feature must be enabled to use it.
For example:
#[cfg(feature = "experimental-declarative-modules")]
mod declarative_module_test {
use pyo3::prelude::*;
#[pyfunction]
fn double(x: usize) -> usize {
x * 2
}
#[pymodule]
mod my_extension {
use super::*;
#[pymodule_export]
use super::double; // Exports the double function as part of the module
#[pyfunction] // This will be part of the module
fn triple(x: usize) -> usize {
x * 3
}
#[pyclass] // This will be part of the module
struct Unit;
#[pymodule]
mod submodule {
// This is a submodule
}
#[pymodule_init]
fn init(m: &Bound<'_, PyModule>) -> PyResult<()> {
// Arbitrary code to run at the module initialization
m.add("double2", m.getattr("double")?)?;
Ok(())
}
}
}
Some changes are planned to this feature before stabilization, like automatically
filling submodules into sys.modules
to allow easier imports (see issue #759)
and filling the module
argument of inlined #[pyclass]
automatically with the proper module name.
Macro names might also change.
See issue #3900 to track this feature progress.
Python functions
The #[pyfunction]
attribute is used to define a Python function from a Rust function. Once defined, the function needs to be added to a module using the wrap_pyfunction!
macro.
The following example defines a function called double
in a Python module called my_extension
:
use pyo3::prelude::*;
#[pyfunction]
fn double(x: usize) -> usize {
x * 2
}
#[pymodule]
fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(double, m)?)?;
Ok(())
}
This chapter of the guide explains full usage of the #[pyfunction]
attribute. In this first section, the following topics are covered:
There are also additional sections on the following topics:
Function options
The #[pyo3]
attribute can be used to modify properties of the generated Python function. It can take any combination of the following options:
-
Overrides the name exposed to Python.
In the following example, the Rust function
no_args_py
will be added to the Python modulemodule_with_functions
as the Python functionno_args
:use pyo3::prelude::*; #[pyfunction] #[pyo3(name = "no_args")] fn no_args_py() -> usize { 42 } #[pymodule] fn module_with_functions(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(no_args_py, m)?)?; Ok(()) } Python::with_gil(|py| { let m = pyo3::wrap_pymodule!(module_with_functions)(py); assert!(m.getattr(py, "no_args").is_ok()); assert!(m.getattr(py, "no_args_py").is_err()); });
-
Defines the function signature in Python. See Function Signatures.
-
#[pyo3(text_signature = "...")]
Overrides the PyO3-generated function signature visible in Python tooling (such as via
inspect.signature
). See the corresponding topic in the Function Signatures subchapter. -
Set this option to make PyO3 pass the containing module as the first argument to the function. It is then possible to use the module in the function body. The first argument must be of type
&Bound<'_, PyModule>
,Bound<'_, PyModule>
, orPy<PyModule>
.The following example creates a function
pyfunction_with_module
which returns the containing module's name (i.e.module_with_fn
):use pyo3::prelude::*; use pyo3::types::PyString; #[pyfunction] #[pyo3(pass_module)] fn pyfunction_with_module<'py>( module: &Bound<'py, PyModule>, ) -> PyResult<Bound<'py, PyString>> { module.name() } #[pymodule] fn module_with_fn(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(pyfunction_with_module, m)?) }
Per-argument options
The #[pyo3]
attribute can be used on individual arguments to modify properties of them in the generated function. It can take any combination of the following options:
-
Set this on an option to specify a custom function to convert the function argument from Python to the desired Rust type, instead of using the default
FromPyObject
extraction. The function signature must befn(&Bound<'_, PyAny>) -> PyResult<T>
whereT
is the Rust type of the argument.The following example uses
from_py_with
to convert the input Python object to its length:use pyo3::prelude::*; fn get_length(obj: &Bound<'_, PyAny>) -> PyResult<usize> { let length = obj.len()?; Ok(length) } #[pyfunction] fn object_length(#[pyo3(from_py_with = "get_length")] argument: usize) -> usize { argument } Python::with_gil(|py| { let f = pyo3::wrap_pyfunction_bound!(object_length)(py).unwrap(); assert_eq!(f.call1((vec![1, 2, 3],)).unwrap().extract::<usize>().unwrap(), 3); });
Advanced function patterns
Calling Python functions in Rust
You can pass Python def
'd functions and built-in functions to Rust functions PyFunction
corresponds to regular Python functions while PyCFunction
describes built-ins such as
repr()
.
You can also use Bound<'_, PyAny>::is_callable
to check if you have a callable object. is_callable
will return true
for functions (including lambdas), methods and objects with a __call__
method.
You can call the object with Bound<'_, PyAny>::call
with the args as first parameter and the kwargs
(or None
) as second parameter. There are also Bound<'_, PyAny>::call0
with no args and
Bound<'_, PyAny>::call1
with only positional args.
Calling Rust functions in Python
The ways to convert a Rust function into a Python object vary depending on the function:
- Named functions, e.g.
fn foo()
: add#[pyfunction]
and then usewrap_pyfunction!
to get the correspondingPyCFunction
. - Anonymous functions (or closures), e.g.
foo: fn()
either:- use a
#[pyclass]
struct which stores the function as a field and implement__call__
to call the stored function. - use
PyCFunction::new_closure
to create an object directly from the function.
- use a
Accessing the FFI functions
In order to make Rust functions callable from Python, PyO3 generates an extern "C"
function whose exact signature depends on the Rust signature. (PyO3 chooses the optimal
Python argument passing convention.) It then embeds the call to the Rust function inside this
FFI-wrapper function. This wrapper handles extraction of the regular arguments and the keyword
arguments from the input PyObject
s.
The wrap_pyfunction
macro can be used to directly get a Bound<PyCFunction>
given a
#[pyfunction]
and a Bound<PyModule>
: wrap_pyfunction!(rust_fun, module)
.
#[pyfn]
shorthand
There is a shorthand to #[pyfunction]
and wrap_pymodule!
: the function can be placed inside the module definition and
annotated with #[pyfn]
. To simplify PyO3, it is expected that #[pyfn]
may be removed in a future release (See #694).
An example of #[pyfn]
is below:
use pyo3::prelude::*;
#[pymodule]
fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> {
#[pyfn(m)]
fn double(x: usize) -> usize {
x * 2
}
Ok(())
}
#[pyfn(m)]
is just syntactic sugar for #[pyfunction]
, and takes all the same options
documented in the rest of this chapter. The code above is expanded to the following:
use pyo3::prelude::*;
#[pymodule]
fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> {
#[pyfunction]
fn double(x: usize) -> usize {
x * 2
}
m.add_function(wrap_pyfunction!(double, m)?)?;
Ok(())
}
Function signatures
The #[pyfunction]
attribute also accepts parameters to control how the generated Python function accepts arguments. Just like in Python, arguments can be positional-only, keyword-only, or accept either. *args
lists and **kwargs
dicts can also be accepted. These parameters also work for #[pymethods]
which will be introduced in the Python Classes section of the guide.
Like Python, by default PyO3 accepts all arguments as either positional or keyword arguments. Most arguments are required by default, except for trailing Option<_>
arguments, which are implicitly given a default of None
. This behaviour can be configured by the #[pyo3(signature = (...))]
option which allows writing a signature in Python syntax.
This section of the guide goes into detail about use of the #[pyo3(signature = (...))]
option and its related option #[pyo3(text_signature = "...")]
Using #[pyo3(signature = (...))]
For example, below is a function that accepts arbitrary keyword arguments (**kwargs
in Python syntax) and returns the number that was passed:
use pyo3::prelude::*;
use pyo3::types::PyDict;
#[pyfunction]
#[pyo3(signature = (**kwds))]
fn num_kwds(kwds: Option<&Bound<'_, PyDict>>) -> usize {
kwds.map_or(0, |dict| dict.len())
}
#[pymodule]
fn module_with_functions(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(num_kwds, m)?).unwrap();
Ok(())
}
Just like in Python, the following constructs can be part of the signature::
/
: positional-only arguments separator, each parameter defined before/
is a positional-only parameter.*
: var arguments separator, each parameter defined after*
is a keyword-only parameter.*args
: "args" is var args. Type of theargs
parameter has to be&Bound<'_, PyTuple>
.**kwargs
: "kwargs" receives keyword arguments. The type of thekwargs
parameter has to beOption<&Bound<'_, PyDict>>
.arg=Value
: arguments with default value. If thearg
argument is defined after var arguments, it is treated as a keyword-only argument. Note thatValue
has to be valid rust code, PyO3 just inserts it into the generated code unmodified.
Example:
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyTuple};
#[pyclass]
struct MyClass {
num: i32,
}
#[pymethods]
impl MyClass {
#[new]
#[pyo3(signature = (num=-1))]
fn new(num: i32) -> Self {
MyClass { num }
}
#[pyo3(signature = (num=10, *py_args, name="Hello", **py_kwargs))]
fn method(
&mut self,
num: i32,
py_args: &Bound<'_, PyTuple>,
name: &str,
py_kwargs: Option<&Bound<'_, PyDict>>,
) -> String {
let num_before = self.num;
self.num = num;
format!(
"num={} (was previously={}), py_args={:?}, name={}, py_kwargs={:?} ",
num, num_before, py_args, name, py_kwargs,
)
}
fn make_change(&mut self, num: i32) -> PyResult<String> {
self.num = num;
Ok(format!("num={}", self.num))
}
}
Arguments of type Python
must not be part of the signature:
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyfunction]
#[pyo3(signature = (lambda))]
pub fn simple_python_bound_function(py: Python<'_>, lambda: PyObject) -> PyResult<()> {
Ok(())
}
N.B. the position of the /
and *
arguments (if included) control the system of handling positional and keyword arguments. In Python:
import mymodule
mc = mymodule.MyClass()
print(mc.method(44, False, "World", 666, x=44, y=55))
print(mc.method(num=-1, name="World"))
print(mc.make_change(44, False))
Produces output:
py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44
py_args=(), py_kwargs=None, name=World, num=-1
num=44
num=-1
Note: to use keywords like
struct
as a function argument, use "raw identifier" syntaxr#struct
in both the signature and the function definition:#![allow(dead_code)] use pyo3::prelude::*; #[pyfunction(signature = (r#struct = "foo"))] fn function_with_keyword(r#struct: &str) { let _ = r#struct; /* ... */ }
Trailing optional arguments
As a convenience, functions without a #[pyo3(signature = (...))]
option will treat trailing Option<T>
arguments as having a default of None
. In the example below, PyO3 will create increment
with a signature of increment(x, amount=None)
.
use pyo3::prelude::*;
/// Returns a copy of `x` increased by `amount`.
///
/// If `amount` is unspecified or `None`, equivalent to `x + 1`.
#[pyfunction]
fn increment(x: u64, amount: Option<u64>) -> u64 {
x + amount.unwrap_or(1)
}
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let fun = pyo3::wrap_pyfunction_bound!(increment, py)?;
let inspect = PyModule::import_bound(py, "inspect")?.getattr("signature")?;
let sig: String = inspect
.call1((fun,))?
.call_method0("__str__")?
.extract()?;
#[cfg(Py_3_8)] // on 3.7 the signature doesn't render b, upstream bug?
assert_eq!(sig, "(x, amount=None)");
Ok(())
})
}
To make trailing Option<T>
arguments required, but still accept None
, add a #[pyo3(signature = (...))]
annotation. For the example above, this would be #[pyo3(signature = (x, amount))]
:
use pyo3::prelude::*;
#[pyfunction]
#[pyo3(signature = (x, amount))]
fn increment(x: u64, amount: Option<u64>) -> u64 {
x + amount.unwrap_or(1)
}
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let fun = pyo3::wrap_pyfunction_bound!(increment, py)?;
let inspect = PyModule::import_bound(py, "inspect")?.getattr("signature")?;
let sig: String = inspect
.call1((fun,))?
.call_method0("__str__")?
.extract()?;
#[cfg(Py_3_8)] // on 3.7 the signature doesn't render b, upstream bug?
assert_eq!(sig, "(x, amount)");
Ok(())
})
}
To help avoid confusion, PyO3 requires #[pyo3(signature = (...))]
when an Option<T>
argument is surrounded by arguments which aren't Option<T>
.
Making the function signature available to Python
The function signature is exposed to Python via the __text_signature__
attribute. PyO3 automatically generates this for every #[pyfunction]
and all #[pymethods]
directly from the Rust function, taking into account any override done with the #[pyo3(signature = (...))]
option.
This automatic generation can only display the value of default arguments for strings, integers, boolean types, and None
. Any other default arguments will be displayed as ...
. (.pyi
type stub files commonly also use ...
for default arguments in the same way.)
In cases where the automatically-generated signature needs adjusting, it can be overridden using the #[pyo3(text_signature)]
option.)
The example below creates a function add
which accepts two positional-only arguments a
and b
, where b
has a default value of zero.
use pyo3::prelude::*;
/// This function adds two unsigned 64-bit integers.
#[pyfunction]
#[pyo3(signature = (a, b=0, /))]
fn add(a: u64, b: u64) -> u64 {
a + b
}
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let fun = pyo3::wrap_pyfunction_bound!(add, py)?;
let doc: String = fun.getattr("__doc__")?.extract()?;
assert_eq!(doc, "This function adds two unsigned 64-bit integers.");
let inspect = PyModule::import_bound(py, "inspect")?.getattr("signature")?;
let sig: String = inspect
.call1((fun,))?
.call_method0("__str__")?
.extract()?;
#[cfg(Py_3_8)] // on 3.7 the signature doesn't render b, upstream bug?
assert_eq!(sig, "(a, b=0, /)");
Ok(())
})
}
The following IPython output demonstrates how this generated signature will be seen from Python tooling:
>>> pyo3_test.add.__text_signature__
'(a, b=..., /)'
>>> pyo3_test.add?
Signature: pyo3_test.add(a, b=0, /)
Docstring: This function adds two unsigned 64-bit integers.
Type: builtin_function_or_method
Overriding the generated signature
The #[pyo3(text_signature = "(<some signature>)")]
attribute can be used to override the default generated signature.
In the snippet below, the text signature attribute is used to include the default value of 0
for the argument b
, instead of the automatically-generated default value of ...
:
use pyo3::prelude::*;
/// This function adds two unsigned 64-bit integers.
#[pyfunction]
#[pyo3(signature = (a, b=0, /), text_signature = "(a, b=0, /)")]
fn add(a: u64, b: u64) -> u64 {
a + b
}
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let fun = pyo3::wrap_pyfunction_bound!(add, py)?;
let doc: String = fun.getattr("__doc__")?.extract()?;
assert_eq!(doc, "This function adds two unsigned 64-bit integers.");
let inspect = PyModule::import_bound(py, "inspect")?.getattr("signature")?;
let sig: String = inspect
.call1((fun,))?
.call_method0("__str__")?
.extract()?;
assert_eq!(sig, "(a, b=0, /)");
Ok(())
})
}
PyO3 will include the contents of the annotation unmodified as the __text_signature__
. Below shows how IPython will now present this (see the default value of 0 for b):
>>> pyo3_test.add.__text_signature__
'(a, b=0, /)'
>>> pyo3_test.add?
Signature: pyo3_test.add(a, b=0, /)
Docstring: This function adds two unsigned 64-bit integers.
Type: builtin_function_or_method
If no signature is wanted at all, #[pyo3(text_signature = None)]
will disable the built-in signature. The snippet below demonstrates use of this:
use pyo3::prelude::*;
/// This function adds two unsigned 64-bit integers.
#[pyfunction]
#[pyo3(signature = (a, b=0, /), text_signature = None)]
fn add(a: u64, b: u64) -> u64 {
a + b
}
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let fun = pyo3::wrap_pyfunction_bound!(add, py)?;
let doc: String = fun.getattr("__doc__")?.extract()?;
assert_eq!(doc, "This function adds two unsigned 64-bit integers.");
assert!(fun.getattr("__text_signature__")?.is_none());
Ok(())
})
}
Now the function's __text_signature__
will be set to None
, and IPython will not display any signature in the help:
>>> pyo3_test.add.__text_signature__ == None
True
>>> pyo3_test.add?
Docstring: This function adds two unsigned 64-bit integers.
Type: builtin_function_or_method
Error handling
This chapter contains a little background of error handling in Rust and how PyO3 integrates this with Python exceptions.
This covers enough detail to create a #[pyfunction]
which raises Python exceptions from errors originating in Rust.
There is a later section of the guide on Python exceptions which covers exception types in more detail.
Representing Python exceptions
Rust code uses the generic Result<T, E>
enum to propagate errors. The error type E
is chosen by the code author to describe the possible errors which can happen.
PyO3 has the PyErr
type which represents a Python exception. If a PyO3 API could result in a Python exception being raised, the return type of that API
will be PyResult<T>
, which is an alias for the type Result<T, PyErr>
.
In summary:
- When Python exceptions are raised and caught by PyO3, the exception will be stored in the
Err
variant of thePyResult
. - Passing Python exceptions through Rust code then uses all the "normal" techniques such as the
?
operator, withPyErr
as the error type. - Finally, when a
PyResult
crosses from Rust back to Python via PyO3, if the result is anErr
variant the contained exception will be raised.
(There are many great tutorials on Rust error handling and the ?
operator, so this guide will not go into detail on Rust-specific topics.)
Raising an exception from a function
As indicated in the previous section, when a PyResult
containing an Err
crosses from Rust to Python, PyO3 will raise the exception contained within.
Accordingly, to raise an exception from a #[pyfunction]
, change the return type T
to PyResult<T>
. When the function returns an Err
it will raise a Python exception. (Other Result<T, E>
types can be used as long as the error E
has a From
conversion for PyErr
, see custom Rust error types below.)
This also works for functions in #[pymethods]
.
For example, the following check_positive
function raises a ValueError
when the input is negative:
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
#[pyfunction]
fn check_positive(x: i32) -> PyResult<()> {
if x < 0 {
Err(PyValueError::new_err("x is negative"))
} else {
Ok(())
}
}
fn main(){
Python::with_gil(|py|{
let fun = pyo3::wrap_pyfunction_bound!(check_positive, py).unwrap();
fun.call1((-1,)).unwrap_err();
fun.call1((1,)).unwrap();
});
}
All built-in Python exception types are defined in the pyo3::exceptions
module. They have a new_err
constructor to directly build a PyErr
, as seen in the example above.
Custom Rust error types
PyO3 will automatically convert a Result<T, E>
returned by a #[pyfunction]
into a PyResult<T>
as long as there is an implementation of std::from::From<E> for PyErr
. Many error types in the Rust standard library have a From
conversion defined in this way.
If the type E
you are handling is defined in a third-party crate, see the section on foreign rust error types below for ways to work with this error.
The following example makes use of the implementation of From<ParseIntError> for PyErr
to raise exceptions encountered when parsing strings as integers:
use pyo3::prelude::*;
use std::num::ParseIntError;
#[pyfunction]
fn parse_int(x: &str) -> Result<usize, ParseIntError> {
x.parse()
}
fn main() {
Python::with_gil(|py| {
let fun = pyo3::wrap_pyfunction_bound!(parse_int, py).unwrap();
let value: usize = fun.call1(("5",)).unwrap().extract().unwrap();
assert_eq!(value, 5);
});
}
When passed a string which doesn't contain a floating-point number, the exception raised will look like the below:
>>> parse_int("bar")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid digit found in string
As a more complete example, the following snippet defines a Rust error named CustomIOError
. It then defines a From<CustomIOError> for PyErr
, which returns a PyErr
representing Python's OSError
.
Therefore, it can use this error in the result of a #[pyfunction]
directly, relying on the conversion if it has to be propagated into a Python exception.
use pyo3::exceptions::PyOSError;
use pyo3::prelude::*;
use std::fmt;
#[derive(Debug)]
struct CustomIOError;
impl std::error::Error for CustomIOError {}
impl fmt::Display for CustomIOError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "Oh no!")
}
}
impl std::convert::From<CustomIOError> for PyErr {
fn from(err: CustomIOError) -> PyErr {
PyOSError::new_err(err.to_string())
}
}
pub struct Connection {/* ... */}
fn bind(addr: String) -> Result<Connection, CustomIOError> {
if &addr == "0.0.0.0" {
Err(CustomIOError)
} else {
Ok(Connection{ /* ... */})
}
}
#[pyfunction]
fn connect(s: String) -> Result<(), CustomIOError> {
bind(s)?;
// etc.
Ok(())
}
fn main() {
Python::with_gil(|py| {
let fun = pyo3::wrap_pyfunction_bound!(connect, py).unwrap();
let err = fun.call1(("0.0.0.0",)).unwrap_err();
assert!(err.is_instance_of::<PyOSError>(py));
});
}
If lazy construction of the Python exception instance is desired, the
PyErrArguments
trait can be implemented instead of From
. In that case, actual exception argument creation is delayed
until the PyErr
is needed.
A final note is that any errors E
which have a From
conversion can be used with the ?
("try") operator with them. An alternative implementation of the above parse_int
which instead returns PyResult
is below:
use pyo3::prelude::*;
fn parse_int(s: String) -> PyResult<usize> {
let x = s.parse()?;
Ok(x)
}
use pyo3::exceptions::PyValueError;
fn main() {
Python::with_gil(|py| {
assert_eq!(parse_int(String::from("1")).unwrap(), 1);
assert_eq!(parse_int(String::from("1337")).unwrap(), 1337);
assert!(parse_int(String::from("-1"))
.unwrap_err()
.is_instance_of::<PyValueError>(py));
assert!(parse_int(String::from("foo"))
.unwrap_err()
.is_instance_of::<PyValueError>(py));
assert!(parse_int(String::from("13.37"))
.unwrap_err()
.is_instance_of::<PyValueError>(py));
})
}
Foreign Rust error types
The Rust compiler will not permit implementation of traits for types outside of the crate where the type is defined. (This is known as the "orphan rule".)
Given a type OtherError
which is defined in third-party code, there are two main strategies available to integrate it with PyO3:
- Create a newtype wrapper, e.g.
MyOtherError
. Then implementFrom<MyOtherError> for PyErr
(orPyErrArguments
), as well asFrom<OtherError>
forMyOtherError
. - Use Rust's Result combinators such as
map_err
to write code freely to convertOtherError
into whatever is needed. This requires boilerplate at every usage however gives unlimited flexibility.
To detail the newtype strategy a little further, the key trick is to return Result<T, MyOtherError>
from the #[pyfunction]
. This means that PyO3 will make use of From<MyOtherError> for PyErr
to create Python exceptions while the #[pyfunction]
implementation can use ?
to convert OtherError
to MyOtherError
automatically.
The following example demonstrates this for some imaginary third-party crate some_crate
with a function get_x
returning Result<i32, OtherError>
:
mod some_crate {
pub struct OtherError(());
impl OtherError {
pub fn message(&self) -> &'static str { "some error occurred" }
}
pub fn get_x() -> Result<i32, OtherError> { Ok(5) }
}
use pyo3::prelude::*;
use pyo3::exceptions::PyValueError;
use some_crate::{OtherError, get_x};
struct MyOtherError(OtherError);
impl From<MyOtherError> for PyErr {
fn from(error: MyOtherError) -> Self {
PyValueError::new_err(error.0.message())
}
}
impl From<OtherError> for MyOtherError {
fn from(other: OtherError) -> Self {
Self(other)
}
}
#[pyfunction]
fn wrapped_get_x() -> Result<i32, MyOtherError> {
// get_x is a function returning Result<i32, OtherError>
let x: i32 = get_x()?;
Ok(x)
}
fn main() {
Python::with_gil(|py| {
let fun = pyo3::wrap_pyfunction_bound!(wrapped_get_x, py).unwrap();
let value: usize = fun.call0().unwrap().extract().unwrap();
assert_eq!(value, 5);
});
}
Python classes
PyO3 exposes a group of attributes powered by Rust's proc macro system for defining Python classes as Rust structs.
The main attribute is #[pyclass]
, which is placed upon a Rust struct
or enum
to generate a Python type for it. They will usually also have one #[pymethods]
-annotated impl
block for the struct, which is used to define Python methods and constants for the generated Python type. (If the multiple-pymethods
feature is enabled, each #[pyclass]
is allowed to have multiple #[pymethods]
blocks.) #[pymethods]
may also have implementations for Python magic methods such as __str__
.
This chapter will discuss the functionality and configuration these attributes offer. Below is a list of links to the relevant section of this chapter for each:
Defining a new class
To define a custom Python class, add the #[pyclass]
attribute to a Rust struct or enum.
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
inner: i32,
}
// A "tuple" struct
#[pyclass]
struct Number(i32);
// PyO3 supports unit-only enums (which contain only unit variants)
// These simple enums behave similarly to Python's enumerations (enum.Enum)
#[pyclass]
enum MyEnum {
Variant,
OtherVariant = 30, // PyO3 supports custom discriminants.
}
// PyO3 supports custom discriminants in unit-only enums
#[pyclass]
enum HttpResponse {
Ok = 200,
NotFound = 404,
Teapot = 418,
// ...
}
// PyO3 also supports enums with non-unit variants
// These complex enums have sligtly different behavior from the simple enums above
// They are meant to work with instance checks and match statement patterns
#[pyclass]
enum Shape {
Circle { radius: f64 },
Rectangle { width: f64, height: f64 },
RegularPolygon { side_count: u32, radius: f64 },
Nothing {},
}
The above example generates implementations for PyTypeInfo
and PyClass
for MyClass
, Number
, MyEnum
, HttpResponse
, and Shape
. To see these generated implementations, refer to the implementation details at the end of this chapter.
Restrictions
To integrate Rust types with Python, PyO3 needs to place some restrictions on the types which can be annotated with #[pyclass]
. In particular, they must have no lifetime parameters, no generic parameters, and must implement Send
. The reason for each of these is explained below.
No lifetime parameters
Rust lifetimes are used by the Rust compiler to reason about a program's memory safety. They are a compile-time only concept; there is no way to access Rust lifetimes at runtime from a dynamic language like Python.
As soon as Rust data is exposed to Python, there is no guarantee that the Rust compiler can make on how long the data will live. Python is a reference-counted language and those references can be held for an arbitrarily long time which is untraceable by the Rust compiler. The only possible way to express this correctly is to require that any #[pyclass]
does not borrow data for any lifetime shorter than the 'static
lifetime, i.e. the #[pyclass]
cannot have any lifetime parameters.
When you need to share ownership of data between Python and Rust, instead of using borrowed references with lifetimes consider using reference-counted smart pointers such as Arc
or Py
.
No generic parameters
A Rust struct Foo<T>
with a generic parameter T
generates new compiled implementations each time it is used with a different concrete type for T
. These new implementations are generated by the compiler at each usage site. This is incompatible with wrapping Foo
in Python, where there needs to be a single compiled implementation of Foo
which is integrated with the Python interpreter.
Currently, the best alternative is to write a macro which expands to a new #[pyclass]
for each instantiation you want:
#![allow(dead_code)]
use pyo3::prelude::*;
struct GenericClass<T> {
data: T,
}
macro_rules! create_interface {
($name: ident, $type: ident) => {
#[pyclass]
pub struct $name {
inner: GenericClass<$type>,
}
#[pymethods]
impl $name {
#[new]
pub fn new(data: $type) -> Self {
Self {
inner: GenericClass { data: data },
}
}
}
};
}
create_interface!(IntClass, i64);
create_interface!(FloatClass, String);
Must be Send
Because Python objects are freely shared between threads by the Python interpreter, there is no guarantee which thread will eventually drop the object. Therefore all types annotated with #[pyclass]
must implement Send
(unless annotated with #[pyclass(unsendable)]
).
Constructor
By default, it is not possible to create an instance of a custom class from Python code.
To declare a constructor, you need to define a method and annotate it with the #[new]
attribute. Only Python's __new__
method can be specified, __init__
is not available.
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
#[new]
fn new(value: i32) -> Self {
Number(value)
}
}
Alternatively, if your new
method may fail you can return PyResult<Self>
.
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::exceptions::PyValueError;
#[pyclass]
struct Nonzero(i32);
#[pymethods]
impl Nonzero {
#[new]
fn py_new(value: i32) -> PyResult<Self> {
if value == 0 {
Err(PyValueError::new_err("cannot be zero"))
} else {
Ok(Nonzero(value))
}
}
}
If you want to return an existing object (for example, because your new
method caches the values it returns), new
can return pyo3::Py<Self>
.
As you can see, the Rust method name is not important here; this way you can
still, use new()
for a Rust-level constructor.
If no method marked with #[new]
is declared, object instances can only be
created from Rust, but not from Python.
For arguments, see the Method arguments
section below.
Adding the class to a module
The next step is to create the module initializer and add our class to it:
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymodule]
fn my_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<Number>()?;
Ok(())
}
Bound and interior mutability
Often is useful to turn a #[pyclass]
type T
into a Python object and access it from Rust code. The [Py<T>
] and [Bound<'py, T>
] smart pointers are the ways to represent a Python object in PyO3's API. More detail can be found about them in the Python objects section of the guide.
Most Python objects do not offer exclusive (&mut
) access (see the section on Python's memory model). However, Rust structs wrapped as Python objects (called pyclass
types) often do need &mut
access. Due to the GIL, PyO3 can guarantee exclusive access to them.
The Rust borrow checker cannot reason about &mut
references once an object's ownership has been passed to the Python interpreter. This means that borrow checking is done at runtime using with a scheme very similar to std::cell::RefCell<T>
. This is known as interior mutability.
Users who are familiar with RefCell<T>
can use Py<T>
and Bound<'py, T>
just like RefCell<T>
.
For users who are not very familiar with RefCell<T>
, here is a reminder of Rust's rules of borrowing:
- At any given time, you can have either (but not both of) one mutable reference or any number of immutable references.
- References can never outlast the data they refer to.
Py<T>
and Bound<'py, T>
, like RefCell<T>
, ensure these borrowing rules by tracking references at runtime.
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
#[pyo3(get)]
num: i32,
}
Python::with_gil(|py| {
let obj = Bound::new(py, MyClass { num: 3 }).unwrap();
{
let obj_ref = obj.borrow(); // Get PyRef
assert_eq!(obj_ref.num, 3);
// You cannot get PyRefMut unless all PyRefs are dropped
assert!(obj.try_borrow_mut().is_err());
}
{
let mut obj_mut = obj.borrow_mut(); // Get PyRefMut
obj_mut.num = 5;
// You cannot get any other refs until the PyRefMut is dropped
assert!(obj.try_borrow().is_err());
assert!(obj.try_borrow_mut().is_err());
}
// You can convert `Bound` to a Python object
pyo3::py_run!(py, obj, "assert obj.num == 5");
});
A Bound<'py, T>
is restricted to the GIL lifetime 'py
. To make the object longer lived (for example, to store it in a struct on the
Rust side), use Py<T>
. Py<T>
needs a Python<'_>
token to allow access:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
num: i32,
}
fn return_myclass() -> Py<MyClass> {
Python::with_gil(|py| Py::new(py, MyClass { num: 1 }).unwrap())
}
let obj = return_myclass();
Python::with_gil(|py| {
let bound = obj.bind(py); // Py<MyClass>::bind returns &Bound<'py, MyClass>
let obj_ref = bound.borrow(); // Get PyRef<T>
assert_eq!(obj_ref.num, 1);
});
frozen classes: Opting out of interior mutability
As detailed above, runtime borrow checking is currently enabled by default. But a class can opt of out it by declaring itself frozen
. It can still use interior mutability via standard Rust types like RefCell
or Mutex
, but it is not bound to the implementation provided by PyO3 and can choose the most appropriate strategy on field-by-field basis.
Classes which are frozen
and also Sync
, e.g. they do use Mutex
but not RefCell
, can be accessed without needing the Python GIL via the Bound::get
and Py::get
methods:
use std::sync::atomic::{AtomicUsize, Ordering};
use pyo3::prelude::*;
#[pyclass(frozen)]
struct FrozenCounter {
value: AtomicUsize,
}
let py_counter: Py<FrozenCounter> = Python::with_gil(|py| {
let counter = FrozenCounter {
value: AtomicUsize::new(0),
};
Py::new(py, counter).unwrap()
});
py_counter.get().value.fetch_add(1, Ordering::Relaxed);
Frozen classes are likely to become the default thereby guiding the PyO3 ecosystem towards a more deliberate application of interior mutability. Eventually, this should enable further optimizations of PyO3's internals and avoid downstream code paying the cost of interior mutability when it is not actually required.
Customizing the class
#[pyclass]
can be used with the following parameters:
Parameter | Description |
---|---|
crate = "some::path" | Path to import the pyo3 crate, if it's not accessible at ::pyo3 . |
dict | Gives instances of this class an empty __dict__ to store custom attributes. |
extends = BaseType | Use a custom baseclass. Defaults to PyAny |
freelist = N | Implements a free list of size N. This can improve performance for types that are often created and deleted in quick succession. Profile your code to see whether freelist is right for you. |
frozen | Declares that your pyclass is immutable. It removes the borrow checker overhead when retrieving a shared reference to the Rust struct, but disables the ability to get a mutable reference. |
get_all | Generates getters for all fields of the pyclass. |
mapping | Inform PyO3 that this class is a Mapping , and so leave its implementation of sequence C-API slots empty. |
module = "module_name" | Python code will see the class as being defined in this module. Defaults to builtins . |
name = "python_name" | Sets the name that Python sees this class as. Defaults to the name of the Rust struct. |
rename_all = "renaming_rule" | Applies renaming rules to every getters and setters of a struct, or every variants of an enum. Possible values are: "camelCase", "kebab-case", "lowercase", "PascalCase", "SCREAMING-KEBAB-CASE", "SCREAMING_SNAKE_CASE", "snake_case", "UPPERCASE". |
sequence | Inform PyO3 that this class is a Sequence , and so leave its C-API mapping length slot empty. |
set_all | Generates setters for all fields of the pyclass. |
subclass | Allows other Python classes and #[pyclass] to inherit from this class. Enums cannot be subclassed. |
text_signature = "(arg1, arg2, ...)" | Sets the text signature for the Python class' __new__ method. |
unsendable | Required if your struct is not Send . Rather than using unsendable , consider implementing your struct in a threadsafe way by e.g. substituting Rc with Arc . By using unsendable , your class will panic when accessed by another thread. Also note the Python's GC is multi-threaded and while unsendable classes will not be traversed on foreign threads to avoid UB, this can lead to memory leaks. |
weakref | Allows this class to be weakly referenceable. |
All of these parameters can either be passed directly on the #[pyclass(...)]
annotation, or as one or
more accompanying #[pyo3(...)]
annotations, e.g.:
// Argument supplied directly to the `#[pyclass]` annotation.
#[pyclass(name = "SomeName", subclass)]
struct MyClass {}
// Argument supplied as a separate annotation.
#[pyclass]
#[pyo3(name = "SomeName", subclass)]
struct MyClass {}
These parameters are covered in various sections of this guide.
Return type
Generally, #[new]
methods have to return T: Into<PyClassInitializer<Self>>
or
PyResult<T> where T: Into<PyClassInitializer<Self>>
.
For constructors that may fail, you should wrap the return type in a PyResult as well. Consult the table below to determine which type your constructor should return:
Cannot fail | May fail | |
---|---|---|
No inheritance | T | PyResult<T> |
Inheritance(T Inherits U) | (T, U) | PyResult<(T, U)> |
Inheritance(General Case) | PyClassInitializer<T> | PyResult<PyClassInitializer<T>> |
Inheritance
By default, object
, i.e. PyAny
is used as the base class. To override this default,
use the extends
parameter for pyclass
with the full path to the base class.
Currently, only classes defined in Rust and builtins provided by PyO3 can be inherited
from; inheriting from other classes defined in Python is not yet supported
(#991).
For convenience, (T, U)
implements Into<PyClassInitializer<T>>
where U
is the
base class of T
.
But for a more deeply nested inheritance, you have to return PyClassInitializer<T>
explicitly.
To get a parent class from a child, use PyRef
instead of &self
for methods,
or PyRefMut
instead of &mut self
.
Then you can access a parent class by self_.as_ref()
as &Self::BaseClass
,
or by self_.into_super()
as PyRef<Self::BaseClass>
.
use pyo3::prelude::*;
#[pyclass(subclass)]
struct BaseClass {
val1: usize,
}
#[pymethods]
impl BaseClass {
#[new]
fn new() -> Self {
BaseClass { val1: 10 }
}
pub fn method(&self) -> PyResult<usize> {
Ok(self.val1)
}
}
#[pyclass(extends=BaseClass, subclass)]
struct SubClass {
val2: usize,
}
#[pymethods]
impl SubClass {
#[new]
fn new() -> (Self, BaseClass) {
(SubClass { val2: 15 }, BaseClass::new())
}
fn method2(self_: PyRef<'_, Self>) -> PyResult<usize> {
let super_ = self_.as_ref(); // Get &BaseClass
super_.method().map(|x| x * self_.val2)
}
}
#[pyclass(extends=SubClass)]
struct SubSubClass {
val3: usize,
}
#[pymethods]
impl SubSubClass {
#[new]
fn new() -> PyClassInitializer<Self> {
PyClassInitializer::from(SubClass::new()).add_subclass(SubSubClass { val3: 20 })
}
fn method3(self_: PyRef<'_, Self>) -> PyResult<usize> {
let v = self_.val3;
let super_ = self_.into_super(); // Get PyRef<'_, SubClass>
SubClass::method2(super_).map(|x| x * v)
}
#[staticmethod]
fn factory_method(py: Python<'_>, val: usize) -> PyResult<PyObject> {
let base = PyClassInitializer::from(BaseClass::new());
let sub = base.add_subclass(SubClass { val2: val });
if val % 2 == 0 {
Ok(Py::new(py, sub)?.to_object(py))
} else {
let sub_sub = sub.add_subclass(SubSubClass { val3: val });
Ok(Py::new(py, sub_sub)?.to_object(py))
}
}
}
Python::with_gil(|py| {
let subsub = pyo3::Py::new(py, SubSubClass::new()).unwrap();
pyo3::py_run!(py, subsub, "assert subsub.method3() == 3000");
let subsub = SubSubClass::factory_method(py, 2).unwrap();
let subsubsub = SubSubClass::factory_method(py, 3).unwrap();
let cls = py.get_type_bound::<SubSubClass>();
pyo3::py_run!(py, subsub cls, "assert not isinstance(subsub, cls)");
pyo3::py_run!(py, subsubsub cls, "assert isinstance(subsubsub, cls)");
});
You can inherit native types such as PyDict
, if they implement
PySizedLayout
.
This is not supported when building for the Python limited API (aka the abi3
feature of PyO3).
To convert between the Rust type and its native base class, you can take
slf
as a Python object. To access the Rust fields use slf.borrow()
or
slf.borrow_mut()
, and to access the base class use slf.downcast::<BaseClass>()
.
#[cfg(not(Py_LIMITED_API))] {
use pyo3::prelude::*;
use pyo3::types::PyDict;
use std::collections::HashMap;
#[pyclass(extends=PyDict)]
#[derive(Default)]
struct DictWithCounter {
counter: HashMap<String, usize>,
}
#[pymethods]
impl DictWithCounter {
#[new]
fn new() -> Self {
Self::default()
}
fn set(slf: &Bound<'_, Self>, key: String, value: Bound<'_, PyAny>) -> PyResult<()> {
slf.borrow_mut().counter.entry(key.clone()).or_insert(0);
let dict = slf.downcast::<PyDict>()?;
dict.set_item(key, value)
}
}
Python::with_gil(|py| {
let cnt = pyo3::Py::new(py, DictWithCounter::new()).unwrap();
pyo3::py_run!(py, cnt, "cnt.set('abc', 10); assert cnt['abc'] == 10")
});
}
If SubClass
does not provide a base class initialization, the compilation fails.
use pyo3::prelude::*;
#[pyclass]
struct BaseClass {
val1: usize,
}
#[pyclass(extends=BaseClass)]
struct SubClass {
val2: usize,
}
#[pymethods]
impl SubClass {
#[new]
fn new() -> Self {
SubClass { val2: 15 }
}
}
The __new__
constructor of a native base class is called implicitly when
creating a new instance from Python. Be sure to accept arguments in the
#[new]
method that you want the base class to get, even if they are not used
in that fn
:
#[allow(dead_code)]
#[cfg(not(Py_LIMITED_API))] {
use pyo3::prelude::*;
use pyo3::types::PyDict;
#[pyclass(extends=PyDict)]
struct MyDict {
private: i32,
}
#[pymethods]
impl MyDict {
#[new]
#[pyo3(signature = (*args, **kwargs))]
fn new(args: &Bound<'_, PyAny>, kwargs: Option<&Bound<'_, PyAny>>) -> Self {
Self { private: 0 }
}
// some custom methods that use `private` here...
}
Python::with_gil(|py| {
let cls = py.get_type_bound::<MyDict>();
pyo3::py_run!(py, cls, "cls(a=1, b=2)")
});
}
Here, the args
and kwargs
allow creating instances of the subclass passing
initial items, such as MyDict(item_sequence)
or MyDict(a=1, b=2)
.
Object properties
PyO3 supports two ways to add properties to your #[pyclass]
:
- For simple struct fields with no side effects, a
#[pyo3(get, set)]
attribute can be added directly to the field definition in the#[pyclass]
. - For properties which require computation you can define
#[getter]
and#[setter]
functions in the#[pymethods]
block.
We'll cover each of these in the following sections.
Object properties using #[pyo3(get, set)]
For simple cases where a member variable is just read and written with no side effects, you can declare getters and setters in your #[pyclass]
field definition using the pyo3
attribute, like in the example below:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
#[pyo3(get, set)]
num: i32,
}
The above would make the num
field available for reading and writing as a self.num
Python property. To expose the property with a different name to the field, specify this alongside the rest of the options, e.g. #[pyo3(get, set, name = "custom_name")]
.
Properties can be readonly or writeonly by using just #[pyo3(get)]
or #[pyo3(set)]
respectively.
To use these annotations, your field type must implement some conversion traits:
- For
get
the field type must implement bothIntoPy<PyObject>
andClone
. - For
set
the field type must implementFromPyObject
.
For example, implementations of those traits are provided for the Cell
type, if the inner type also implements the trait. This means you can use #[pyo3(get, set)]
on fields wrapped in a Cell
.
Object properties using #[getter]
and #[setter]
For cases which don't satisfy the #[pyo3(get, set)]
trait requirements, or need side effects, descriptor methods can be defined in a #[pymethods]
impl
block.
This is done using the #[getter]
and #[setter]
attributes, like in the example below:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
num: i32,
}
#[pymethods]
impl MyClass {
#[getter]
fn num(&self) -> PyResult<i32> {
Ok(self.num)
}
}
A getter or setter's function name is used as the property name by default. There are several ways how to override the name.
If a function name starts with get_
or set_
for getter or setter respectively,
the descriptor name becomes the function name with this prefix removed. This is also useful in case of
Rust keywords like type
(raw identifiers
can be used since Rust 2018).
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
num: i32,
}
#[pymethods]
impl MyClass {
#[getter]
fn get_num(&self) -> PyResult<i32> {
Ok(self.num)
}
#[setter]
fn set_num(&mut self, value: i32) -> PyResult<()> {
self.num = value;
Ok(())
}
}
In this case, a property num
is defined and available from Python code as self.num
.
Both the #[getter]
and #[setter]
attributes accept one parameter.
If this parameter is specified, it is used as the property name, i.e.
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
num: i32,
}
#[pymethods]
impl MyClass {
#[getter(number)]
fn num(&self) -> PyResult<i32> {
Ok(self.num)
}
#[setter(number)]
fn set_num(&mut self, value: i32) -> PyResult<()> {
self.num = value;
Ok(())
}
}
In this case, the property number
is defined and available from Python code as self.number
.
Attributes defined by #[setter]
or #[pyo3(set)]
will always raise AttributeError
on del
operations. Support for defining custom del
behavior is tracked in
#1778.
Instance methods
To define a Python compatible method, an impl
block for your struct has to be annotated with the
#[pymethods]
attribute. PyO3 generates Python compatible wrappers for all functions in this
block with some variations, like descriptors, class method static methods, etc.
Since Rust allows any number of impl
blocks, you can easily split methods
between those accessible to Python (and Rust) and those accessible only to Rust. However to have multiple
#[pymethods]
-annotated impl
blocks for the same struct you must enable the multiple-pymethods
feature of PyO3.
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
num: i32,
}
#[pymethods]
impl MyClass {
fn method1(&self) -> PyResult<i32> {
Ok(10)
}
fn set_method(&mut self, value: i32) -> PyResult<()> {
self.num = value;
Ok(())
}
}
Calls to these methods are protected by the GIL, so both &self
and &mut self
can be used.
The return type must be PyResult<T>
or T
for some T
that implements IntoPy<PyObject>
;
the latter is allowed if the method cannot raise Python exceptions.
A Python
parameter can be specified as part of method signature, in this case the py
argument
gets injected by the method wrapper, e.g.
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
#[allow(dead_code)]
num: i32,
}
#[pymethods]
impl MyClass {
fn method2(&self, py: Python<'_>) -> PyResult<i32> {
Ok(10)
}
}
From the Python perspective, the method2
in this example does not accept any arguments.
Class methods
To create a class method for a custom class, the method needs to be annotated
with the #[classmethod]
attribute.
This is the equivalent of the Python decorator @classmethod
.
use pyo3::prelude::*;
use pyo3::types::PyType;
#[pyclass]
struct MyClass {
#[allow(dead_code)]
num: i32,
}
#[pymethods]
impl MyClass {
#[classmethod]
fn cls_method(cls: &Bound<'_, PyType>) -> PyResult<i32> {
Ok(10)
}
}
Declares a class method callable from Python.
- The first parameter is the type object of the class on which the method is called. This may be the type object of a derived class.
- The first parameter implicitly has type
&Bound<'_, PyType>
. - For details on
parameter-list
, see the documentation ofMethod arguments
section. - The return type must be
PyResult<T>
orT
for someT
that implementsIntoPy<PyObject>
.
Constructors which accept a class argument
To create a constructor which takes a positional class argument, you can combine the #[classmethod]
and #[new]
modifiers:
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyType;
#[pyclass]
struct BaseClass(PyObject);
#[pymethods]
impl BaseClass {
#[new]
#[classmethod]
fn py_new(cls: &Bound<'_, PyType>) -> PyResult<Self> {
// Get an abstract attribute (presumably) declared on a subclass of this class.
let subclass_attr: Bound<'_, PyAny> = cls.getattr("a_class_attr")?;
Ok(Self(subclass_attr.unbind()))
}
}
Static methods
To create a static method for a custom class, the method needs to be annotated with the
#[staticmethod]
attribute. The return type must be T
or PyResult<T>
for some T
that implements
IntoPy<PyObject>
.
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
#[allow(dead_code)]
num: i32,
}
#[pymethods]
impl MyClass {
#[staticmethod]
fn static_method(param1: i32, param2: &str) -> PyResult<i32> {
Ok(10)
}
}
Class attributes
To create a class attribute (also called class variable), a method without
any arguments can be annotated with the #[classattr]
attribute.
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
#[classattr]
fn my_attribute() -> String {
"hello".to_string()
}
}
Python::with_gil(|py| {
let my_class = py.get_type_bound::<MyClass>();
pyo3::py_run!(py, my_class, "assert my_class.my_attribute == 'hello'")
});
Note: if the method has a
Result
return type and returns anErr
, PyO3 will panic during class creation.
If the class attribute is defined with const
code only, one can also annotate associated
constants:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
#[classattr]
const MY_CONST_ATTRIBUTE: &'static str = "foobar";
}
Classes as function arguments
Free functions defined using #[pyfunction]
interact with classes through the same mechanisms as the self parameters of instance methods, i.e. they can take GIL-bound references, GIL-bound reference wrappers or GIL-indepedent references:
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
my_field: i32,
}
// Take a reference when the underlying `Bound` is irrelevant.
#[pyfunction]
fn increment_field(my_class: &mut MyClass) {
my_class.my_field += 1;
}
// Take a reference wrapper when borrowing should be automatic,
// but interaction with the underlying `Bound` is desired.
#[pyfunction]
fn print_field(my_class: PyRef<'_, MyClass>) {
println!("{}", my_class.my_field);
}
// Take a reference to the underlying Bound
// when borrowing needs to be managed manually.
#[pyfunction]
fn increment_then_print_field(my_class: &Bound<'_, MyClass>) {
my_class.borrow_mut().my_field += 1;
println!("{}", my_class.borrow().my_field);
}
// Take a GIL-indepedent reference when you want to store the reference elsewhere.
#[pyfunction]
fn print_refcnt(my_class: Py<MyClass>, py: Python<'_>) {
println!("{}", my_class.get_refcnt(py));
}
Classes can also be passed by value if they can be cloned, i.e. they automatically implement FromPyObject
if they implement Clone
, e.g. via #[derive(Clone)]
:
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyclass]
#[derive(Clone)]
struct MyClass {
my_field: Box<i32>,
}
#[pyfunction]
fn dissamble_clone(my_class: MyClass) {
let MyClass { mut my_field } = my_class;
*my_field += 1;
}
Note that #[derive(FromPyObject)]
on a class is usually not useful as it tries to construct a new Rust value by filling in the fields by looking up attributes of any given Python value.
Method arguments
Similar to #[pyfunction]
, the #[pyo3(signature = (...))]
attribute can be used to specify the way that #[pymethods]
accept arguments. Consult the documentation for function signatures
to see the parameters this attribute accepts.
The following example defines a class MyClass
with a method method
. This method has a signature that sets default values for num
and name
, and indicates that py_args
should collect all extra positional arguments and py_kwargs
all extra keyword arguments:
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyTuple};
#[pyclass]
struct MyClass {
num: i32,
}
#[pymethods]
impl MyClass {
#[new]
#[pyo3(signature = (num=-1))]
fn new(num: i32) -> Self {
MyClass { num }
}
#[pyo3(signature = (num=10, *py_args, name="Hello", **py_kwargs))]
fn method(
&mut self,
num: i32,
py_args: &Bound<'_, PyTuple>,
name: &str,
py_kwargs: Option<&Bound<'_, PyDict>>,
) -> String {
let num_before = self.num;
self.num = num;
format!(
"num={} (was previously={}), py_args={:?}, name={}, py_kwargs={:?} ",
num, num_before, py_args, name, py_kwargs,
)
}
}
In Python, this might be used like:
>>> import mymodule
>>> mc = mymodule.MyClass()
>>> print(mc.method(44, False, "World", 666, x=44, y=55))
py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44, num_before=-1
>>> print(mc.method(num=-1, name="World"))
py_args=(), py_kwargs=None, name=World, num=-1, num_before=44
The #[pyo3(text_signature = "...")
option for #[pyfunction]
also works for #[pymethods]
.
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyType;
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
#[new]
#[pyo3(text_signature = "(c, d)")]
fn new(c: i32, d: &str) -> Self {
Self {}
}
// the self argument should be written $self
#[pyo3(text_signature = "($self, e, f)")]
fn my_method(&self, e: i32, f: i32) -> i32 {
e + f
}
// similarly for classmethod arguments, use $cls
#[classmethod]
#[pyo3(text_signature = "($cls, e, f)")]
fn my_class_method(cls: &Bound<'_, PyType>, e: i32, f: i32) -> i32 {
e + f
}
#[staticmethod]
#[pyo3(text_signature = "(e, f)")]
fn my_static_method(e: i32, f: i32) -> i32 {
e + f
}
}
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let inspect = PyModule::import_bound(py, "inspect")?.getattr("signature")?;
let module = PyModule::new_bound(py, "my_module")?;
module.add_class::<MyClass>()?;
let class = module.getattr("MyClass")?;
if cfg!(not(Py_LIMITED_API)) || py.version_info() >= (3, 10) {
let doc: String = class.getattr("__doc__")?.extract()?;
assert_eq!(doc, "");
let sig: String = inspect
.call1((&class,))?
.call_method0("__str__")?
.extract()?;
assert_eq!(sig, "(c, d)");
} else {
let doc: String = class.getattr("__doc__")?.extract()?;
assert_eq!(doc, "");
inspect.call1((&class,)).expect_err("`text_signature` on classes is not compatible with compilation in `abi3` mode until Python 3.10 or greater");
}
{
let method = class.getattr("my_method")?;
assert!(method.getattr("__doc__")?.is_none());
let sig: String = inspect
.call1((method,))?
.call_method0("__str__")?
.extract()?;
assert_eq!(sig, "(self, /, e, f)");
}
{
let method = class.getattr("my_class_method")?;
assert!(method.getattr("__doc__")?.is_none());
let sig: String = inspect
.call1((method,))?
.call_method0("__str__")?
.extract()?;
assert_eq!(sig, "(e, f)"); // inspect.signature skips the $cls arg
}
{
let method = class.getattr("my_static_method")?;
assert!(method.getattr("__doc__")?.is_none());
let sig: String = inspect
.call1((method,))?
.call_method0("__str__")?
.extract()?;
assert_eq!(sig, "(e, f)");
}
Ok(())
})
}
Note that text_signature
on #[new]
is not compatible with compilation in
abi3
mode until Python 3.10 or greater.
#[pyclass]
enums
Enum support in PyO3 comes in two flavors, depending on what kind of variants the enum has: simple and complex.
Simple enums
A simple enum (a.k.a. C-like enum) has only unit variants.
PyO3 adds a class attribute for each variant, so you can access them in Python without defining #[new]
. PyO3 also provides default implementations of __richcmp__
and __int__
, so they can be compared using ==
:
use pyo3::prelude::*;
#[pyclass]
enum MyEnum {
Variant,
OtherVariant,
}
Python::with_gil(|py| {
let x = Py::new(py, MyEnum::Variant).unwrap();
let y = Py::new(py, MyEnum::OtherVariant).unwrap();
let cls = py.get_type_bound::<MyEnum>();
pyo3::py_run!(py, x y cls, r#"
assert x == cls.Variant
assert y == cls.OtherVariant
assert x != y
"#)
})
You can also convert your simple enums into int
:
use pyo3::prelude::*;
#[pyclass]
enum MyEnum {
Variant,
OtherVariant = 10,
}
Python::with_gil(|py| {
let cls = py.get_type_bound::<MyEnum>();
let x = MyEnum::Variant as i32; // The exact value is assigned by the compiler.
pyo3::py_run!(py, cls x, r#"
assert int(cls.Variant) == x
assert int(cls.OtherVariant) == 10
assert cls.OtherVariant == 10 # You can also compare against int.
assert 10 == cls.OtherVariant
"#)
})
PyO3 also provides __repr__
for enums:
use pyo3::prelude::*;
#[pyclass]
enum MyEnum{
Variant,
OtherVariant,
}
Python::with_gil(|py| {
let cls = py.get_type_bound::<MyEnum>();
let x = Py::new(py, MyEnum::Variant).unwrap();
pyo3::py_run!(py, cls x, r#"
assert repr(x) == 'MyEnum.Variant'
assert repr(cls.OtherVariant) == 'MyEnum.OtherVariant'
"#)
})
All methods defined by PyO3 can be overridden. For example here's how you override __repr__
:
use pyo3::prelude::*;
#[pyclass]
enum MyEnum {
Answer = 42,
}
#[pymethods]
impl MyEnum {
fn __repr__(&self) -> &'static str {
"42"
}
}
Python::with_gil(|py| {
let cls = py.get_type_bound::<MyEnum>();
pyo3::py_run!(py, cls, "assert repr(cls.Answer) == '42'")
})
Enums and their variants can also be renamed using #[pyo3(name)]
.
use pyo3::prelude::*;
#[pyclass(name = "RenamedEnum")]
enum MyEnum {
#[pyo3(name = "UPPERCASE")]
Variant,
}
Python::with_gil(|py| {
let x = Py::new(py, MyEnum::Variant).unwrap();
let cls = py.get_type_bound::<MyEnum>();
pyo3::py_run!(py, x cls, r#"
assert repr(x) == 'RenamedEnum.UPPERCASE'
assert x == cls.UPPERCASE
"#)
})
You may not use enums as a base class or let enums inherit from other classes.
use pyo3::prelude::*;
#[pyclass(subclass)]
enum BadBase {
Var1,
}
use pyo3::prelude::*;
#[pyclass(subclass)]
struct Base;
#[pyclass(extends=Base)]
enum BadSubclass {
Var1,
}
#[pyclass]
enums are currently not interoperable with IntEnum
in Python.
Complex enums
An enum is complex if it has any non-unit (struct or tuple) variants.
Currently PyO3 supports only struct variants in a complex enum. Support for unit and tuple variants is planned.
PyO3 adds a class attribute for each variant, which may be used to construct values and in match patterns. PyO3 also provides getter methods for all fields of each variant.
use pyo3::prelude::*;
#[pyclass]
enum Shape {
Circle { radius: f64 },
Rectangle { width: f64, height: f64 },
RegularPolygon { side_count: u32, radius: f64 },
Nothing { },
}
#[cfg(Py_3_10)]
Python::with_gil(|py| {
let circle = Shape::Circle { radius: 10.0 }.into_py(py);
let square = Shape::RegularPolygon { side_count: 4, radius: 10.0 }.into_py(py);
let cls = py.get_type_bound::<Shape>();
pyo3::py_run!(py, circle square cls, r#"
assert isinstance(circle, cls)
assert isinstance(circle, cls.Circle)
assert circle.radius == 10.0
assert isinstance(square, cls)
assert isinstance(square, cls.RegularPolygon)
assert square.side_count == 4
assert square.radius == 10.0
def count_vertices(cls, shape):
match shape:
case cls.Circle():
return 0
case cls.Rectangle():
return 4
case cls.RegularPolygon(side_count=n):
return n
case cls.Nothing():
return 0
assert count_vertices(cls, circle) == 0
assert count_vertices(cls, square) == 4
"#)
})
WARNING: Py::new
and .into_py
are currently inconsistent. Note how the constructed value is not an instance of the specific variant. For this reason, constructing values is only recommended using .into_py
.
use pyo3::prelude::*;
#[pyclass]
enum MyEnum {
Variant { i: i32 },
}
Python::with_gil(|py| {
let x = Py::new(py, MyEnum::Variant { i: 42 }).unwrap();
let cls = py.get_type_bound::<MyEnum>();
pyo3::py_run!(py, x cls, r#"
assert isinstance(x, cls)
assert not isinstance(x, cls.Variant)
"#)
})
Implementation details
The #[pyclass]
macros rely on a lot of conditional code generation: each #[pyclass]
can optionally have a #[pymethods]
block.
To support this flexibility the #[pyclass]
macro expands to a blob of boilerplate code which sets up the structure for "dtolnay specialization". This implementation pattern enables the Rust compiler to use #[pymethods]
implementations when they are present, and fall back to default (empty) definitions when they are not.
This simple technique works for the case when there is zero or one implementations. To support multiple #[pymethods]
for a #[pyclass]
(in the multiple-pymethods
feature), a registry mechanism provided by the inventory
crate is used instead. This collects impl
s at library load time, but isn't supported on all platforms. See inventory: how it works for more details.
The #[pyclass]
macro expands to roughly the code seen below. The PyClassImplCollector
is the type used internally by PyO3 for dtolnay specialization:
#[cfg(not(feature = "multiple-pymethods"))] {
use pyo3::prelude::*;
// Note: the implementation differs slightly with the `multiple-pymethods` feature enabled.
struct MyClass {
#[allow(dead_code)]
num: i32,
}
impl pyo3::types::DerefToPyAny for MyClass {}
#[allow(deprecated)]
unsafe impl pyo3::type_object::HasPyGilRef for MyClass {
type AsRefTarget = pyo3::PyCell<Self>;
}
unsafe impl pyo3::type_object::PyTypeInfo for MyClass {
const NAME: &'static str = "MyClass";
const MODULE: ::std::option::Option<&'static str> = ::std::option::Option::None;
#[inline]
fn type_object_raw(py: pyo3::Python<'_>) -> *mut pyo3::ffi::PyTypeObject {
<Self as pyo3::impl_::pyclass::PyClassImpl>::lazy_type_object()
.get_or_init(py)
.as_type_ptr()
}
}
impl pyo3::PyClass for MyClass {
type Frozen = pyo3::pyclass::boolean_struct::False;
}
impl<'a, 'py> pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'py> for &'a MyClass
{
type Holder = ::std::option::Option<pyo3::PyRef<'py, MyClass>>;
#[inline]
fn extract(obj: &'a pyo3::Bound<'py, PyAny>, holder: &'a mut Self::Holder) -> pyo3::PyResult<Self> {
pyo3::impl_::extract_argument::extract_pyclass_ref(obj, holder)
}
}
impl<'a, 'py> pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'py> for &'a mut MyClass
{
type Holder = ::std::option::Option<pyo3::PyRefMut<'py, MyClass>>;
#[inline]
fn extract(obj: &'a pyo3::Bound<'py, PyAny>, holder: &'a mut Self::Holder) -> pyo3::PyResult<Self> {
pyo3::impl_::extract_argument::extract_pyclass_ref_mut(obj, holder)
}
}
impl pyo3::IntoPy<PyObject> for MyClass {
fn into_py(self, py: pyo3::Python<'_>) -> pyo3::PyObject {
pyo3::IntoPy::into_py(pyo3::Py::new(py, self).unwrap(), py)
}
}
impl pyo3::impl_::pyclass::PyClassImpl for MyClass {
const IS_BASETYPE: bool = false;
const IS_SUBCLASS: bool = false;
const IS_MAPPING: bool = false;
const IS_SEQUENCE: bool = false;
type BaseType = PyAny;
type ThreadChecker = pyo3::impl_::pyclass::SendablePyClass<MyClass>;
type PyClassMutability = <<pyo3::PyAny as pyo3::impl_::pyclass::PyClassBaseType>::PyClassMutability as pyo3::impl_::pycell::PyClassMutability>::MutableChild;
type Dict = pyo3::impl_::pyclass::PyClassDummySlot;
type WeakRef = pyo3::impl_::pyclass::PyClassDummySlot;
type BaseNativeType = pyo3::PyAny;
fn items_iter() -> pyo3::impl_::pyclass::PyClassItemsIter {
use pyo3::impl_::pyclass::*;
let collector = PyClassImplCollector::<MyClass>::new();
static INTRINSIC_ITEMS: PyClassItems = PyClassItems { slots: &[], methods: &[] };
PyClassItemsIter::new(&INTRINSIC_ITEMS, collector.py_methods())
}
fn lazy_type_object() -> &'static pyo3::impl_::pyclass::LazyTypeObject<MyClass> {
use pyo3::impl_::pyclass::LazyTypeObject;
static TYPE_OBJECT: LazyTypeObject<MyClass> = LazyTypeObject::new();
&TYPE_OBJECT
}
fn doc(py: Python<'_>) -> pyo3::PyResult<&'static ::std::ffi::CStr> {
use pyo3::impl_::pyclass::*;
static DOC: pyo3::sync::GILOnceCell<::std::borrow::Cow<'static, ::std::ffi::CStr>> = pyo3::sync::GILOnceCell::new();
DOC.get_or_try_init(py, || {
let collector = PyClassImplCollector::<Self>::new();
build_pyclass_doc(<MyClass as pyo3::PyTypeInfo>::NAME, "\0", collector.new_text_signature())
}).map(::std::ops::Deref::deref)
}
}
Python::with_gil(|py| {
let cls = py.get_type_bound::<MyClass>();
pyo3::py_run!(py, cls, "assert cls.__name__ == 'MyClass'")
});
}
Magic methods and slots
Python's object model defines several protocols for different object behavior, such as the sequence, mapping, and number protocols. You may be familiar with implementing these protocols in Python classes by "magic" methods, such as __str__
or __repr__
. Because of the double-underscores surrounding their name, these are also known as "dunder" methods.
In the Python C-API which PyO3 is implemented upon, many of these magic methods have to be placed into special "slots" on the class type object, as covered in the previous section.
If a function name in #[pymethods]
is a recognised magic method, it will be automatically placed into the correct slot in the Python type object. The function name is taken from the usual rules for naming #[pymethods]
: the #[pyo3(name = "...")]
attribute is used if present, otherwise the Rust function name is used.
The magic methods handled by PyO3 are very similar to the standard Python ones on this page - in particular they are the subset which have slots as defined here. Some of the slots do not have a magic method in Python, which leads to a few additional magic methods defined only in PyO3:
- Magic methods for garbage collection
- Magic methods for the buffer protocol
When PyO3 handles a magic method, a couple of changes apply compared to other #[pymethods]
:
- The Rust function signature is restricted to match the magic method.
- The
#[pyo3(signature = (...)]
and#[pyo3(text_signature = "...")]
attributes are not allowed.
The following sections list of all magic methods PyO3 currently handles. The given signatures should be interpreted as follows:
- All methods take a receiver as first argument, shown as
<self>
. It can be&self
,&mut self
or aBound
reference likeself_: PyRef<'_, Self>
andself_: PyRefMut<'_, Self>
, as described here. - An optional
Python<'py>
argument is always allowed as the first argument. - Return values can be optionally wrapped in
PyResult
. object
means that any type is allowed that can be extracted from a Python object (if argument) or converted to a Python object (if return value).- Other types must match what's given, e.g.
pyo3::basic::CompareOp
for__richcmp__
's second argument. - For the comparison and arithmetic methods, extraction errors are not
propagated as exceptions, but lead to a return of
NotImplemented
. - For some magic methods, the return values are not restricted by PyO3, but
checked by the Python interpreter. For example,
__str__
needs to return a string object. This is indicated byobject (Python type)
.
Basic object customization
-
__str__(<self>) -> object (str)
-
__repr__(<self>) -> object (str)
-
__hash__(<self>) -> isize
Objects that compare equal must have the same hash value. Any type up to 64 bits may be returned instead of
isize
, PyO3 will convert to an isize automatically (wrapping unsigned types likeu64
andusize
).Disabling Python's default hash
By default, all `#[pyclass]` types have a default hash implementation from Python. Types which should not be hashable can override this by setting `__hash__` to `None`. This is the same mechanism as for a pure-Python class. This is done like so:use pyo3::prelude::*; #[pyclass] struct NotHashable {} #[pymethods] impl NotHashable { #[classattr] const __hash__: Option<PyObject> = None; }
-
__lt__(<self>, object) -> object
-
__le__(<self>, object) -> object
-
__eq__(<self>, object) -> object
-
__ne__(<self>, object) -> object
-
__gt__(<self>, object) -> object
-
__ge__(<self>, object) -> object
The implementations of Python's "rich comparison" operators
<
,<=
,==
,!=
,>
and>=
respectively.Note that implementing any of these methods will cause Python not to generate a default
__hash__
implementation, so consider also implementing__hash__
.Return type
The return type will normally be `bool` or `PyResult`, however any Python object can be returned. -
__richcmp__(<self>, object, pyo3::basic::CompareOp) -> object
Implements Python comparison operations (
==
,!=
,<
,<=
,>
, and>=
) in a single method. TheCompareOp
argument indicates the comparison operation being performed. You can useCompareOp::matches
to adapt a Ruststd::cmp::Ordering
result to the requested comparison.This method cannot be implemented in combination with any of
__lt__
,__le__
,__eq__
,__ne__
,__gt__
, or__ge__
.Note that implementing
__richcmp__
will cause Python not to generate a default__hash__
implementation, so consider implementing__hash__
when implementing__richcmp__
.Return type
The return type will normally be `PyResult`, but any Python object can be returned. If you want to leave some operations unimplemented, you can return
py.NotImplemented()
for some of the operations:use pyo3::class::basic::CompareOp; use pyo3::prelude::*; #[pyclass] struct Number(i32); #[pymethods] impl Number { fn __richcmp__(&self, other: &Self, op: CompareOp, py: Python<'_>) -> PyObject { match op { CompareOp::Eq => (self.0 == other.0).into_py(py), CompareOp::Ne => (self.0 != other.0).into_py(py), _ => py.NotImplemented(), } } }
If the second argument
object
is not of the type specified in the signature, the generated code will automaticallyreturn NotImplemented
. -
__getattr__(<self>, object) -> object
-
__getattribute__(<self>, object) -> object
Differences between `__getattr__` and `__getattribute__`
As in Python, `__getattr__` is only called if the attribute is not found by normal attribute lookup. `__getattribute__`, on the other hand, is called for *every* attribute access. If it wants to access existing attributes on `self`, it needs to be very careful not to introduce infinite recursion, and use `baseclass.__getattribute__()`. -
__setattr__(<self>, value: object) -> ()
-
__delattr__(<self>, object) -> ()
Overrides attribute access.
-
__bool__(<self>) -> bool
Determines the "truthyness" of an object.
-
__call__(<self>, ...) -> object
- here, any argument list can be defined as for normalpymethods
Iterable objects
Iterators can be defined using these methods:
__iter__(<self>) -> object
__next__(<self>) -> Option<object> or IterNextOutput
(see details)
Returning None
from __next__
indicates that that there are no further items.
Example:
use pyo3::prelude::*;
#[pyclass]
struct MyIterator {
iter: Box<dyn Iterator<Item = PyObject> + Send>,
}
#[pymethods]
impl MyIterator {
fn __iter__(slf: PyRef<'_, Self>) -> PyRef<'_, Self> {
slf
}
fn __next__(mut slf: PyRefMut<'_, Self>) -> Option<PyObject> {
slf.iter.next()
}
}
In many cases you'll have a distinction between the type being iterated over
(i.e. the iterable) and the iterator it provides. In this case, the iterable
only needs to implement __iter__()
while the iterator must implement both
__iter__()
and __next__()
. For example:
use pyo3::prelude::*;
#[pyclass]
struct Iter {
inner: std::vec::IntoIter<usize>,
}
#[pymethods]
impl Iter {
fn __iter__(slf: PyRef<'_, Self>) -> PyRef<'_, Self> {
slf
}
fn __next__(mut slf: PyRefMut<'_, Self>) -> Option<usize> {
slf.inner.next()
}
}
#[pyclass]
struct Container {
iter: Vec<usize>,
}
#[pymethods]
impl Container {
fn __iter__(slf: PyRef<'_, Self>) -> PyResult<Py<Iter>> {
let iter = Iter {
inner: slf.iter.clone().into_iter(),
};
Py::new(slf.py(), iter)
}
}
Python::with_gil(|py| {
let container = Container { iter: vec![1, 2, 3, 4] };
let inst = pyo3::Py::new(py, container).unwrap();
pyo3::py_run!(py, inst, "assert list(inst) == [1, 2, 3, 4]");
pyo3::py_run!(py, inst, "assert list(iter(iter(inst))) == [1, 2, 3, 4]");
});
For more details on Python's iteration protocols, check out the "Iterator Types" section of the library documentation.
Returning a value from iteration
This guide has so far shown how to use Option<T>
to implement yielding values
during iteration. In Python a generator can also return a value. To express
this in Rust, PyO3 provides the IterNextOutput
enum to both Yield
values
and Return
a final value - see its docs for further details and an example.
Awaitable objects
__await__(<self>) -> object
__aiter__(<self>) -> object
__anext__(<self>) -> Option<object> or IterANextOutput
Mapping & Sequence types
The magic methods in this section can be used to implement Python container types. They are two main categories of container in Python: "mappings" such as dict
, with arbitrary keys, and "sequences" such as list
and tuple
, with integer keys.
The Python C-API which PyO3 is built upon has separate "slots" for sequences and mappings. When writing a class
in pure Python, there is no such distinction in the implementation - a __getitem__
implementation will fill the slots for both the mapping and sequence forms, for example.
By default PyO3 reproduces the Python behaviour of filling both mapping and sequence slots. This makes sense for the "simple" case which matches Python, and also for sequences, where the mapping slot is used anyway to implement slice indexing.
Mapping types usually will not want the sequence slots filled. Having them filled will lead to outcomes which may be unwanted, such as:
- The mapping type will successfully cast to
PySequence
. This may lead to consumers of the type handling it incorrectly. - Python provides a default implementation of
__iter__
for sequences, which calls__getitem__
with consecutive positive integers starting from 0 until anIndexError
is returned. Unless the mapping only contains consecutive positive integer keys, this__iter__
implementation will likely not be the intended behavior.
Use the #[pyclass(mapping)]
annotation to instruct PyO3 to only fill the mapping slots, leaving the sequence ones empty. This will apply to __getitem__
, __setitem__
, and __delitem__
.
Use the #[pyclass(sequence)]
annotation to instruct PyO3 to fill the sq_length
slot instead of the mp_length
slot for __len__
. This will help libraries such as numpy
recognise the class as a sequence, however will also cause CPython to automatically add the sequence length to any negative indices before passing them to __getitem__
. (__getitem__
, __setitem__
and __delitem__
mapping slots are still used for sequences, for slice operations.)
-
__len__(<self>) -> usize
Implements the built-in function
len()
. -
__contains__(<self>, object) -> bool
Implements membership test operators. Should return true if
item
is inself
, false otherwise. For objects that don’t define__contains__()
, the membership test simply traverses the sequence until it finds a match.Disabling Python's default contains
By default, all
#[pyclass]
types with an__iter__
method support a default implementation of thein
operator. Types which do not want this can override this by setting__contains__
toNone
. This is the same mechanism as for a pure-Python class. This is done like so:use pyo3::prelude::*; #[pyclass] struct NoContains {} #[pymethods] impl NoContains { #[classattr] const __contains__: Option<PyObject> = None; }
-
__getitem__(<self>, object) -> object
Implements retrieval of the
self[a]
element.Note: Negative integer indexes are not handled specially by PyO3. However, for classes with
#[pyclass(sequence)]
, when a negative index is accessed viaPySequence::get_item
, the underlying C API already adjusts the index to be positive. -
__setitem__(<self>, object, object) -> ()
Implements assignment to the
self[a]
element. Should only be implemented if elements can be replaced.Same behavior regarding negative indices as for
__getitem__
. -
__delitem__(<self>, object) -> ()
Implements deletion of the
self[a]
element. Should only be implemented if elements can be deleted.Same behavior regarding negative indices as for
__getitem__
.
-
fn __concat__(&self, other: impl FromPyObject) -> PyResult<impl ToPyObject>
Concatenates two sequences. Used by the
+
operator, after trying the numeric addition via the__add__
and__radd__
methods. -
fn __repeat__(&self, count: isize) -> PyResult<impl ToPyObject>
Repeats the sequence
count
times. Used by the*
operator, after trying the numeric multiplication via the__mul__
and__rmul__
methods. -
fn __inplace_concat__(&self, other: impl FromPyObject) -> PyResult<impl ToPyObject>
Concatenates two sequences. Used by the
+=
operator, after trying the numeric addition via the__iadd__
method. -
fn __inplace_repeat__(&self, count: isize) -> PyResult<impl ToPyObject>
Concatenates two sequences. Used by the
*=
operator, after trying the numeric multiplication via the__imul__
method.
Descriptors
__get__(<self>, object, object) -> object
__set__(<self>, object, object) -> ()
__delete__(<self>, object) -> ()
Numeric types
Binary arithmetic operations (+
, -
, *
, @
, /
, //
, %
, divmod()
,
pow()
and **
, <<
, >>
, &
, ^
, and |
) and their reflected versions:
(If the object
is not of the type specified in the signature, the generated code
will automatically return NotImplemented
.)
__add__(<self>, object) -> object
__radd__(<self>, object) -> object
__sub__(<self>, object) -> object
__rsub__(<self>, object) -> object
__mul__(<self>, object) -> object
__rmul__(<self>, object) -> object
__matmul__(<self>, object) -> object
__rmatmul__(<self>, object) -> object
__floordiv__(<self>, object) -> object
__rfloordiv__(<self>, object) -> object
__truediv__(<self>, object) -> object
__rtruediv__(<self>, object) -> object
__divmod__(<self>, object) -> object
__rdivmod__(<self>, object) -> object
__mod__(<self>, object) -> object
__rmod__(<self>, object) -> object
__lshift__(<self>, object) -> object
__rlshift__(<self>, object) -> object
__rshift__(<self>, object) -> object
__rrshift__(<self>, object) -> object
__and__(<self>, object) -> object
__rand__(<self>, object) -> object
__xor__(<self>, object) -> object
__rxor__(<self>, object) -> object
__or__(<self>, object) -> object
__ror__(<self>, object) -> object
__pow__(<self>, object, object) -> object
__rpow__(<self>, object, object) -> object
In-place assignment operations (+=
, -=
, *=
, @=
, /=
, //=
, %=
,
**=
, <<=
, >>=
, &=
, ^=
, |=
):
__iadd__(<self>, object) -> ()
__isub__(<self>, object) -> ()
__imul__(<self>, object) -> ()
__imatmul__(<self>, object) -> ()
__itruediv__(<self>, object) -> ()
__ifloordiv__(<self>, object) -> ()
__imod__(<self>, object) -> ()
__ipow__(<self>, object, object) -> ()
__ilshift__(<self>, object) -> ()
__irshift__(<self>, object) -> ()
__iand__(<self>, object) -> ()
__ixor__(<self>, object) -> ()
__ior__(<self>, object) -> ()
Unary operations (-
, +
, abs()
and ~
):
__pos__(<self>) -> object
__neg__(<self>) -> object
__abs__(<self>) -> object
__invert__(<self>) -> object
Coercions:
__index__(<self>) -> object (int)
__int__(<self>) -> object (int)
__float__(<self>) -> object (float)
Buffer objects
__getbuffer__(<self>, *mut ffi::Py_buffer, flags) -> ()
__releasebuffer__(<self>, *mut ffi::Py_buffer) -> ()
Errors returned from__releasebuffer__
will be sent tosys.unraiseablehook
. It is strongly advised to never return an error from__releasebuffer__
, and if it really is necessary, to make best effort to perform any required freeing operations before returning.__releasebuffer__
will not be called a second time; anything not freed will be leaked.
Garbage Collector Integration
If your type owns references to other Python objects, you will need to integrate
with Python's garbage collector so that the GC is aware of those references. To
do this, implement the two methods __traverse__
and __clear__
. These
correspond to the slots tp_traverse
and tp_clear
in the Python C API.
__traverse__
must call visit.call()
for each reference to another Python
object. __clear__
must clear out any mutable references to other Python
objects (thus breaking reference cycles). Immutable references do not have to be
cleared, as every cycle must contain at least one mutable reference.
__traverse__(<self>, pyo3::class::gc::PyVisit<'_>) -> Result<(), pyo3::class::gc::PyTraverseError>
__clear__(<self>) -> ()
Example:
use pyo3::prelude::*;
use pyo3::PyTraverseError;
use pyo3::gc::PyVisit;
#[pyclass]
struct ClassWithGCSupport {
obj: Option<PyObject>,
}
#[pymethods]
impl ClassWithGCSupport {
fn __traverse__(&self, visit: PyVisit<'_>) -> Result<(), PyTraverseError> {
if let Some(obj) = &self.obj {
visit.call(obj)?
}
Ok(())
}
fn __clear__(&mut self) {
// Clear reference, this decrements ref counter.
self.obj = None;
}
}
Usually, an implementation of __traverse__
should do nothing but calls to visit.call
.
Most importantly, safe access to the GIL is prohibited inside implementations of __traverse__
,
i.e. Python::with_gil
will panic.
Note: these methods are part of the C API, PyPy does not necessarily honor them. If you are building for PyPy you should measure memory consumption to make sure you do not have runaway memory growth. See this issue on the PyPy bug tracker.
Basic object customization
Recall the Number
class from the previous chapter:
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
#[new]
fn new(value: i32) -> Self {
Self(value)
}
}
#[pymodule]
fn my_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<Number>()?;
Ok(())
}
At this point Python code can import the module, access the class and create class instances - but nothing else.
from my_module import Number
n = Number(5)
print(n)
<builtins.Number object at 0x000002B4D185D7D0>
String representations
It can't even print an user-readable representation of itself! We can fix that by defining the
__repr__
and __str__
methods inside a #[pymethods]
block. We do this by accessing the value
contained inside Number
.
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
// For `__repr__` we want to return a string that Python code could use to recreate
// the `Number`, like `Number(5)` for example.
fn __repr__(&self) -> String {
// We use the `format!` macro to create a string. Its first argument is a
// format string, followed by any number of parameters which replace the
// `{}`'s in the format string.
//
// 👇 Tuple field access in Rust uses a dot
format!("Number({})", self.0)
}
// `__str__` is generally used to create an "informal" representation, so we
// just forward to `i32`'s `ToString` trait implementation to print a bare number.
fn __str__(&self) -> String {
self.0.to_string()
}
}
Accessing the class name
In the __repr__
, we used a hard-coded class name. This is sometimes not ideal,
because if the class is subclassed in Python, we would like the repr to reflect
the subclass name. This is typically done in Python code by accessing
self.__class__.__name__
. In order to be able to access the Python type information
and the Rust struct, we need to use a Bound
as the self
argument.
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
fn __repr__(slf: &Bound<'_, Self>) -> PyResult<String> {
// This is the equivalent of `self.__class__.__name__` in Python.
let class_name: String = slf.get_type().qualname()?;
// To access fields of the Rust struct, we need to borrow the `PyCell`.
Ok(format!("{}({})", class_name, slf.borrow().0))
}
}
Hashing
Let's also implement hashing. We'll just hash the i32
. For that we need a Hasher
. The one
provided by std
is DefaultHasher
, which uses the SipHash algorithm.
use std::collections::hash_map::DefaultHasher;
// Required to call the `.hash` and `.finish` methods, which are defined on traits.
use std::hash::{Hash, Hasher};
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
fn __hash__(&self) -> u64 {
let mut hasher = DefaultHasher::new();
self.0.hash(&mut hasher);
hasher.finish()
}
}
Note: When implementing
__hash__
and comparisons, it is important that the following property holds:k1 == k2 -> hash(k1) == hash(k2)
In other words, if two keys are equal, their hashes must also be equal. In addition you must take care that your classes' hash doesn't change during its lifetime. In this tutorial we do that by not letting Python code change our
Number
class. In other words, it is immutable.By default, all
#[pyclass]
types have a default hash implementation from Python. Types which should not be hashable can override this by setting__hash__
to None. This is the same mechanism as for a pure-Python class. This is done like so:use pyo3::prelude::*; #[pyclass] struct NotHashable {} #[pymethods] impl NotHashable { #[classattr] const __hash__: Option<Py<PyAny>> = None; }
Comparisons
PyO3 supports the usual magic comparison methods available in Python such as __eq__
, __lt__
and so on. It is also possible to support all six operations at once with __richcmp__
.
This method will be called with a value of CompareOp
depending on the operation.
use pyo3::class::basic::CompareOp;
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
fn __richcmp__(&self, other: &Self, op: CompareOp) -> PyResult<bool> {
match op {
CompareOp::Lt => Ok(self.0 < other.0),
CompareOp::Le => Ok(self.0 <= other.0),
CompareOp::Eq => Ok(self.0 == other.0),
CompareOp::Ne => Ok(self.0 != other.0),
CompareOp::Gt => Ok(self.0 > other.0),
CompareOp::Ge => Ok(self.0 >= other.0),
}
}
}
If you obtain the result by comparing two Rust values, as in this example, you
can take a shortcut using CompareOp::matches
:
use pyo3::class::basic::CompareOp;
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
fn __richcmp__(&self, other: &Self, op: CompareOp) -> bool {
op.matches(self.0.cmp(&other.0))
}
}
It checks that the std::cmp::Ordering
obtained from Rust's Ord
matches
the given CompareOp
.
Alternatively, you can implement just equality using __eq__
:
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
fn __eq__(&self, other: &Self) -> bool {
self.0 == other.0
}
}
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let x = &Bound::new(py, Number(4))?;
let y = &Bound::new(py, Number(4))?;
assert!(x.eq(y)?);
assert!(!x.ne(y)?);
Ok(())
})
}
Truthyness
We'll consider Number
to be True
if it is nonzero:
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
fn __bool__(&self) -> bool {
self.0 != 0
}
}
Final code
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
use pyo3::prelude::*;
use pyo3::class::basic::CompareOp;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
#[new]
fn new(value: i32) -> Self {
Self(value)
}
fn __repr__(slf: &Bound<'_, Self>) -> PyResult<String> {
let class_name: String = slf.get_type().qualname()?;
Ok(format!("{}({})", class_name, slf.borrow().0))
}
fn __str__(&self) -> String {
self.0.to_string()
}
fn __hash__(&self) -> u64 {
let mut hasher = DefaultHasher::new();
self.0.hash(&mut hasher);
hasher.finish()
}
fn __richcmp__(&self, other: &Self, op: CompareOp) -> PyResult<bool> {
match op {
CompareOp::Lt => Ok(self.0 < other.0),
CompareOp::Le => Ok(self.0 <= other.0),
CompareOp::Eq => Ok(self.0 == other.0),
CompareOp::Ne => Ok(self.0 != other.0),
CompareOp::Gt => Ok(self.0 > other.0),
CompareOp::Ge => Ok(self.0 >= other.0),
}
}
fn __bool__(&self) -> bool {
self.0 != 0
}
}
#[pymodule]
fn my_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<Number>()?;
Ok(())
}
Emulating numeric types
At this point we have a Number
class that we can't actually do any math on!
Before proceeding, we should think about how we want to handle overflows. There are three obvious solutions:
- We can have infinite precision just like Python's
int
. However that would be quite boring - we'd be reinventing the wheel. - We can raise exceptions whenever
Number
overflows, but that makes the API painful to use. - We can wrap around the boundary of
i32
. This is the approach we'll take here. To do that we'll just forward toi32
'swrapping_*
methods.
Fixing our constructor
Let's address the first overflow, in Number
's constructor:
from my_module import Number
n = Number(1 << 1337)
Traceback (most recent call last):
File "example.py", line 3, in <module>
n = Number(1 << 1337)
OverflowError: Python int too large to convert to C long
Instead of relying on the default FromPyObject
extraction to parse arguments, we can specify our
own extraction function, using the #[pyo3(from_py_with = "...")]
attribute. Unfortunately PyO3
doesn't provide a way to wrap Python integers out of the box, but we can do a Python call to mask it
and cast it to an i32
.
#![allow(dead_code)]
use pyo3::prelude::*;
fn wrap(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
let val = obj.call_method1("__and__", (0xFFFFFFFF_u32,))?;
let val: u32 = val.extract()?;
// 👇 This intentionally overflows!
Ok(val as i32)
}
We also add documentation, via ///
comments, which are visible to Python users.
#![allow(dead_code)]
use pyo3::prelude::*;
fn wrap(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
let val = obj.call_method1("__and__", (0xFFFFFFFF_u32,))?;
let val: u32 = val.extract()?;
Ok(val as i32)
}
/// Did you ever hear the tragedy of Darth Signed The Overfloweth? I thought not.
/// It's not a story C would tell you. It's a Rust legend.
#[pyclass(module = "my_module")]
struct Number(i32);
#[pymethods]
impl Number {
#[new]
fn new(#[pyo3(from_py_with = "wrap")] value: i32) -> Self {
Self(value)
}
}
With that out of the way, let's implement some operators:
use pyo3::exceptions::{PyZeroDivisionError, PyValueError};
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
fn __add__(&self, other: &Self) -> Self {
Self(self.0.wrapping_add(other.0))
}
fn __sub__(&self, other: &Self) -> Self {
Self(self.0.wrapping_sub(other.0))
}
fn __mul__(&self, other: &Self) -> Self {
Self(self.0.wrapping_mul(other.0))
}
fn __truediv__(&self, other: &Self) -> PyResult<Self> {
match self.0.checked_div(other.0) {
Some(i) => Ok(Self(i)),
None => Err(PyZeroDivisionError::new_err("division by zero")),
}
}
fn __floordiv__(&self, other: &Self) -> PyResult<Self> {
match self.0.checked_div(other.0) {
Some(i) => Ok(Self(i)),
None => Err(PyZeroDivisionError::new_err("division by zero")),
}
}
fn __rshift__(&self, other: &Self) -> PyResult<Self> {
match other.0.try_into() {
Ok(rhs) => Ok(Self(self.0.wrapping_shr(rhs))),
Err(_) => Err(PyValueError::new_err("negative shift count")),
}
}
fn __lshift__(&self, other: &Self) -> PyResult<Self> {
match other.0.try_into() {
Ok(rhs) => Ok(Self(self.0.wrapping_shl(rhs))),
Err(_) => Err(PyValueError::new_err("negative shift count")),
}
}
}
Unary arithmetic operations
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
#[pymethods]
impl Number {
fn __pos__(slf: PyRef<'_, Self>) -> PyRef<'_, Self> {
slf
}
fn __neg__(&self) -> Self {
Self(-self.0)
}
fn __abs__(&self) -> Self {
Self(self.0.abs())
}
fn __invert__(&self) -> Self {
Self(!self.0)
}
}
Support for the complex()
, int()
and float()
built-in functions.
use pyo3::prelude::*;
#[pyclass]
struct Number(i32);
use pyo3::types::PyComplex;
#[pymethods]
impl Number {
fn __int__(&self) -> i32 {
self.0
}
fn __float__(&self) -> f64 {
self.0 as f64
}
fn __complex__<'py>(&self, py: Python<'py>) -> Bound<'py, PyComplex> {
PyComplex::from_doubles_bound(py, self.0 as f64, 0.0)
}
}
We do not implement the in-place operations like __iadd__
because we do not wish to mutate Number
.
Similarly we're not interested in supporting operations with different types, so we do not implement
the reflected operations like __radd__
either.
Now Python can use our Number
class:
from my_module import Number
def hash_djb2(s: str):
'''
A version of Daniel J. Bernstein's djb2 string hashing algorithm
Like many hashing algorithms, it relies on integer wrapping.
'''
n = Number(0)
five = Number(5)
for x in s:
n = Number(ord(x)) + ((n << five) - n)
return n
assert hash_djb2('l50_50') == Number(-1152549421)
Final code
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
use pyo3::exceptions::{PyValueError, PyZeroDivisionError};
use pyo3::prelude::*;
use pyo3::class::basic::CompareOp;
use pyo3::types::PyComplex;
fn wrap(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
let val = obj.call_method1("__and__", (0xFFFFFFFF_u32,))?;
let val: u32 = val.extract()?;
Ok(val as i32)
}
/// Did you ever hear the tragedy of Darth Signed The Overfloweth? I thought not.
/// It's not a story C would tell you. It's a Rust legend.
#[pyclass(module = "my_module")]
struct Number(i32);
#[pymethods]
impl Number {
#[new]
fn new(#[pyo3(from_py_with = "wrap")] value: i32) -> Self {
Self(value)
}
fn __repr__(slf: &Bound<'_, Self>) -> PyResult<String> {
// Get the class name dynamically in case `Number` is subclassed
let class_name: String = slf.get_type().qualname()?;
Ok(format!("{}({})", class_name, slf.borrow().0))
}
fn __str__(&self) -> String {
self.0.to_string()
}
fn __hash__(&self) -> u64 {
let mut hasher = DefaultHasher::new();
self.0.hash(&mut hasher);
hasher.finish()
}
fn __richcmp__(&self, other: &Self, op: CompareOp) -> PyResult<bool> {
match op {
CompareOp::Lt => Ok(self.0 < other.0),
CompareOp::Le => Ok(self.0 <= other.0),
CompareOp::Eq => Ok(self.0 == other.0),
CompareOp::Ne => Ok(self.0 != other.0),
CompareOp::Gt => Ok(self.0 > other.0),
CompareOp::Ge => Ok(self.0 >= other.0),
}
}
fn __bool__(&self) -> bool {
self.0 != 0
}
fn __add__(&self, other: &Self) -> Self {
Self(self.0.wrapping_add(other.0))
}
fn __sub__(&self, other: &Self) -> Self {
Self(self.0.wrapping_sub(other.0))
}
fn __mul__(&self, other: &Self) -> Self {
Self(self.0.wrapping_mul(other.0))
}
fn __truediv__(&self, other: &Self) -> PyResult<Self> {
match self.0.checked_div(other.0) {
Some(i) => Ok(Self(i)),
None => Err(PyZeroDivisionError::new_err("division by zero")),
}
}
fn __floordiv__(&self, other: &Self) -> PyResult<Self> {
match self.0.checked_div(other.0) {
Some(i) => Ok(Self(i)),
None => Err(PyZeroDivisionError::new_err("division by zero")),
}
}
fn __rshift__(&self, other: &Self) -> PyResult<Self> {
match other.0.try_into() {
Ok(rhs) => Ok(Self(self.0.wrapping_shr(rhs))),
Err(_) => Err(PyValueError::new_err("negative shift count")),
}
}
fn __lshift__(&self, other: &Self) -> PyResult<Self> {
match other.0.try_into() {
Ok(rhs) => Ok(Self(self.0.wrapping_shl(rhs))),
Err(_) => Err(PyValueError::new_err("negative shift count")),
}
}
fn __xor__(&self, other: &Self) -> Self {
Self(self.0 ^ other.0)
}
fn __or__(&self, other: &Self) -> Self {
Self(self.0 | other.0)
}
fn __and__(&self, other: &Self) -> Self {
Self(self.0 & other.0)
}
fn __int__(&self) -> i32 {
self.0
}
fn __float__(&self) -> f64 {
self.0 as f64
}
fn __complex__<'py>(&self, py: Python<'py>) -> Bound<'py, PyComplex> {
PyComplex::from_doubles_bound(py, self.0 as f64, 0.0)
}
}
#[pymodule]
fn my_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<Number>()?;
Ok(())
}
const SCRIPT: &'static str = r#"
def hash_djb2(s: str):
n = Number(0)
five = Number(5)
for x in s:
n = Number(ord(x)) + ((n << five) - n)
return n
assert hash_djb2('l50_50') == Number(-1152549421)
assert hash_djb2('logo') == Number(3327403)
assert hash_djb2('horizon') == Number(1097468315)
assert Number(2) + Number(2) == Number(4)
assert Number(2) + Number(2) != Number(5)
assert Number(13) - Number(7) == Number(6)
assert Number(13) - Number(-7) == Number(20)
assert Number(13) / Number(7) == Number(1)
assert Number(13) // Number(7) == Number(1)
assert Number(13) * Number(7) == Number(13*7)
assert Number(13) > Number(7)
assert Number(13) < Number(20)
assert Number(13) == Number(13)
assert Number(13) >= Number(7)
assert Number(13) <= Number(20)
assert Number(13) == Number(13)
assert (True if Number(1) else False)
assert (False if Number(0) else True)
assert int(Number(13)) == 13
assert float(Number(13)) == 13
assert Number.__doc__ == "Did you ever hear the tragedy of Darth Signed The Overfloweth? I thought not.\nIt's not a story C would tell you. It's a Rust legend."
assert Number(12345234523452) == Number(1498514748)
try:
import inspect
assert inspect.signature(Number).__str__() == '(value)'
except ValueError:
# Not supported with `abi3` before Python 3.10
pass
assert Number(1337).__str__() == '1337'
assert Number(1337).__repr__() == 'Number(1337)'
"#;
use pyo3::PyTypeInfo;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let globals = PyModule::import_bound(py, "__main__")?.dict();
globals.set_item("Number", Number::type_object_bound(py))?;
py.run_bound(SCRIPT, Some(&globals), None)?;
Ok(())
})
}
Appendix: Writing some unsafe code
At the beginning of this chapter we said that PyO3 doesn't provide a way to wrap Python integers out of the box but that's a half truth. There's not a PyO3 API for it, but there's a Python C API function that does:
unsigned long PyLong_AsUnsignedLongMask(PyObject *obj)
We can call this function from Rust by using pyo3::ffi::PyLong_AsUnsignedLongMask
. This is an unsafe
function, which means we have to use an unsafe block to call it and take responsibility for upholding
the contracts of this function. Let's review those contracts:
- The GIL must be held. If it's not, calling this function causes a data race.
- The pointer must be valid, i.e. it must be properly aligned and point to a valid Python object.
Let's create that helper function. The signature has to be fn(&Bound<'_, PyAny>) -> PyResult<T>
.
&Bound<'_, PyAny>
represents a checked borrowed reference, so the pointer derived from it is valid (and not null).- Whenever we have borrowed references to Python objects in scope, it is guaranteed that the GIL is held. This reference is also where we can get a
Python
token to use in our call toPyErr::take
.
#![allow(dead_code)]
use std::os::raw::c_ulong;
use pyo3::prelude::*;
use pyo3::ffi;
fn wrap(obj: &Bound<'_, PyAny>) -> Result<i32, PyErr> {
let py: Python<'_> = obj.py();
unsafe {
let ptr = obj.as_ptr();
let ret: c_ulong = ffi::PyLong_AsUnsignedLongMask(ptr);
if ret == c_ulong::MAX {
if let Some(err) = PyErr::take(py) {
return Err(err);
}
}
Ok(ret as i32)
}
}
Emulating callable objects
Classes can be callable if they have a #[pymethod]
named __call__
.
This allows instances of a class to behave similar to functions.
This method's signature must look like __call__(<self>, ...) -> object
- here,
any argument list can be defined as for normal pymethods
Example: Implementing a call counter
The following pyclass is a basic decorator - its constructor takes a Python object as argument and calls that object when called. An equivalent Python implementation is linked at the end.
An example crate containing this pyclass can be found here
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyTuple};
use std::cell::Cell;
/// A function decorator that keeps track how often it is called.
///
/// It otherwise doesn't do anything special.
#[pyclass(name = "Counter")]
pub struct PyCounter {
// Keeps track of how many calls have gone through.
//
// See the discussion at the end for why `Cell` is used.
count: Cell<u64>,
// This is the actual function being wrapped.
wraps: Py<PyAny>,
}
#[pymethods]
impl PyCounter {
// Note that we don't validate whether `wraps` is actually callable.
//
// While we could use `PyAny::is_callable` for that, it has some flaws:
// 1. It doesn't guarantee the object can actually be called successfully
// 2. We still need to handle any exceptions that the function might raise
#[new]
fn __new__(wraps: Py<PyAny>) -> Self {
PyCounter {
count: Cell::new(0),
wraps,
}
}
#[getter]
fn count(&self) -> u64 {
self.count.get()
}
#[pyo3(signature = (*args, **kwargs))]
fn __call__(
&self,
py: Python<'_>,
args: &Bound<'_, PyTuple>,
kwargs: Option<&Bound<'_, PyDict>>,
) -> PyResult<Py<PyAny>> {
let old_count = self.count.get();
let new_count = old_count + 1;
self.count.set(new_count);
let name = self.wraps.getattr(py, "__name__")?;
println!("{} has been called {} time(s).", name, new_count);
// After doing something, we finally forward the call to the wrapped function
let ret = self.wraps.call_bound(py, args, kwargs)?;
// We could do something with the return value of
// the function before returning it
Ok(ret)
}
}
#[pymodule]
pub fn decorator(module: &Bound<'_, PyModule>) -> PyResult<()> {
module.add_class::<PyCounter>()?;
Ok(())
}
Python code:
from decorator import Counter
@Counter
def say_hello():
print("hello")
say_hello()
say_hello()
say_hello()
say_hello()
assert say_hello.count == 4
Output:
say_hello has been called 1 time(s).
hello
say_hello has been called 2 time(s).
hello
say_hello has been called 3 time(s).
hello
say_hello has been called 4 time(s).
hello
Pure Python implementation
A Python implementation of this looks similar to the Rust version:
class Counter:
def __init__(self, wraps):
self.count = 0
self.wraps = wraps
def __call__(self, *args, **kwargs):
self.count += 1
print(f"{self.wraps.__name__} has been called {self.count} time(s)")
self.wraps(*args, **kwargs)
Note that it can also be implemented as a higher order function:
def Counter(wraps):
count = 0
def call(*args, **kwargs):
nonlocal count
count += 1
print(f"{wraps.__name__} has been called {count} time(s)")
return wraps(*args, **kwargs)
return call
What is the Cell
for?
A previous implementation used a normal u64
, which meant it required a &mut self
receiver to update the count:
#[pyo3(signature = (*args, **kwargs))]
fn __call__(
&mut self,
py: Python<'_>,
args: &Bound<'_, PyTuple>,
kwargs: Option<&Bound<'_, PyDict>>,
) -> PyResult<Py<PyAny>> {
self.count += 1;
let name = self.wraps.getattr(py, "__name__")?;
println!("{} has been called {} time(s).", name, self.count);
// After doing something, we finally forward the call to the wrapped function
let ret = self.wraps.call(py, args, kwargs)?;
// We could do something with the return value of
// the function before returning it
Ok(ret)
}
The problem with this is that the &mut self
receiver means PyO3 has to borrow it exclusively,
and hold this borrow across theself.wraps.call(py, args, kwargs)
call. This call returns control to the user's Python code
which is free to call arbitrary things, including the decorated function. If that happens PyO3 is unable to create a second unique borrow and will be forced to raise an exception.
As a result, something innocent like this will raise an exception:
@Counter
def say_hello():
if say_hello.count < 2:
print(f"hello from decorator")
say_hello()
# RuntimeError: Already borrowed
The implementation in this chapter fixes that by never borrowing exclusively; all the methods take &self
as receivers, of which multiple may exist simultaneously. This requires a shared counter and the easiest way to do that is to use Cell
, so that's what is used here.
This shows the dangers of running arbitrary Python code - note that "running arbitrary Python code" can be far more subtle than the example above:
- Python's asynchronous executor may park the current thread in the middle of Python code, even in Python code that you control, and let other Python code run.
- Dropping arbitrary Python objects may invoke destructors defined in Python (
__del__
methods). - Calling Python's C-api (most PyO3 apis call C-api functions internally) may raise exceptions, which may allow Python code in signal handlers to run.
This is especially important if you are writing unsafe code; Python code must never be able to cause undefined behavior. You must ensure that your Rust code is in a consistent state before doing any of the above things.
Calling Python in Rust code
This chapter of the guide documents some ways to interact with Python code from Rust.
Below is an introduction to the 'py
lifetime and some general remarks about how PyO3's API reasons about Python code.
The subchapters also cover the following topics:
- Python object types available in PyO3's API
- How to work with Python exceptions
- How to call Python functions
- How to execute existing Python code
The 'py
lifetime
To safely interact with the Python interpreter a Rust thread must have a corresponding Python thread state and hold the Global Interpreter Lock (GIL). PyO3 has a Python<'py>
token that is used to prove that these conditions
are met. Its lifetime 'py
is a central part of PyO3's API.
The Python<'py>
token serves three purposes:
- It provides global APIs for the Python interpreter, such as [
py.eval_bound()
][eval] and [py.import_bound()
][import]. - It can be passed to functions that require a proof of holding the GIL, such as [
Py::clone_ref
][clone_ref]. - Its lifetime
'py
is used to bind many of PyO3's types to the Python interpreter, such as [Bound<'py, T>
][Bound].
PyO3's types that are bound to the 'py
lifetime, for example Bound<'py, T>
, all contain a Python<'py>
token. This means they have full access to the Python interpreter and offer a complete API for interacting with Python objects.
Consult PyO3's API documentation to learn how to acquire one of these tokens.
The Global Interpreter Lock
Concurrent programming in Python is aided by the Global Interpreter Lock (GIL), which ensures that only one Python thread can use the Python interpreter and its API at the same time. This allows it to be used to synchronize code. See the pyo3::sync
module for synchronization tools PyO3 offers that are based on the GIL's guarantees.
Non-Python operations (system calls and native Rust code) can unlock the GIL. See the section on parallelism for how to do that using PyO3's API.
Python's memory model
Python's memory model differs from Rust's memory model in two key ways:
- There is no concept of ownership; all Python objects are shared and usually implemented via reference counting
- There is no concept of exclusive (
&mut
) references; any reference can mutate a Python object
PyO3's API reflects this by providing smart pointer types, Py<T>
, Bound<'py, T>
, and (the very rarely used) Borrowed<'a, 'py, T>
. These smart pointers all use Python reference counting. See the subchapter on types for more detail on these types.
Because of the lack of exclusive &mut
references, PyO3's APIs for Python objects, for example [PyListMethods::append
], use shared references. This is safe because Python objects have internal mechanisms to prevent data races (as of time of writing, the Python GIL).
Python object types
PyO3 offers two main sets of types to interact with Python objects. This section of the guide expands into detail about these types and how to choose which to use.
The first set of types is are the smart pointers which all Python objects are wrapped in. These are Py<T>
, Bound<'py, T>
, and Borrowed<'a, 'py, T>
. The first section below expands on each of these in detail and why there are three of them.
The second set of types are types which fill in the generic parameter T
of the smart pointers. The most common is PyAny
, which represents any Python object (similar to Python's typing.Any
). There are also concrete types for many Python built-in types, such as PyList
, PyDict
, and PyTuple
. User defined #[pyclass]
types also fit this category. The second section below expands on how to use these types.
Before PyO3 0.21, PyO3's main API to interact with Python objects was a deprecated API known as the "GIL Refs" API, containing reference types such as &PyAny
, &PyList
, and &PyCell<T>
for user-defined #[pyclass]
types. The third section below details this deprecated API.
PyO3's smart pointers
PyO3's API offers three generic smart pointers: Py<T>
, Bound<'py, T>
and Borrowed<'a, 'py, T>
. For each of these the type parameter T
will be filled by a concrete Python type. For example, a Python list object can be represented by Py<PyList>
, Bound<'py, PyList>
, and Borrowed<'a, 'py, PyList>
.
These smart pointers behave differently due to their lifetime parameters. Py<T>
has no lifetime parameters, Bound<'py, T>
has the 'py
lifetime as a parameter, and Borrowed<'a, 'py, T>
has the 'py
lifetime plus an additional lifetime 'a
to denote the lifetime it is borrowing data for. (You can read more about these lifetimes in the subsections below).
Python objects are reference counted, like std::sync::Arc
. A major reason for these smart pointers is to bring Python's reference counting to a Rust API.
The recommendation of when to use each of these smart pointers is as follows:
- Use
Bound<'py, T>
for as much as possible, as it offers the most efficient and complete API. - Use
Py<T>
mostly just for storage inside Ruststruct
s which do not want to or can't add a lifetime parameter forBound<'py, T>
. Borrowed<'a, 'py, T>
is almost never used. It is occasionally present at the boundary between Rust and the Python interpreter, for example when borrowing data from Python tuples (which is safe because they are immutable).
The sections below also explain these smart pointers in a little more detail.
Py<T>
(and PyObject
)
Py<T>
is the foundational smart pointer in PyO3's API. The type parameter T
denotes the type of the Python object. Very frequently this is PyAny
, meaning any Python object. This is so common that Py<PyAny>
has a type alias PyObject
.
Because Py<T>
is not bound to the 'py
lifetime, it is the type to use when storing a Python object inside a Rust struct
or enum
which do not want to have a lifetime parameter. In particular, #[pyclass]
types are not permitted to have a lifetime, so Py<T>
is the correct type to store Python objects inside them.
The lack of binding to the 'py
lifetime also carries drawbacks:
- Almost all methods on
Py<T>
require aPython<'py>
token as the first argument - Other functionality, such as
Drop
, needs to check at runtime for attachment to the Python GIL, at a small performance cost
Because of the drawbacks Bound<'py, T>
is preferred for many of PyO3's APIs. In particular, Bound<'py, T>
is the better for function arguments.
To convert a Py<T>
into a Bound<'py, T>
, the Py::bind
and Py::into_bound
methods are available. Bound<'py, T>
can be converted back into Py<T>
using Bound::unbind
.
Bound<'py, T>
Bound<'py, T>
is the counterpart to Py<T>
which is also bound to the 'py
lifetime. It can be thought of as equivalent to the Rust tuple (Python<'py>, Py<T>)
.
By having the binding to the 'py
lifetime, Bound<'py, T>
can offer the complete PyO3 API at maximum efficiency. This means that in almost all cases where Py<T>
is not necessary for lifetime reasons, Bound<'py, T>
should be used.
Bound<'py, T>
engages in Python reference counting. This means that Bound<'py, T>
owns a Python object. Rust code which just wants to borrow a Python object should use a shared reference &Bound<'py, T>
. Just like std::sync::Arc
, using .clone()
and drop()
will cheaply increment and decrement the reference count of the object (just in this case, the reference counting is implemented by the Python interpreter itself).
To give an example of how Bound<'py, T>
is PyO3's primary API type, consider the following Python code:
def example():
x = list() # create a Python list
x.append(1) # append the integer 1 to it
y = x # create a second reference to the list
del x # delete the original reference
Using PyO3's API, and in particular Bound<'py, PyList>
, this code translates into the following Rust code:
use pyo3::prelude::*;
use pyo3::types::PyList;
fn example<'py>(py: Python<'py>) -> PyResult<()> {
let x: Bound<'py, PyList> = PyList::empty_bound(py);
x.append(1)?;
let y: Bound<'py, PyList> = x.clone(); // y is a new reference to the same list
drop(x); // release the original reference x
Ok(())
}
Python::with_gil(example).unwrap();
Or, without the type annotations:
use pyo3::prelude::*;
use pyo3::types::PyList;
fn example(py: Python<'_>) -> PyResult<()> {
let x = PyList::empty_bound(py);
x.append(1)?;
let y = x.clone();
drop(x);
Ok(())
}
Python::with_gil(example).unwrap();
Function argument lifetimes
Because the 'py
lifetime often appears in many function arguments as part of the Bound<'py, T>
smart pointer, the Rust compiler will often require annotations of input and output lifetimes. This occurs when the function output has at least one lifetime, and there is more than one lifetime present on the inputs.
To demonstrate, consider this function which takes accepts Python objects and applies the Python +
operation to them:
use pyo3::prelude::*;
fn add(left: &'_ Bound<'_, PyAny>, right: &'_ Bound<'_, PyAny>) -> PyResult<Bound<'_, PyAny>> {
left.add(right)
}
Because the Python +
operation might raise an exception, this function returns PyResult<Bound<'_, PyAny>>
. It doesn't need ownership of the inputs, so it takes &Bound<'_, PyAny>
shared references. To demonstrate the point, all lifetimes have used the wildcard '_
to allow the Rust compiler to attempt to infer them. Because there are four input lifetimes (two lifetimes of the shared references, and two 'py
lifetimes unnamed inside the Bound<'_, PyAny>
pointers), the compiler cannot reason about which must be connected to the output.
The correct way to solve this is to add the 'py
lifetime as a parameter for the function, and name all the 'py
lifetimes inside the Bound<'py, PyAny>
smart pointers. For the shared references, it's also fine to reduce &'_
to just &
. The working end result is below:
use pyo3::prelude::*;
fn add<'py>(
left: &Bound<'py, PyAny>,
right: &Bound<'py, PyAny>,
) -> PyResult<Bound<'py, PyAny>> {
left.add(right)
}
Python::with_gil(|py| {
let s = pyo3::types::PyString::new_bound(py, "s");
assert!(add(&s, &s).unwrap().eq("ss").unwrap());
})
If naming the 'py
lifetime adds unwanted complexity to the function signature, it is also acceptable to return PyObject
(aka Py<PyAny>
), which has no lifetime. The cost is instead paid by a slight increase in implementation complexity, as seen by the introduction of a call to Bound::unbind
:
use pyo3::prelude::*;
fn add(left: &Bound<'_, PyAny>, right: &Bound<'_, PyAny>) -> PyResult<PyObject> {
let output: Bound<'_, PyAny> = left.add(right)?;
Ok(output.unbind())
}
Python::with_gil(|py| {
let s = pyo3::types::PyString::new_bound(py, "s");
assert!(add(&s, &s).unwrap().bind(py).eq("ss").unwrap());
})
Borrowed<'a, 'py, T>
Borrowed<'a, 'py, T>
is an advanced type used just occasionally at the edge of interaction with the Python interpreter. It can be thought of as analogous to the shared reference &'a Bound<'py, T>
. The difference is that Borrowed<'a, 'py, T>
is just a smart pointer rather than a reference-to-a-smart-pointer, which is a helpful reduction in indirection in specific interactions with the Python interpreter.
Borrowed<'a, 'py, T>
dereferences to Bound<'py, T>
, so all methods on Bound<'py, T>
are available on Borrowed<'a, 'py, T>
.
An example where Borrowed<'a, 'py, T>
is used is in PyTupleMethods::get_borrowed_item
:
use pyo3::prelude::*;
use pyo3::types::PyTuple;
fn example<'py>(py: Python<'py>) -> PyResult<()> {
// Create a new tuple with the elements (0, 1, 2)
let t = PyTuple::new_bound(py, [0, 1, 2]);
for i in 0..=2 {
let entry: Borrowed<'_, 'py, PyAny> = t.get_borrowed_item(i)?;
// `PyAnyMethods::extract` is available on `Borrowed`
// via the dereference to `Bound`
let value: usize = entry.extract()?;
assert_eq!(i, value);
}
Ok(())
}
Python::with_gil(example).unwrap();
Casting between smart pointer types
To convert between Py<T>
and Bound<'py, T>
use the bind()
/ into_bound()
methods. Use the as_unbound()
/ unbind()
methods to go back from Bound<'py, T>
to Py<T>
.
let obj: Py<PyAny> = ...;
let bound: &Bound<'py, PyAny> = obj.bind(py);
let bound: Bound<'py, PyAny> = obj.into_bound(py);
let obj: &Py<PyAny> = bound.as_unbound();
let obj: Py<PyAny> = bound.unbind();
To convert between Bound<'py, T>
and Borrowed<'a, 'py, T>
use the as_borrowed()
method. Borrowed<'a, 'py, T>
has a deref coercion to Bound<'py, T>
. Use the to_owned()
method to increment the Python reference count and to create a new Bound<'py, T>
from the Borrowed<'a, 'py, T>
.
let bound: Bound<'py, PyAny> = ...;
let borrowed: Borrowed<'_, 'py, PyAny> = bound.as_borrowed();
// deref coercion
let bound: &Bound<'py, PyAny> = &borrowed;
// create a new Bound by increase the Python reference count
let bound: Bound<'py, PyAny> = borrowed.to_owned();
To convert between Py<T>
and Borrowed<'a, 'py, T>
use the bind_borrowed()
method. Use either as_unbound()
or .to_owned().unbind()
to go back to Py<T>
from Borrowed<'a, 'py, T>
, via Bound<'py, T>
.
let obj: Py<PyAny> = ...;
let borrowed: Borrowed<'_, 'py, PyAny> = bound.as_borrowed();
// via deref coercion to Bound and then using Bound::as_unbound
let obj: &Py<PyAny> = borrowed.as_unbound();
// via a new Bound by increasing the Python reference count, and unbind it
let obj: Py<PyAny> = borrowed.to_owned().unbind().
Concrete Python types
In all of Py<T>
, Bound<'py, T>
, and Borrowed<'a, 'py, T>
, the type parameter T
denotes the type of the Python object referred to by the smart pointer.
This parameter T
can be filled by:
PyAny
, which represents any Python object,- Native Python types such as
PyList
,PyTuple
, andPyDict
, and #[pyclass]
types defined from Rust
The following subsections covers some further detail about how to work with these types:
- the APIs that are available for these concrete types,
- how to cast
Bound<'py, T>
to a specific concrete type, and - how to get Rust data out of a
Bound<'py, T>
.
Using APIs for concrete Python types
Each concrete Python type such as PyAny
, PyTuple
and PyDict
exposes its API on the corresponding bound smart pointer Bound<'py, PyAny>
, Bound<'py, PyTuple>
and Bound<'py, PyDict>
.
Each type's API is exposed as a trait: PyAnyMethods
, PyTupleMethods
, PyDictMethods
, and so on for all concrete types. Using traits rather than associated methods on the Bound
smart pointer is done for a couple of reasons:
- Clarity of documentation: each trait gets its own documentation page in the PyO3 API docs. If all methods were on the
Bound
smart pointer directly, the vast majority of PyO3's API would be on a single, extremely long, documentation page. - Consistency: downstream code implementing Rust APIs for existing Python types can also follow this pattern of using a trait. Downstream code would not be allowed to add new associated methods directly on the
Bound
type. - Future design: it is hoped that a future Rust with arbitrary self types will remove the need for these traits in favour of placing the methods directly on
PyAny
,PyTuple
,PyDict
, and so on.
These traits are all included in the pyo3::prelude
module, so with the glob import use pyo3::prelude::*
the full PyO3 API is made available to downstream code.
The following function accesses the first item in the input Python list, using the .get_item()
method from the PyListMethods
trait:
use pyo3::prelude::*;
use pyo3::types::PyList;
fn get_first_item<'py>(list: &Bound<'py, PyList>) -> PyResult<Bound<'py, PyAny>> {
list.get_item(0)
}
Python::with_gil(|py| {
let l = PyList::new_bound(py, ["hello world"]);
assert!(get_first_item(&l).unwrap().eq("hello world").unwrap());
})
Casting between Python object types
To cast Bound<'py, T>
smart pointers to some other type, use the .downcast()
family of functions. This converts &Bound<'py, T>
to a different &Bound<'py, U>
, without transferring ownership. There is also .downcast_into()
to convert Bound<'py, T>
to Bound<'py, U>
with transfer of ownership. These methods are available for all types T
which implement the PyTypeCheck
trait.
Casting to Bound<'py, PyAny>
can be done with .as_any()
or .into_any()
.
For example, the following snippet shows how to cast Bound<'py, PyAny>
to Bound<'py, PyTuple>
:
use pyo3::prelude::*;
use pyo3::types::PyTuple;
fn example<'py>(py: Python<'py>) -> PyResult<()> {
// create a new Python `tuple`, and use `.into_any()` to erase the type
let obj: Bound<'py, PyAny> = PyTuple::empty_bound(py).into_any();
// use `.downcast()` to cast to `PyTuple` without transferring ownership
let _: &Bound<'py, PyTuple> = obj.downcast()?;
// use `.downcast_into()` to cast to `PyTuple` with transfer of ownership
let _: Bound<'py, PyTuple> = obj.downcast_into()?;
Ok(())
}
Python::with_gil(example).unwrap()
Custom #[pyclass]
types implement PyTypeCheck
, so .downcast()
also works for these types. The snippet below is the same as the snippet above casting instead to a custom type MyClass
:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
fn example<'py>(py: Python<'py>) -> PyResult<()> {
// create a new Python `tuple`, and use `.into_any()` to erase the type
let obj: Bound<'py, PyAny> = Bound::new(py, MyClass {})?.into_any();
// use `.downcast()` to cast to `MyClass` without transferring ownership
let _: &Bound<'py, MyClass> = obj.downcast()?;
// use `.downcast_into()` to cast to `MyClass` with transfer of ownership
let _: Bound<'py, MyClass> = obj.downcast_into()?;
Ok(())
}
Python::with_gil(example).unwrap()
Extracting Rust data from Python objects
To extract Rust data from Python objects, use .extract()
instead of .downcast()
. This method is available for all types which implement the [FromPyObject
] trait.
For example, the following snippet extracts a Rust tuple of integers from a Python tuple:
use pyo3::prelude::*;
use pyo3::types::PyTuple;
fn example<'py>(py: Python<'py>) -> PyResult<()> {
// create a new Python `tuple`, and use `.into_any()` to erase the type
let obj: Bound<'py, PyAny> = PyTuple::new_bound(py, [1, 2, 3]).into_any();
// extracting the Python `tuple` to a rust `(i32, i32, i32)` tuple
let (x, y, z) = obj.extract::<(i32, i32, i32)>()?;
assert_eq!((x, y, z), (1, 2, 3));
Ok(())
}
Python::with_gil(example).unwrap()
To avoid copying data, #[pyclass]
types can directly reference Rust data stored within the Python objects without needing to .extract()
. See the corresponding documentation in the class section of the guide
for more detail.
The GIL Refs API
The GIL Refs API was PyO3's primary API prior to PyO3 0.21. The main difference was that instead of the Bound<'py, PyAny>
smart pointer, the "GIL Reference" &'py PyAny
was used. (This was similar for other Python types.)
As of PyO3 0.21, the GIL Refs API is deprecated. See the migration guide for details on how to upgrade.
The following sections note some historical detail about the GIL Refs API.
PyAny
Represented: a Python object of unspecified type. In the GIL Refs API, this was only accessed as the GIL Ref &'py PyAny
.
Used: &'py PyAny
was used to refer to some Python object when the GIL lifetime was available for the whole duration access was needed. For example, intermediate values and arguments to pyfunction
s or pymethod
s implemented in Rust where any type is allowed.
Conversions:
For a &PyAny
object reference any
where the underlying object is a Python-native type such as
a list:
use pyo3::prelude::*;
use pyo3::types::PyList;
Python::with_gil(|py| -> PyResult<()> {
#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API.
let obj: &PyAny = PyList::empty(py);
// To &PyList with PyAny::downcast
let _: &PyList = obj.downcast()?;
// To Py<PyAny> (aka PyObject) with .into()
let _: Py<PyAny> = obj.into();
// To Py<PyList> with PyAny::extract
let _: Py<PyList> = obj.extract()?;
Ok(())
}).unwrap();
For a &PyAny
object reference any
where the underlying object is a #[pyclass]
:
use pyo3::prelude::*;
#[pyclass] #[derive(Clone)] struct MyClass { }
Python::with_gil(|py| -> PyResult<()> {
#[allow(deprecated)] // into_ref is part of the deprecated GIL Refs API
let obj: &PyAny = Py::new(py, MyClass {})?.into_ref(py);
// To &PyCell<MyClass> with PyAny::downcast
#[allow(deprecated)] // &PyCell is part of the deprecated GIL Refs API
let _: &PyCell<MyClass> = obj.downcast()?;
// To Py<PyAny> (aka PyObject) with .into()
let _: Py<PyAny> = obj.into();
// To Py<MyClass> with PyAny::extract
let _: Py<MyClass> = obj.extract()?;
// To MyClass with PyAny::extract, if MyClass: Clone
let _: MyClass = obj.extract()?;
// To PyRef<'_, MyClass> or PyRefMut<'_, MyClass> with PyAny::extract
let _: PyRef<'_, MyClass> = obj.extract()?;
let _: PyRefMut<'_, MyClass> = obj.extract()?;
Ok(())
}).unwrap();
PyTuple
, PyDict
, and many more
Represented: a native Python object of known type. In the GIL Refs API, they were only accessed as the GIL Refs &'py PyTuple
, &'py PyDict
.
Used: &'py PyTuple
and similar were used to operate with native Python types while holding the GIL. Like PyAny
, this is the most convenient form to use for function arguments and intermediate values.
These GIL Refs implement Deref<Target = PyAny>
, so they all expose the same methods which can be found on PyAny
.
To see all Python types exposed by PyO3
consult the pyo3::types
module.
Conversions:
use pyo3::prelude::*;
use pyo3::types::PyList;
Python::with_gil(|py| -> PyResult<()> {
#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API.
let list = PyList::empty(py);
// Use methods from PyAny on all Python types with Deref implementation
let _ = list.repr()?;
// To &PyAny automatically with Deref implementation
let _: &PyAny = list;
// To &PyAny explicitly with .as_ref()
#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API.
let _: &PyAny = list.as_ref();
// To Py<T> with .into() or Py::from()
let _: Py<PyList> = list.into();
// To PyObject with .into() or .to_object(py)
let _: PyObject = list.into();
Ok(())
}).unwrap();
Py<T>
and PyObject
Represented: a GIL-independent reference to a Python object. This can be a Python native type
(like PyTuple
), or a pyclass
type implemented in Rust. The most commonly-used variant,
Py<PyAny>
, is also known as PyObject
.
Used: Whenever you want to carry around references to a Python object without caring about a GIL lifetime. For example, storing Python object references in a Rust struct that outlives the Python-Rust FFI boundary, or returning objects from functions implemented in Rust back to Python.
Can be cloned using Python reference counts with .clone()
.
PyCell<SomeType>
Represented: a reference to a Rust object (instance of PyClass
) wrapped in a Python object. The cell part is an analog to stdlib's RefCell
to allow access to &mut
references.
Used: for accessing pure-Rust API of the instance (members and functions taking &SomeType
or &mut SomeType
) while maintaining the aliasing rules of Rust references.
Like PyO3's Python native types, the GIL Ref &PyCell<T>
implements Deref<Target = PyAny>
, so it also exposed all of the methods on PyAny
.
Conversions:
PyCell<T>
was used to access &T
and &mut T
via PyRef<T>
and PyRefMut<T>
respectively.
use pyo3::prelude::*;
#[pyclass] struct MyClass { }
Python::with_gil(|py| -> PyResult<()> {
#[allow(deprecated)] // &PyCell is part of the deprecated GIL Refs API
let cell: &PyCell<MyClass> = PyCell::new(py, MyClass {})?;
// To PyRef<T> with .borrow() or .try_borrow()
let py_ref: PyRef<'_, MyClass> = cell.try_borrow()?;
let _: &MyClass = &*py_ref;
drop(py_ref);
// To PyRefMut<T> with .borrow_mut() or .try_borrow_mut()
let mut py_ref_mut: PyRefMut<'_, MyClass> = cell.try_borrow_mut()?;
let _: &mut MyClass = &mut *py_ref_mut;
Ok(())
}).unwrap();
PyCell<T>
was also accessed like a Python-native type.
use pyo3::prelude::*;
#[pyclass] struct MyClass { }
Python::with_gil(|py| -> PyResult<()> {
#[allow(deprecated)] // &PyCell is part of the deprecate GIL Refs API
let cell: &PyCell<MyClass> = PyCell::new(py, MyClass {})?;
// Use methods from PyAny on PyCell<T> with Deref implementation
let _ = cell.repr()?;
// To &PyAny automatically with Deref implementation
let _: &PyAny = cell;
// To &PyAny explicitly with .as_ref()
#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API.
let _: &PyAny = cell.as_ref();
Ok(())
}).unwrap();
Python exceptions
Defining a new exception
Use the create_exception!
macro:
use pyo3::create_exception;
create_exception!(module, MyError, pyo3::exceptions::PyException);
module
is the name of the containing module.MyError
is the name of the new exception type.
For example:
use pyo3::prelude::*;
use pyo3::create_exception;
use pyo3::types::IntoPyDict;
use pyo3::exceptions::PyException;
create_exception!(mymodule, CustomError, PyException);
Python::with_gil(|py| {
let ctx = [("CustomError", py.get_type_bound::<CustomError>())].into_py_dict_bound(py);
pyo3::py_run!(
py,
*ctx,
"assert str(CustomError) == \"<class 'mymodule.CustomError'>\""
);
pyo3::py_run!(py, *ctx, "assert CustomError('oops').args == ('oops',)");
});
When using PyO3 to create an extension module, you can add the new exception to the module like this, so that it is importable from Python:
use pyo3::prelude::*;
use pyo3::exceptions::PyException;
pyo3::create_exception!(mymodule, CustomError, PyException);
#[pymodule]
fn mymodule(py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {
// ... other elements added to module ...
m.add("CustomError", py.get_type_bound::<CustomError>())?;
Ok(())
}
Raising an exception
As described in the function error handling chapter, to raise an exception from a #[pyfunction]
or #[pymethods]
, return an Err(PyErr)
. PyO3 will automatically raise this exception for you when returning the result to Python.
You can also manually write and fetch errors in the Python interpreter's global state:
use pyo3::{Python, PyErr};
use pyo3::exceptions::PyTypeError;
Python::with_gil(|py| {
PyTypeError::new_err("Error").restore(py);
assert!(PyErr::occurred(py));
drop(PyErr::fetch(py));
});
Checking exception types
Python has an isinstance
method to check an object's type.
In PyO3 every object has the PyAny::is_instance
and PyAny::is_instance_of
methods which do the same thing.
use pyo3::prelude::*;
use pyo3::types::{PyBool, PyList};
Python::with_gil(|py| {
assert!(PyBool::new_bound(py, true).is_instance_of::<PyBool>());
let list = PyList::new_bound(py, &[1, 2, 3, 4]);
assert!(!list.is_instance_of::<PyBool>());
assert!(list.is_instance_of::<PyList>());
});
To check the type of an exception, you can similarly do:
use pyo3::exceptions::PyTypeError;
use pyo3::prelude::*;
Python::with_gil(|py| {
let err = PyTypeError::new_err(());
err.is_instance_of::<PyTypeError>(py);
});
Using exceptions defined in Python code
It is possible to use an exception defined in Python code as a native Rust type.
The import_exception!
macro allows importing a specific exception class and defines a Rust type
for that exception.
#![allow(dead_code)]
use pyo3::prelude::*;
mod io {
pyo3::import_exception!(io, UnsupportedOperation);
}
fn tell(file: &Bound<'_, PyAny>) -> PyResult<u64> {
match file.call_method0("tell") {
Err(_) => Err(io::UnsupportedOperation::new_err("not supported: tell")),
Ok(x) => x.extract::<u64>(),
}
}
pyo3::exceptions
defines exceptions for several standard library modules.
Calling Python functions
The Bound<'py, T>
smart pointer (such as Bound<'py, PyAny>
, Bound<'py, PyList>
, or Bound<'py, MyClass>
) can be used to call Python functions.
PyO3 offers two APIs to make function calls:
call
- call any callable Python object.call_method
- call a method on the Python object.
Both of these APIs take args
and kwargs
arguments (for positional and keyword arguments respectively). There are variants for less complex calls:
call1
andcall_method1
to call only with positionalargs
.call0
andcall_method0
to call with no arguments.
For convenience the Py<T>
smart pointer also exposes these same six API methods, but needs a Python
token as an additional first argument to prove the GIL is held.
The example below calls a Python function behind a PyObject
(aka Py<PyAny>
) reference:
use pyo3::prelude::*;
use pyo3::types::PyTuple;
fn main() -> PyResult<()> {
let arg1 = "arg1";
let arg2 = "arg2";
let arg3 = "arg3";
Python::with_gil(|py| {
let fun: Py<PyAny> = PyModule::from_code_bound(
py,
"def example(*args, **kwargs):
if args != ():
print('called with args', args)
if kwargs != {}:
print('called with kwargs', kwargs)
if args == () and kwargs == {}:
print('called with no arguments')",
"",
"",
)?
.getattr("example")?
.into();
// call object without any arguments
fun.call0(py)?;
// pass object with Rust tuple of positional arguments
let args = (arg1, arg2, arg3);
fun.call1(py, args)?;
// call object with Python tuple of positional arguments
let args = PyTuple::new_bound(py, &[arg1, arg2, arg3]);
fun.call1(py, args)?;
Ok(())
})
}
Creating keyword arguments
For the call
and call_method
APIs, kwargs
are Option<&Bound<'py, PyDict>>
, so can either be None
or Some(&dict)
. You can use the IntoPyDict
trait to convert other dict-like containers, e.g. HashMap
or BTreeMap
, as well as tuples with up to 10 elements and Vec
s where each element is a two-element tuple.
use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
use std::collections::HashMap;
fn main() -> PyResult<()> {
let key1 = "key1";
let val1 = 1;
let key2 = "key2";
let val2 = 2;
Python::with_gil(|py| {
let fun: Py<PyAny> = PyModule::from_code_bound(
py,
"def example(*args, **kwargs):
if args != ():
print('called with args', args)
if kwargs != {}:
print('called with kwargs', kwargs)
if args == () and kwargs == {}:
print('called with no arguments')",
"",
"",
)?
.getattr("example")?
.into();
// call object with PyDict
let kwargs = [(key1, val1)].into_py_dict_bound(py);
fun.call_bound(py, (), Some(&kwargs))?;
// pass arguments as Vec
let kwargs = vec![(key1, val1), (key2, val2)];
fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?;
// pass arguments as HashMap
let mut kwargs = HashMap::<&str, i32>::new();
kwargs.insert(key1, 1);
fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?;
Ok(())
})
}
During PyO3's migration from "GIL Refs" to the Bound<T>
smart pointer, Py<T>::call
is temporarily named call_bound
(and call_method
is temporarily call_method_bound
).
(This temporary naming is only the case for the Py<T>
smart pointer. The methods on the &PyAny
GIL Ref such as call
have not been given replacements, and the methods on the Bound<PyAny>
smart pointer such as Bound<PyAny>::call
already use follow the newest API conventions.)
Executing existing Python code
If you already have some existing Python code that you need to execute from Rust, the following FAQs can help you select the right PyO3 functionality for your situation:
Want to access Python APIs? Then use PyModule::import
.
Pymodule::import
can
be used to get handle to a Python module from Rust. You can use this to import and use any Python
module available in your environment.
use pyo3::prelude::*;
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let builtins = PyModule::import_bound(py, "builtins")?;
let total: i32 = builtins
.getattr("sum")?
.call1((vec![1, 2, 3],))?
.extract()?;
assert_eq!(total, 6);
Ok(())
})
}
Want to run just an expression? Then use eval
.
Python::eval
is
a method to execute a Python expression
and return the evaluated value as a Bound<'py, PyAny>
object.
use pyo3::prelude::*;
fn main() -> Result<(), ()> {
Python::with_gil(|py| {
let result = py
.eval_bound("[i * 10 for i in range(5)]", None, None)
.map_err(|e| {
e.print_and_set_sys_last_vars(py);
})?;
let res: Vec<i64> = result.extract().unwrap();
assert_eq!(res, vec![0, 10, 20, 30, 40]);
Ok(())
})
}
Want to run statements? Then use run
.
Python::run
is a method to execute one or more
Python statements.
This method returns nothing (like any Python statement), but you can get
access to manipulated objects via the locals
dict.
You can also use the py_run!
macro, which is a shorthand for Python::run
.
Since py_run!
panics on exceptions, we recommend you use this macro only for
quickly testing your Python extensions.
use pyo3::prelude::*;
use pyo3::py_run;
fn main() {
#[pyclass]
struct UserData {
id: u32,
name: String,
}
#[pymethods]
impl UserData {
fn as_tuple(&self) -> (u32, String) {
(self.id, self.name.clone())
}
fn __repr__(&self) -> PyResult<String> {
Ok(format!("User {}(id: {})", self.name, self.id))
}
}
Python::with_gil(|py| {
let userdata = UserData {
id: 34,
name: "Yu".to_string(),
};
let userdata = Py::new(py, userdata).unwrap();
let userdata_as_tuple = (34, "Yu");
py_run!(py, userdata userdata_as_tuple, r#"
assert repr(userdata) == "User Yu(id: 34)"
assert userdata.as_tuple() == userdata_as_tuple
"#);
})
}
You have a Python file or code snippet? Then use PyModule::from_code
.
PyModule::from_code
can be used to generate a Python module which can then be used just as if it was imported with
PyModule::import
.
Warning: This will compile and execute code. Never pass untrusted code to this function!
use pyo3::{prelude::*, types::IntoPyDict};
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let activators = PyModule::from_code_bound(
py,
r#"
def relu(x):
"""see https://en.wikipedia.org/wiki/Rectifier_(neural_networks)"""
return max(0.0, x)
def leaky_relu(x, slope=0.01):
return x if x >= 0 else x * slope
"#,
"activators.py",
"activators",
)?;
let relu_result: f64 = activators.getattr("relu")?.call1((-1.0,))?.extract()?;
assert_eq!(relu_result, 0.0);
let kwargs = [("slope", 0.2)].into_py_dict_bound(py);
let lrelu_result: f64 = activators
.getattr("leaky_relu")?
.call((-1.0,), Some(&kwargs))?
.extract()?;
assert_eq!(lrelu_result, -0.2);
Ok(())
})
}
Want to embed Python in Rust with additional modules?
Python maintains the sys.modules
dict as a cache of all imported modules.
An import in Python will first attempt to lookup the module from this dict,
and if not present will use various strategies to attempt to locate and load
the module.
The append_to_inittab
macro can be used to add additional #[pymodule]
modules to an embedded
Python interpreter. The macro must be invoked before initializing Python.
As an example, the below adds the module foo
to the embedded interpreter:
use pyo3::prelude::*;
#[pyfunction]
fn add_one(x: i64) -> i64 {
x + 1
}
#[pymodule]
fn foo(foo_module: &Bound<'_, PyModule>) -> PyResult<()> {
foo_module.add_function(wrap_pyfunction!(add_one, foo_module)?)?;
Ok(())
}
fn main() -> PyResult<()> {
pyo3::append_to_inittab!(foo);
Python::with_gil(|py| Python::run_bound(py, "import foo; foo.add_one(6)", None, None))
}
If append_to_inittab
cannot be used due to constraints in the program,
an alternative is to create a module using PyModule::new
and insert it manually into sys.modules
:
use pyo3::prelude::*;
use pyo3::types::PyDict;
#[pyfunction]
pub fn add_one(x: i64) -> i64 {
x + 1
}
fn main() -> PyResult<()> {
Python::with_gil(|py| {
// Create new module
let foo_module = PyModule::new_bound(py, "foo")?;
foo_module.add_function(wrap_pyfunction!(add_one, &foo_module)?)?;
// Import and get sys.modules
let sys = PyModule::import_bound(py, "sys")?;
let py_modules: Bound<'_, PyDict> = sys.getattr("modules")?.downcast_into()?;
// Insert foo into sys.modules
py_modules.set_item("foo", foo_module)?;
// Now we can import + run our python code
Python::run_bound(py, "import foo; foo.add_one(6)", None, None)
})
}
Include multiple Python files
You can include a file at compile time by using
std::include_str
macro.
Or you can load a file at runtime by using
std::fs::read_to_string
function.
Many Python files can be included and loaded as modules. If one file depends on
another you must preserve correct order while declaring PyModule
.
Example directory structure:
.
├── Cargo.lock
├── Cargo.toml
├── python_app
│ ├── app.py
│ └── utils
│ └── foo.py
└── src
└── main.rs
python_app/app.py
:
from utils.foo import bar
def run():
return bar()
python_app/utils/foo.py
:
def bar():
return "baz"
The example below shows:
- how to include content of
app.py
andutils/foo.py
into your rust binary - how to call function
run()
(declared inapp.py
) that needs function imported fromutils/foo.py
src/main.rs
:
use pyo3::prelude::*;
fn main() -> PyResult<()> {
let py_foo = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/python_app/utils/foo.py"
));
let py_app = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/python_app/app.py"));
let from_python = Python::with_gil(|py| -> PyResult<Py<PyAny>> {
PyModule::from_code_bound(py, py_foo, "utils.foo", "utils.foo")?;
let app: Py<PyAny> = PyModule::from_code_bound(py, py_app, "", "")?
.getattr("run")?
.into();
app.call0(py)
});
println!("py: {}", from_python?);
Ok(())
}
The example below shows:
- how to load content of
app.py
at runtime so that it sees its dependencies automatically - how to call function
run()
(declared inapp.py
) that needs function imported fromutils/foo.py
It is recommended to use absolute paths because then your binary can be run
from anywhere as long as your app.py
is in the expected directory (in this example
that directory is /usr/share/python_app
).
src/main.rs
:
use pyo3::prelude::*;
use pyo3::types::PyList;
use std::fs;
use std::path::Path;
fn main() -> PyResult<()> {
let path = Path::new("/usr/share/python_app");
let py_app = fs::read_to_string(path.join("app.py"))?;
let from_python = Python::with_gil(|py| -> PyResult<Py<PyAny>> {
let syspath = py
.import_bound("sys")?
.getattr("path")?
.downcast_into::<PyList>()?;
syspath.insert(0, &path)?;
let app: Py<PyAny> = PyModule::from_code_bound(py, &py_app, "", "")?
.getattr("run")?
.into();
app.call0(py)
});
println!("py: {}", from_python?);
Ok(())
}
Need to use a context manager from Rust?
Use context managers by directly invoking __enter__
and __exit__
.
use pyo3::prelude::*;
fn main() {
Python::with_gil(|py| {
let custom_manager = PyModule::from_code_bound(
py,
r#"
class House(object):
def __init__(self, address):
self.address = address
def __enter__(self):
print(f"Welcome to {self.address}!")
def __exit__(self, type, value, traceback):
if type:
print(f"Sorry you had {type} trouble at {self.address}")
else:
print(f"Thank you for visiting {self.address}, come again soon!")
"#,
"house.py",
"house",
)
.unwrap();
let house_class = custom_manager.getattr("House").unwrap();
let house = house_class.call1(("123 Main Street",)).unwrap();
house.call_method0("__enter__").unwrap();
let result = py.eval_bound("undefined_variable + 1", None, None);
// If the eval threw an exception we'll pass it through to the context manager.
// Otherwise, __exit__ is called with empty arguments (Python "None").
match result {
Ok(_) => {
let none = py.None();
house
.call_method1("__exit__", (&none, &none, &none))
.unwrap();
}
Err(e) => {
house
.call_method1(
"__exit__",
(
e.get_type_bound(py),
e.value_bound(py),
e.traceback_bound(py),
),
)
.unwrap();
}
}
})
}
Handling system signals/interrupts (Ctrl-C)
The best way to handle system signals when running Rust code is to periodically call Python::check_signals
to handle any signals captured by Python's signal handler. See also the FAQ entry.
Alternatively, set Python's signal
module to take the default action for a signal:
use pyo3::prelude::*;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let signal = py.import_bound("signal")?;
// Set SIGINT to have the default action
signal
.getattr("signal")?
.call1((signal.getattr("SIGINT")?, signal.getattr("SIG_DFL")?))?;
Ok(())
})
}
Type conversions
In this portion of the guide we'll talk about the mapping of Python types to Rust types offered by PyO3, as well as the traits available to perform conversions between them.
Mapping of Rust types to Python types
When writing functions callable from Python (such as a #[pyfunction]
or in a #[pymethods]
block), the trait FromPyObject
is required for function arguments, and IntoPy<PyObject>
is required for function return values.
Consult the tables in the following section to find the Rust types provided by PyO3 which implement these traits.
Argument Types
When accepting a function argument, it is possible to either use Rust library types or PyO3's Python-native types. (See the next section for discussion on when to use each.)
The table below contains the Python type and the corresponding function argument types that will accept them:
Python | Rust | Rust (Python-native) |
---|---|---|
object | - | PyAny |
str | String , Cow<str> , &str , char , OsString , PathBuf , Path | PyString , PyUnicode |
bytes | Vec<u8> , &[u8] , Cow<[u8]> | PyBytes |
bool | bool | PyBool |
int | i8 , u8 , i16 , u16 , i32 , u32 , i64 , u64 , i128 , u128 , isize , usize , num_bigint::BigInt 1, num_bigint::BigUint 1 | PyLong |
float | f32 , f64 | PyFloat |
complex | num_complex::Complex 2 | PyComplex |
list[T] | Vec<T> | PyList |
dict[K, V] | HashMap<K, V> , BTreeMap<K, V> , hashbrown::HashMap<K, V> 3, indexmap::IndexMap<K, V> 4 | PyDict |
tuple[T, U] | (T, U) , Vec<T> | PyTuple |
set[T] | HashSet<T> , BTreeSet<T> , hashbrown::HashSet<T> 3 | PySet |
frozenset[T] | HashSet<T> , BTreeSet<T> , hashbrown::HashSet<T> 3 | PyFrozenSet |
bytearray | Vec<u8> , Cow<[u8]> | PyByteArray |
slice | - | PySlice |
type | - | PyType |
module | - | PyModule |
collections.abc.Buffer | - | PyBuffer<T> |
datetime.datetime | SystemTime , chrono::DateTime<Tz> 5, chrono::NaiveDateTime 5 | PyDateTime |
datetime.date | chrono::NaiveDate 5 | PyDate |
datetime.time | chrono::NaiveTime 5 | PyTime |
datetime.tzinfo | chrono::FixedOffset 5, chrono::Utc 5, chrono_tz::TimeZone 6 | PyTzInfo |
datetime.timedelta | Duration , chrono::Duration 5 | PyDelta |
decimal.Decimal | rust_decimal::Decimal 7 | - |
ipaddress.IPv4Address | std::net::IpAddr , std::net::IpV4Addr | - |
ipaddress.IPv6Address | std::net::IpAddr , std::net::IpV6Addr | - |
os.PathLike | PathBuf , Path | PyString , PyUnicode |
pathlib.Path | PathBuf , Path | PyString , PyUnicode |
typing.Optional[T] | Option<T> | - |
typing.Sequence[T] | Vec<T> | PySequence |
typing.Mapping[K, V] | HashMap<K, V> , BTreeMap<K, V> , hashbrown::HashMap<K, V> 3, indexmap::IndexMap<K, V> 4 | &PyMapping |
typing.Iterator[Any] | - | PyIterator |
typing.Union[...] | See #[derive(FromPyObject)] | - |
It is also worth remembering the following special types:
What | Description |
---|---|
Python<'py> | A GIL token, used to pass to PyO3 constructors to prove ownership of the GIL. |
Bound<'py, T> | A Python object connected to the GIL lifetime. This provides access to most of PyO3's APIs. |
Py<T> | A Python object isolated from the GIL lifetime. This can be sent to other threads. |
PyObject | An alias for Py<PyAny> |
PyRef<T> | A #[pyclass] borrowed immutably. |
PyRefMut<T> | A #[pyclass] borrowed mutably. |
For more detail on accepting #[pyclass]
values as function arguments, see the section of this guide on Python Classes.
Using Rust library types vs Python-native types
Using Rust library types as function arguments will incur a conversion cost compared to using the Python-native types. Using the Python-native types is almost zero-cost (they just require a type check similar to the Python builtin function isinstance()
).
However, once that conversion cost has been paid, the Rust standard library types offer a number of benefits:
- You can write functionality in native-speed Rust code (free of Python's runtime costs).
- You get better interoperability with the rest of the Rust ecosystem.
- You can use
Python::allow_threads
to release the Python GIL and let other Python threads make progress while your Rust code is executing. - You also benefit from stricter type checking. For example you can specify
Vec<i32>
, which will only accept a Pythonlist
containing integers. The Python-native equivalent,&PyList
, would accept a Pythonlist
containing Python objects of any type.
For most PyO3 usage the conversion cost is worth paying to get these benefits. As always, if you're not sure it's worth it in your case, benchmark it!
Returning Rust values to Python
When returning values from functions callable from Python, PyO3's smart pointers (Py<T>
, Bound<'py, T>
, and Borrowed<'a, 'py, T>
) can be used with zero cost.
Because Bound<'py, T>
and Borrowed<'a, 'py, T>
have lifetime parameters, the Rust compiler may ask for lifetime annotations to be added to your function. See the section of the guide dedicated to this.
If your function is fallible, it should return PyResult<T>
or Result<T, E>
where E
implements From<E> for PyErr
. This will raise a Python
exception if the Err
variant is returned.
Finally, the following Rust types are also able to convert to Python as return values:
Rust type | Resulting Python Type |
---|---|
String | str |
&str | str |
bool | bool |
Any integer type (i32 , u32 , usize , etc) | int |
f32 , f64 | float |
Option<T> | Optional[T] |
(T, U) | Tuple[T, U] |
Vec<T> | List[T] |
Cow<[u8]> | bytes |
HashMap<K, V> | Dict[K, V] |
BTreeMap<K, V> | Dict[K, V] |
HashSet<T> | Set[T] |
BTreeSet<T> | Set[T] |
Py<T> | T |
Bound<T> | T |
PyRef<T: PyClass> | T |
PyRefMut<T: PyClass> | T |
Requires the num-bigint
optional feature.
Requires the num-complex
optional feature.
Requires the hashbrown
optional feature.
Requires the indexmap
optional feature.
Requires the chrono
optional feature.
Requires the chrono-tz
optional feature.
Requires the rust_decimal
optional feature.
Conversion traits
PyO3 provides some handy traits to convert between Python types and Rust types.
.extract()
and the FromPyObject
trait
The easiest way to convert a Python object to a Rust value is using
.extract()
. It returns a PyResult
with a type error if the conversion
fails, so usually you will use something like
use pyo3::prelude::*;
use pyo3::types::PyList;
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let list = PyList::new_bound(py, b"foo");
let v: Vec<i32> = list.extract()?;
assert_eq!(&v, &[102, 111, 111]);
Ok(())
})
}
This method is available for many Python object types, and can produce a wide
variety of Rust types, which you can check out in the implementor list of
FromPyObject
.
FromPyObject
is also implemented for your own Rust types wrapped as Python
objects (see the chapter about classes). There, in order to both be
able to operate on mutable references and satisfy Rust's rules of non-aliasing
mutable references, you have to extract the PyO3 reference wrappers PyRef
and PyRefMut
. They work like the reference wrappers of
std::cell::RefCell
and ensure (at runtime) that Rust borrows are allowed.
Deriving FromPyObject
FromPyObject
can be automatically derived for many kinds of structs and enums
if the member types themselves implement FromPyObject
. This even includes members
with a generic type T: FromPyObject
. Derivation for empty enums, enum variants and
structs is not supported.
Deriving FromPyObject
for structs
The derivation generates code that will attempt to access the attribute my_string
on
the Python object, i.e. obj.getattr("my_string")
, and call extract()
on the attribute.
use pyo3::prelude::*;
#[derive(FromPyObject)]
struct RustyStruct {
my_string: String,
}
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let module = PyModule::from_code_bound(
py,
"class Foo:
def __init__(self):
self.my_string = 'test'",
"",
"",
)?;
let class = module.getattr("Foo")?;
let instance = class.call0()?;
let rustystruct: RustyStruct = instance.extract()?;
assert_eq!(rustystruct.my_string, "test");
Ok(())
})
}
By setting the #[pyo3(item)]
attribute on the field, PyO3 will attempt to extract the value by calling the get_item
method on the Python object.
use pyo3::prelude::*;
#[derive(FromPyObject)]
struct RustyStruct {
#[pyo3(item)]
my_string: String,
}
use pyo3::types::PyDict;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let dict = PyDict::new_bound(py);
dict.set_item("my_string", "test")?;
let rustystruct: RustyStruct = dict.extract()?;
assert_eq!(rustystruct.my_string, "test");
Ok(())
})
}
The argument passed to getattr
and get_item
can also be configured:
use pyo3::prelude::*;
#[derive(FromPyObject)]
struct RustyStruct {
#[pyo3(item("key"))]
string_in_mapping: String,
#[pyo3(attribute("name"))]
string_attr: String,
}
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let module = PyModule::from_code_bound(
py,
"class Foo(dict):
def __init__(self):
self.name = 'test'
self['key'] = 'test2'",
"",
"",
)?;
let class = module.getattr("Foo")?;
let instance = class.call0()?;
let rustystruct: RustyStruct = instance.extract()?;
assert_eq!(rustystruct.string_attr, "test");
assert_eq!(rustystruct.string_in_mapping, "test2");
Ok(())
})
}
This tries to extract string_attr
from the attribute name
and string_in_mapping
from a mapping with the key "key"
. The arguments for attribute
are restricted to
non-empty string literals while item
can take any valid literal that implements
ToBorrowedObject
.
You can use #[pyo3(from_item_all)]
on a struct to extract every field with get_item
method.
In this case, you can't use #[pyo3(attribute)]
or barely use #[pyo3(item)]
on any field.
However, using #[pyo3(item("key"))]
to specify the key for a field is still allowed.
use pyo3::prelude::*;
#[derive(FromPyObject)]
#[pyo3(from_item_all)]
struct RustyStruct {
foo: String,
bar: String,
#[pyo3(item("foobar"))]
baz: String,
}
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let py_dict = py.eval_bound("{'foo': 'foo', 'bar': 'bar', 'foobar': 'foobar'}", None, None)?;
let rustystruct: RustyStruct = py_dict.extract()?;
assert_eq!(rustystruct.foo, "foo");
assert_eq!(rustystruct.bar, "bar");
assert_eq!(rustystruct.baz, "foobar");
Ok(())
})
}
Deriving FromPyObject
for tuple structs
Tuple structs are also supported but do not allow customizing the extraction. The input is
always assumed to be a Python tuple with the same length as the Rust type, the n
th field
is extracted from the n
th item in the Python tuple.
use pyo3::prelude::*;
#[derive(FromPyObject)]
struct RustyTuple(String, String);
use pyo3::types::PyTuple;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let tuple = PyTuple::new_bound(py, vec!["test", "test2"]);
let rustytuple: RustyTuple = tuple.extract()?;
assert_eq!(rustytuple.0, "test");
assert_eq!(rustytuple.1, "test2");
Ok(())
})
}
Tuple structs with a single field are treated as wrapper types which are described in the following section. To override this behaviour and ensure that the input is in fact a tuple, specify the struct as
use pyo3::prelude::*;
#[derive(FromPyObject)]
struct RustyTuple((String,));
use pyo3::types::PyTuple;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let tuple = PyTuple::new_bound(py, vec!["test"]);
let rustytuple: RustyTuple = tuple.extract()?;
assert_eq!((rustytuple.0).0, "test");
Ok(())
})
}
Deriving FromPyObject
for wrapper types
The pyo3(transparent)
attribute can be used on structs with exactly one field. This results
in extracting directly from the input object, i.e. obj.extract()
, rather than trying to access
an item or attribute. This behaviour is enabled per default for newtype structs and tuple-variants
with a single field.
use pyo3::prelude::*;
#[derive(FromPyObject)]
struct RustyTransparentTupleStruct(String);
#[derive(FromPyObject)]
#[pyo3(transparent)]
struct RustyTransparentStruct {
inner: String,
}
use pyo3::types::PyString;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
let s = PyString::new_bound(py, "test");
let tup: RustyTransparentTupleStruct = s.extract()?;
assert_eq!(tup.0, "test");
let stru: RustyTransparentStruct = s.extract()?;
assert_eq!(stru.inner, "test");
Ok(())
})
}
Deriving FromPyObject
for enums
The FromPyObject
derivation for enums generates code that tries to extract the variants in the
order of the fields. As soon as a variant can be extracted successfully, that variant is returned.
This makes it possible to extract Python union types like str | int
.
The same customizations and restrictions described for struct derivations apply to enum variants,
i.e. a tuple variant assumes that the input is a Python tuple, and a struct variant defaults to
extracting fields as attributes but can be configured in the same manner. The transparent
attribute can be applied to single-field-variants.
use pyo3::prelude::*;
#[derive(FromPyObject)]
#[derive(Debug)]
enum RustyEnum<'a> {
Int(usize), // input is a positive int
String(String), // input is a string
IntTuple(usize, usize), // input is a 2-tuple with positive ints
StringIntTuple(String, usize), // input is a 2-tuple with String and int
Coordinates3d {
// needs to be in front of 2d
x: usize,
y: usize,
z: usize,
},
Coordinates2d {
// only gets checked if the input did not have `z`
#[pyo3(attribute("x"))]
a: usize,
#[pyo3(attribute("y"))]
b: usize,
},
#[pyo3(transparent)]
CatchAll(&'a PyAny), // This extraction never fails
}
use pyo3::types::{PyBytes, PyString};
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
{
let thing = 42_u8.to_object(py);
let rust_thing: RustyEnum<'_> = thing.extract(py)?;
assert_eq!(
42,
match rust_thing {
RustyEnum::Int(i) => i,
other => unreachable!("Error extracting: {:?}", other),
}
);
}
{
let thing = PyString::new_bound(py, "text");
let rust_thing: RustyEnum<'_> = thing.extract()?;
assert_eq!(
"text",
match rust_thing {
RustyEnum::String(i) => i,
other => unreachable!("Error extracting: {:?}", other),
}
);
}
{
let thing = (32_u8, 73_u8).to_object(py);
let rust_thing: RustyEnum<'_> = thing.extract(py)?;
assert_eq!(
(32, 73),
match rust_thing {
RustyEnum::IntTuple(i, j) => (i, j),
other => unreachable!("Error extracting: {:?}", other),
}
);
}
{
let thing = ("foo", 73_u8).to_object(py);
let rust_thing: RustyEnum<'_> = thing.extract(py)?;
assert_eq!(
(String::from("foo"), 73),
match rust_thing {
RustyEnum::StringIntTuple(i, j) => (i, j),
other => unreachable!("Error extracting: {:?}", other),
}
);
}
{
let module = PyModule::from_code_bound(
py,
"class Foo(dict):
def __init__(self):
self.x = 0
self.y = 1
self.z = 2",
"",
"",
)?;
let class = module.getattr("Foo")?;
let instance = class.call0()?;
let rust_thing: RustyEnum<'_> = instance.extract()?;
assert_eq!(
(0, 1, 2),
match rust_thing {
RustyEnum::Coordinates3d { x, y, z } => (x, y, z),
other => unreachable!("Error extracting: {:?}", other),
}
);
}
{
let module = PyModule::from_code_bound(
py,
"class Foo(dict):
def __init__(self):
self.x = 3
self.y = 4",
"",
"",
)?;
let class = module.getattr("Foo")?;
let instance = class.call0()?;
let rust_thing: RustyEnum<'_> = instance.extract()?;
assert_eq!(
(3, 4),
match rust_thing {
RustyEnum::Coordinates2d { a, b } => (a, b),
other => unreachable!("Error extracting: {:?}", other),
}
);
}
{
let thing = PyBytes::new_bound(py, b"text");
let rust_thing: RustyEnum<'_> = thing.extract()?;
assert_eq!(
b"text",
match rust_thing {
RustyEnum::CatchAll(i) => i.downcast::<PyBytes>()?.as_bytes(),
other => unreachable!("Error extracting: {:?}", other),
}
);
}
Ok(())
})
}
If none of the enum variants match, a PyTypeError
containing the names of the
tested variants is returned. The names reported in the error message can be customized
through the #[pyo3(annotation = "name")]
attribute, e.g. to use conventional Python type
names:
use pyo3::prelude::*;
#[derive(FromPyObject)]
#[derive(Debug)]
enum RustyEnum {
#[pyo3(transparent, annotation = "str")]
String(String),
#[pyo3(transparent, annotation = "int")]
Int(isize),
}
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
{
let thing = 42_u8.to_object(py);
let rust_thing: RustyEnum = thing.extract(py)?;
assert_eq!(
42,
match rust_thing {
RustyEnum::Int(i) => i,
other => unreachable!("Error extracting: {:?}", other),
}
);
}
{
let thing = "foo".to_object(py);
let rust_thing: RustyEnum = thing.extract(py)?;
assert_eq!(
"foo",
match rust_thing {
RustyEnum::String(i) => i,
other => unreachable!("Error extracting: {:?}", other),
}
);
}
{
let thing = b"foo".to_object(py);
let error = thing.extract::<RustyEnum>(py).unwrap_err();
assert!(error.is_instance_of::<pyo3::exceptions::PyTypeError>(py));
}
Ok(())
})
}
If the input is neither a string nor an integer, the error message will be:
"'<INPUT_TYPE>' cannot be converted to 'str | int'"
.
#[derive(FromPyObject)]
Container Attributes
pyo3(transparent)
- extract the field directly from the object as
obj.extract()
instead ofget_item()
orgetattr()
- Newtype structs and tuple-variants are treated as transparent per default.
- only supported for single-field structs and enum variants
- extract the field directly from the object as
pyo3(annotation = "name")
- changes the name of the failed variant in the generated error message in case of failure.
- e.g.
pyo3("int")
reports the variant's type asint
. - only supported for enum variants
#[derive(FromPyObject)]
Field Attributes
pyo3(attribute)
,pyo3(attribute("name"))
- retrieve the field from an attribute, possibly with a custom name specified as an argument
- argument must be a string-literal.
pyo3(item)
,pyo3(item("key"))
- retrieve the field from a mapping, possibly with the custom key specified as an argument.
- can be any literal that implements
ToBorrowedObject
pyo3(from_py_with = "...")
- apply a custom function to convert the field from Python the desired Rust type.
- the argument must be the name of the function as a string.
- the function signature must be
fn(&Bound<PyAny>) -> PyResult<T>
whereT
is the Rust type of the argument.
IntoPy<T>
This trait defines the to-python conversion for a Rust type. It is usually implemented as
IntoPy<PyObject>
, which is the trait needed for returning a value from #[pyfunction]
and
#[pymethods]
.
All types in PyO3 implement this trait, as does a #[pyclass]
which doesn't use extends
.
Occasionally you may choose to implement this for custom types which are mapped to Python types without having a unique python type.
use pyo3::prelude::*;
#[allow(dead_code)]
struct MyPyObjectWrapper(PyObject);
impl IntoPy<PyObject> for MyPyObjectWrapper {
fn into_py(self, py: Python<'_>) -> PyObject {
self.0
}
}
The ToPyObject
trait
ToPyObject
is a conversion trait that allows various objects to be
converted into PyObject
. IntoPy<PyObject>
serves the
same purpose, except that it consumes self
.
Using async
and await
This feature is still in active development. See the related issue.
#[pyfunction]
and #[pymethods]
attributes also support async fn
.
#![allow(dead_code)]
#[cfg(feature = "experimental-async")] {
use std::{thread, time::Duration};
use futures::channel::oneshot;
use pyo3::prelude::*;
#[pyfunction]
async fn sleep(seconds: f64, result: Option<PyObject>) -> Option<PyObject> {
let (tx, rx) = oneshot::channel();
thread::spawn(move || {
thread::sleep(Duration::from_secs_f64(seconds));
tx.send(()).unwrap();
});
rx.await.unwrap();
result
}
}
Python awaitables instantiated with this method can only be awaited in asyncio context. Other Python async runtime may be supported in the future.
Send + 'static
constraint
Resulting future of an async fn
decorated by #[pyfunction]
must be Send + 'static
to be embedded in a Python object.
As a consequence, async fn
parameters and return types must also be Send + 'static
, so it is not possible to have a signature like async fn does_not_compile<'py>(arg: Bound<'py, PyAny>) -> Bound<'py, PyAny>
.
However, there is an exception for method receivers, so async methods can accept &self
/&mut self
. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows &self
over exclusive ones &mut self
to avoid racy borrow check failures at runtime.
Implicit GIL holding
Even if it is not possible to pass a py: Python<'py>
parameter to async fn
, the GIL is still held during the execution of the future – it's also the case for regular fn
without Python<'py>
/Bound<'py, PyAny>
parameter, yet the GIL is held.
It is still possible to get a Python
marker using Python::with_gil
; because with_gil
is reentrant and optimized, the cost will be negligible.
Release the GIL across .await
There is currently no simple way to release the GIL when awaiting a future, but solutions are currently in development.
Here is the advised workaround for now:
use std::{
future::Future,
pin::{Pin, pin},
task::{Context, Poll},
};
use pyo3::prelude::*;
struct AllowThreads<F>(F);
impl<F> Future for AllowThreads<F>
where
F: Future + Unpin + Send,
F::Output: Send,
{
type Output = F::Output;
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
let waker = cx.waker();
Python::with_gil(|gil| {
gil.allow_threads(|| pin!(&mut self.0).poll(&mut Context::from_waker(waker)))
})
}
}
Cancellation
Cancellation on the Python side can be caught using CancelHandle
type, by annotating a function parameter with #[pyo3(cancel_handle)]
.
#![allow(dead_code)]
#[cfg(feature = "experimental-async")] {
use futures::FutureExt;
use pyo3::prelude::*;
use pyo3::coroutine::CancelHandle;
#[pyfunction]
async fn cancellable(#[pyo3(cancel_handle)] mut cancel: CancelHandle) {
futures::select! {
/* _ = ... => println!("done"), */
_ = cancel.cancelled().fuse() => println!("cancelled"),
}
}
}
The Coroutine
type
To make a Rust future awaitable in Python, PyO3 defines a Coroutine
type, which implements the Python coroutine protocol.
Each coroutine.send
call is translated to a Future::poll
call. If a CancelHandle
parameter is declared, the exception passed to coroutine.throw
call is stored in it and can be retrieved with CancelHandle::cancelled
; otherwise, it cancels the Rust future, and the exception is reraised;
The type does not yet have a public constructor until the design is finalized.
Parallelism
CPython has the infamous Global Interpreter Lock, which prevents several threads from executing Python bytecode in parallel. This makes threading in Python a bad fit for CPU-bound tasks and often forces developers to accept the overhead of multiprocessing.
In PyO3 parallelism can be easily achieved in Rust-only code. Let's take a look at our word-count example, where we have a search
function that utilizes the rayon crate to count words in parallel.
#![allow(dead_code)]
use pyo3::prelude::*;
// These traits let us use `par_lines` and `map`.
use rayon::str::ParallelString;
use rayon::iter::ParallelIterator;
/// Count the occurrences of needle in line, case insensitive
fn count_line(line: &str, needle: &str) -> usize {
let mut total = 0;
for word in line.split(' ') {
if word == needle {
total += 1;
}
}
total
}
#[pyfunction]
fn search(contents: &str, needle: &str) -> usize {
contents
.par_lines()
.map(|line| count_line(line, needle))
.sum()
}
But let's assume you have a long running Rust function which you would like to execute several times in parallel. For the sake of example let's take a sequential version of the word count:
#![allow(dead_code)]
fn count_line(line: &str, needle: &str) -> usize {
let mut total = 0;
for word in line.split(' ') {
if word == needle {
total += 1;
}
}
total
}
fn search_sequential(contents: &str, needle: &str) -> usize {
contents.lines().map(|line| count_line(line, needle)).sum()
}
To enable parallel execution of this function, the Python::allow_threads
method can be used to temporarily release the GIL, thus allowing other Python threads to run. We then have a function exposed to the Python runtime which calls search_sequential
inside a closure passed to Python::allow_threads
to enable true parallelism:
#![allow(dead_code)]
use pyo3::prelude::*;
fn count_line(line: &str, needle: &str) -> usize {
let mut total = 0;
for word in line.split(' ') {
if word == needle {
total += 1;
}
}
total
}
fn search_sequential(contents: &str, needle: &str) -> usize {
contents.lines().map(|line| count_line(line, needle)).sum()
}
#[pyfunction]
fn search_sequential_allow_threads(py: Python<'_>, contents: &str, needle: &str) -> usize {
py.allow_threads(|| search_sequential(contents, needle))
}
Now Python threads can use more than one CPU core, resolving the limitation which usually makes multi-threading in Python only good for IO-bound tasks:
from concurrent.futures import ThreadPoolExecutor
from word_count import search_sequential_allow_threads
executor = ThreadPoolExecutor(max_workers=2)
future_1 = executor.submit(
word_count.search_sequential_allow_threads, contents, needle
)
future_2 = executor.submit(
word_count.search_sequential_allow_threads, contents, needle
)
result_1 = future_1.result()
result_2 = future_2.result()
Benchmark
Let's benchmark the word-count
example to verify that we really did unlock parallelism with PyO3.
We are using pytest-benchmark
to benchmark four word count functions:
- Pure Python version
- Rust parallel version
- Rust sequential version
- Rust sequential version executed twice with two Python threads
The benchmark script can be found here, and we can run nox
in the word-count
folder to benchmark these functions.
While the results of the benchmark of course depend on your machine, the relative results should be similar to this (mid 2020):
-------------------------------------------------------------------------------------------------- benchmark: 4 tests -------------------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_word_count_rust_parallel 1.7315 (1.0) 4.6495 (1.0) 1.9972 (1.0) 0.4299 (1.0) 1.8142 (1.0) 0.2049 (1.0) 40;46 500.6943 (1.0) 375 1
test_word_count_rust_sequential 7.3348 (4.24) 10.3556 (2.23) 8.0035 (4.01) 0.7785 (1.81) 7.5597 (4.17) 0.8641 (4.22) 26;5 124.9457 (0.25) 121 1
test_word_count_rust_sequential_twice_with_threads 7.9839 (4.61) 10.3065 (2.22) 8.4511 (4.23) 0.4709 (1.10) 8.2457 (4.55) 0.3927 (1.92) 17;17 118.3274 (0.24) 114 1
test_word_count_python_sequential 27.3985 (15.82) 45.4527 (9.78) 28.9604 (14.50) 4.1449 (9.64) 27.5781 (15.20) 0.4638 (2.26) 3;5 34.5299 (0.07) 35 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
You can see that the Python threaded version is not much slower than the Rust sequential version, which means compared to an execution on a single CPU core the speed has doubled.
Debugging
Macros
PyO3's attributes (#[pyclass]
, #[pymodule]
, etc.) are procedural macros, which means that they rewrite the source of the annotated item. You can view the generated source with the following command, which also expands a few other things:
cargo rustc --profile=check -- -Z unstable-options --pretty=expanded > expanded.rs; rustfmt expanded.rs
(You might need to install rustfmt if you don't already have it.)
You can also debug classic !
-macros by adding -Z trace-macros
:
cargo rustc --profile=check -- -Z unstable-options --pretty=expanded -Z trace-macros > expanded.rs; rustfmt expanded.rs
Note that those commands require using the nightly build of rust and may occasionally have bugs. See cargo expand for a more elaborate and stable version of those commands.
Running with Valgrind
Valgrind is a tool to detect memory management bugs such as memory leaks.
You first need to install a debug build of Python, otherwise Valgrind won't produce usable results. In Ubuntu there's e.g. a python3-dbg
package.
Activate an environment with the debug interpreter and recompile. If you're on Linux, use ldd
with the name of your binary and check that you're linking e.g. libpython3.7d.so.1.0
instead of libpython3.7.so.1.0
.
Download the suppressions file for CPython.
Run Valgrind with valgrind --suppressions=valgrind-python.supp ./my-command --with-options
Getting a stacktrace
The best start to investigate a crash such as an segmentation fault is a backtrace. You can set RUST_BACKTRACE=1
as an environment variable to get the stack trace on a panic!
. Alternatively you can use a debugger such as gdb
to explore the issue. Rust provides a wrapper, rust-gdb
, which has pretty-printers for inspecting Rust variables. Since PyO3 uses cdylib
for Python shared objects, it does not receive the pretty-print debug hooks in rust-gdb
(rust-lang/rust#96365). The mentioned issue contains a workaround for enabling pretty-printers in this case.
- Link against a debug build of python as described in the previous chapter
- Run
rust-gdb <my-binary>
- Set a breakpoint (
b
) onrust_panic
if you are investigating apanic!
- Enter
r
to run - After the crash occurred, enter
bt
orbt full
to print the stacktrace
Often it is helpful to run a small piece of Python code to exercise a section of Rust.
rust-gdb --args python -c "import my_package; my_package.sum_to_string(1, 2)"
Features reference
PyO3 provides a number of Cargo features to customize functionality. This chapter of the guide provides detail on each of them.
By default, only the macros
feature is enabled.
Features for extension module authors
extension-module
This feature is required when building a Python extension module using PyO3.
It tells PyO3's build script to skip linking against libpython.so
on Unix platforms, where this must not be done.
See the building and distribution section for further detail.
abi3
This feature is used when building Python extension modules to create wheels which are compatible with multiple Python versions.
It restricts PyO3's API to a subset of the full Python API which is guaranteed by PEP 384 to be forwards-compatible with future Python versions.
See the building and distribution section for further detail.
The abi3-pyXY
features
(abi3-py37
, abi3-py38
, abi3-py39
, abi3-py310
and abi3-py311
)
These features are extensions of the abi3
feature to specify the exact minimum Python version which the multiple-version-wheel will support.
See the building and distribution section for further detail.
generate-import-lib
This experimental feature is used to generate import libraries for Python DLL for MinGW-w64 and MSVC (cross-)compile targets.
Enabling it allows to (cross-)compile extension modules to any Windows targets without having to install the Windows Python distribution files for the target.
See the building and distribution section for further detail.
Features for embedding Python in Rust
auto-initialize
This feature changes Python::with_gil
to automatically initialize a Python interpreter (by calling prepare_freethreaded_python
) if needed.
If you do not enable this feature, you should call pyo3::prepare_freethreaded_python()
before attempting to call any other Python APIs.
Advanced Features
experimental-async
This feature adds support for async fn
in #[pyfunction]
and #[pymethods]
.
The feature has some unfinished refinements and performance improvements. To help finish this off, see issue #1632 and its associated draft PRs.
experimental-declarative-modules
This feature allows to declare Python modules using #[pymodule] mod my_module { ... }
syntax.
The feature has some unfinished refinements and edge cases. To help finish this off, see issue #3900.
experimental-inspect
This feature adds the pyo3::inspect
module, as well as IntoPy::type_output
and FromPyObject::type_input
APIs to produce Python type "annotations" for Rust types.
This is a first step towards adding first-class support for generating type annotations automatically in PyO3, however work is needed to finish this off. All feedback and offers of help welcome on issue #2454.
gil-refs
This feature is a backwards-compatibility feature to allow continued use of the "GIL Refs" APIs deprecated in PyO3 0.21. These APIs have performance drawbacks and soundness edge cases which the newer Bound<T>
smart pointer and accompanying APIs resolve.
This feature and the APIs it enables is expected to be removed in a future PyO3 version.
macros
This feature enables a dependency on the pyo3-macros
crate, which provides the procedural macros portion of PyO3's API:
#[pymodule]
#[pyfunction]
#[pyclass]
#[pymethods]
#[derive(FromPyObject)]
It also provides the py_run!
macro.
These macros require a number of dependencies which may not be needed by users who just need PyO3 for Python FFI. Disabling this feature enables faster builds for those users, as these dependencies will not be built if this feature is disabled.
This feature is enabled by default. To disable it, set
default-features = false
for thepyo3
entry in your Cargo.toml.
multiple-pymethods
This feature enables a dependency on inventory
, which enables each #[pyclass]
to have more than one #[pymethods]
block. This feature also requires a minimum Rust version of 1.62 due to limitations in the inventory
crate.
Most users should only need a single #[pymethods]
per #[pyclass]
. In addition, not all platforms (e.g. Wasm) are supported by inventory
. For this reason this feature is not enabled by default, meaning fewer dependencies and faster compilation for the majority of users.
See the #[pyclass]
implementation details for more information.
nightly
The nightly
feature needs the nightly Rust compiler. This allows PyO3 to use the auto_traits
and negative_impls
features to fix the Python::allow_threads
function.
resolve-config
The resolve-config
feature of the pyo3-build-config
crate controls whether that crate's
build script automatically resolves a Python interpreter / build configuration. This feature is primarily useful when building PyO3
itself. By default this feature is not enabled, meaning you can freely use pyo3-build-config
as a standalone library to read or write PyO3 build configuration files or resolve metadata about a Python interpreter.
Optional Dependencies
These features enable conversions between Python types and types from other Rust crates, enabling easy access to the rest of the Rust ecosystem.
anyhow
Adds a dependency on anyhow. Enables a conversion from anyhow’s Error
type to PyErr
, for easy error handling.
chrono
Adds a dependency on chrono. Enables a conversion from chrono's types to python:
- TimeDelta ->
PyDelta
- FixedOffset ->
PyDelta
- Utc ->
PyTzInfo
- NaiveDate ->
PyDate
- NaiveTime ->
PyTime
- DateTime ->
PyDateTime
chrono-tz
Adds a dependency on chrono-tz.
Enables conversion from and to Tz
.
It requires at least Python 3.9.
either
Adds a dependency on either. Enables a conversions into either’s Either
type.
eyre
Adds a dependency on eyre. Enables a conversion from eyre’s Report
type to PyErr
, for easy error handling.
hashbrown
Adds a dependency on hashbrown and enables conversions into its HashMap
and HashSet
types.
indexmap
Adds a dependency on indexmap and enables conversions into its IndexMap
type.
num-bigint
Adds a dependency on num-bigint and enables conversions into its BigInt
and BigUint
types.
num-complex
Adds a dependency on num-complex and enables conversions into its Complex
type.
rust_decimal
Adds a dependency on rust_decimal and enables conversions into its Decimal
type.
serde
Enables (de)serialization of Py<T>
objects via serde.
This allows to use #[derive(Serialize, Deserialize)
on structs that hold references to #[pyclass]
instances
#[cfg(feature = "serde")]
#[allow(dead_code)]
mod serde_only {
use pyo3::prelude::*;
use serde::{Deserialize, Serialize};
#[pyclass]
#[derive(Serialize, Deserialize)]
struct Permission {
name: String,
}
#[pyclass]
#[derive(Serialize, Deserialize)]
struct User {
username: String,
permissions: Vec<Py<Permission>>,
}
}
smallvec
Adds a dependency on smallvec and enables conversions into its SmallVec
type.
Memory management
⚠️ Warning: API update in progress 🛠️
PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer Bound<T>
.
This section on memory management is heavily weighted towards the now-deprecated "GIL Refs" API, which suffered from the drawbacks detailed here as well as CPU overheads.
See the smart pointer types for description on the new, simplified, memory model of the Bound API, which is built as a thin wrapper on Python reference counting.
Rust and Python have very different notions of memory management. Rust has a strict memory model with concepts of ownership, borrowing, and lifetimes, where memory is freed at predictable points in program execution. Python has a looser memory model in which variables are reference-counted with shared, mutable state by default. A global interpreter lock (GIL) is needed to prevent race conditions, and a garbage collector is needed to break reference cycles. Memory in Python is freed eventually by the garbage collector, but not usually in a predictable way.
PyO3 bridges the Rust and Python memory models with two different strategies for
accessing memory allocated on Python's heap from inside Rust. These are
GIL Refs such as &'py PyAny
, and GIL-independent Py<Any>
smart pointers.
GIL-bound memory
PyO3's GIL Refs such as &'py PyAny
make PyO3 more ergonomic to
use by ensuring that their lifetime can never be longer than the duration the
Python GIL is held. This means that most of PyO3's API can assume the GIL is
held. (If PyO3 could not assume this, every PyO3 API would need to take a
Python
GIL token to prove that the GIL is held.) This allows us to write
very simple and easy-to-understand programs like this:
use pyo3::prelude::*;
use pyo3::types::PyString;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
let hello = py
.eval("\"Hello World!\"", None, None)?
.downcast::<PyString>()?;
println!("Python says: {}", hello);
Ok(())
})?;
Ok(())
}
Internally, calling Python::with_gil()
creates a GILPool
which owns the
memory pointed to by the reference. In the example above, the lifetime of the
reference hello
is bound to the GILPool
. When the with_gil()
closure ends
the GILPool
is also dropped and the Python reference counts of the variables
it owns are decreased, releasing them to the Python garbage collector. Most
of the time we don't have to think about this, but consider the following:
use pyo3::prelude::*;
use pyo3::types::PyString;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
for _ in 0..10 {
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
let hello = py
.eval("\"Hello World!\"", None, None)?
.downcast::<PyString>()?;
println!("Python says: {}", hello);
}
// There are 10 copies of `hello` on Python's heap here.
Ok(())
})?;
Ok(())
}
We might assume that the hello
variable's memory is freed at the end of each
loop iteration, but in fact we create 10 copies of hello
on Python's heap.
This may seem surprising at first, but it is completely consistent with Rust's
memory model. The hello
variable is dropped at the end of each loop, but it
is only a reference to the memory owned by the GILPool
, and its lifetime is
bound to the GILPool
, not the for loop. The GILPool
isn't dropped until
the end of the with_gil()
closure, at which point the 10 copies of hello
are finally released to the Python garbage collector.
⚠️ Warning: GILPool
is no longer the preferred way to manage memory with PyO3 🛠️
PyO3 0.21 has introduced a new API known as the Bound API, which doesn't have the same surprising results. Instead, each Bound<T>
smart pointer releases the Python reference immediately on drop. See the smart pointer types for more details.
In general we don't want unbounded memory growth during loops! One workaround is to acquire and release the GIL with each iteration of the loop.
use pyo3::prelude::*;
use pyo3::types::PyString;
fn main() -> PyResult<()> {
for _ in 0..10 {
Python::with_gil(|py| -> PyResult<()> {
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
let hello = py
.eval("\"Hello World!\"", None, None)?
.downcast::<PyString>()?;
println!("Python says: {}", hello);
Ok(())
})?; // only one copy of `hello` at a time
}
Ok(())
}
It might not be practical or performant to acquire and release the GIL so many
times. Another workaround is to work with the GILPool
object directly, but
this is unsafe.
use pyo3::prelude::*;
use pyo3::types::PyString;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
for _ in 0..10 {
#[allow(deprecated)] // `new_pool` is not needed in code not using the GIL Refs API
let pool = unsafe { py.new_pool() };
let py = pool.python();
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
let hello = py
.eval("\"Hello World!\"", None, None)?
.downcast::<PyString>()?;
println!("Python says: {}", hello);
}
Ok(())
})?;
Ok(())
}
The unsafe method Python::new_pool
allows you to create a nested GILPool
from which you can retrieve a new py: Python
GIL token. Variables created
with this new GIL token are bound to the nested GILPool
and will be released
when the nested GILPool
is dropped. Here, the nested GILPool
is dropped
at the end of each loop iteration, before the with_gil()
closure ends.
When doing this, you must be very careful to ensure that once the GILPool
is
dropped you do not retain access to any owned references created after the
GILPool
was created. Read the
documentation for Python::new_pool()
for more information on safety.
This memory management can also be applicable when writing extension modules.
#[pyfunction]
and #[pymethods]
will create a GILPool
which lasts the entire
function call, releasing objects when the function returns. Most functions only create
a few objects, meaning this doesn't have a significant impact. Occasionally functions
with long complex loops may need to use Python::new_pool
as shown above.
⚠️ Warning: GILPool
is no longer the preferred way to manage memory with PyO3 🛠️
PyO3 0.21 has introduced a new API known as the Bound API, which doesn't have the same surprising results. Instead, each Bound<T>
smart pointer releases the Python reference immediately on drop. See the smart pointer types for more details.
GIL-independent memory
Sometimes we need a reference to memory on Python's heap that can outlive the
GIL. Python's Py<PyAny>
is analogous to Arc<T>
, but for variables whose
memory is allocated on Python's heap. Cloning a Py<PyAny>
increases its
internal reference count just like cloning Arc<T>
. The smart pointer can
outlive the "GIL is held" period in which it was created. It isn't magic,
though. We need to reacquire the GIL to access the memory pointed to by the
Py<PyAny>
.
What happens to the memory when the last Py<PyAny>
is dropped and its
reference count reaches zero? It depends whether or not we are holding the GIL.
use pyo3::prelude::*;
use pyo3::types::PyString;
fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
let hello: Py<PyString> = py.eval("\"Hello World!\"", None, None)?.extract()?;
#[allow(deprecated)] // as_ref is part of the GIL Refs API
{
println!("Python says: {}", hello.as_ref(py));
}
Ok(())
})?;
Ok(())
}
At the end of the Python::with_gil()
closure hello
is dropped, and then the
GIL is dropped. Since hello
is dropped while the GIL is still held by the
current thread, its memory is released to the Python garbage collector
immediately.
This example wasn't very interesting. We could have just used a GIL-bound
&PyString
reference. What happens when the last Py<Any>
is dropped while
we are not holding the GIL?
use pyo3::prelude::*;
use pyo3::types::PyString;
fn main() -> PyResult<()> {
let hello: Py<PyString> = Python::with_gil(|py| {
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
py.eval("\"Hello World!\"", None, None)?.extract()
})?;
// Do some stuff...
// Now sometime later in the program we want to access `hello`.
Python::with_gil(|py| {
#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API.
let hello = hello.as_ref(py);
println!("Python says: {}", hello);
});
// Now we're done with `hello`.
drop(hello); // Memory *not* released here.
// Sometime later we need the GIL again for something...
Python::with_gil(|py|
// Memory for `hello` is released here.
()
);
Ok(())
}
When hello
is dropped nothing happens to the pointed-to memory on Python's
heap because nothing can happen if we're not holding the GIL. Fortunately,
the memory isn't leaked. PyO3 keeps track of the memory internally and will
release it the next time we acquire the GIL.
We can avoid the delay in releasing memory if we are careful to drop the
Py<Any>
while the GIL is held.
use pyo3::prelude::*;
use pyo3::types::PyString;
fn main() -> PyResult<()> {
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
let hello: Py<PyString> =
Python::with_gil(|py| py.eval("\"Hello World!\"", None, None)?.extract())?;
// Do some stuff...
// Now sometime later in the program:
Python::with_gil(|py| {
#[allow(deprecated)] // as_ref is part of the GIL Refs API
{
println!("Python says: {}", hello.as_ref(py));
}
drop(hello); // Memory released here.
});
Ok(())
}
We could also have used Py::into_ref()
, which consumes self
, instead of
Py::as_ref()
. But note that in addition to being slower than as_ref()
,
into_ref()
binds the memory to the lifetime of the GILPool
, which means
that rather than being released immediately, the memory will not be released
until the GIL is dropped.
use pyo3::prelude::*;
use pyo3::types::PyString;
fn main() -> PyResult<()> {
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
let hello: Py<PyString> =
Python::with_gil(|py| py.eval("\"Hello World!\"", None, None)?.extract())?;
// Do some stuff...
// Now sometime later in the program:
Python::with_gil(|py| {
#[allow(deprecated)] // into_ref is part of the GIL Refs API
{
println!("Python says: {}", hello.into_ref(py));
}
// Memory not released yet.
// Do more stuff...
// Memory released here at end of `with_gil()` closure.
});
Ok(())
}
Performance
To achieve the best possible performance, it is useful to be aware of several tricks and sharp edges concerning PyO3's API.
extract
versus downcast
Pythonic API implemented using PyO3 are often polymorphic, i.e. they will accept &Bound<'_, PyAny>
and try to turn this into multiple more concrete types to which the requested operation is applied. This often leads to chains of calls to extract
, e.g.
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::{exceptions::PyTypeError, types::PyList};
fn frobnicate_list<'py>(list: &Bound<'_, PyList>) -> PyResult<Bound<'py, PyAny>> {
todo!()
}
fn frobnicate_vec<'py>(vec: Vec<Bound<'py, PyAny>>) -> PyResult<Bound<'py, PyAny>> {
todo!()
}
#[pyfunction]
fn frobnicate<'py>(value: &Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
if let Ok(list) = value.extract::<Bound<'_, PyList>>() {
frobnicate_list(&list)
} else if let Ok(vec) = value.extract::<Vec<Bound<'_, PyAny>>>() {
frobnicate_vec(vec)
} else {
Err(PyTypeError::new_err("Cannot frobnicate that type."))
}
}
This suboptimal as the FromPyObject<T>
trait requires extract
to have a Result<T, PyErr>
return type. For native types like PyList
, it faster to use downcast
(which extract
calls internally) when the error value is ignored. This avoids the costly conversion of a PyDowncastError
to a PyErr
required to fulfil the FromPyObject
contract, i.e.
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::{exceptions::PyTypeError, types::PyList};
fn frobnicate_list<'py>(list: &Bound<'_, PyList>) -> PyResult<Bound<'py, PyAny>> { todo!() }
fn frobnicate_vec<'py>(vec: Vec<Bound<'py, PyAny>>) -> PyResult<Bound<'py, PyAny>> { todo!() }
#[pyfunction]
fn frobnicate<'py>(value: &Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
// Use `downcast` instead of `extract` as turning `PyDowncastError` into `PyErr` is quite costly.
if let Ok(list) = value.downcast::<PyList>() {
frobnicate_list(list)
} else if let Ok(vec) = value.extract::<Vec<Bound<'_, PyAny>>>() {
frobnicate_vec(vec)
} else {
Err(PyTypeError::new_err("Cannot frobnicate that type."))
}
}
Access to Bound implies access to GIL token
Calling Python::with_gil
is effectively a no-op when the GIL is already held, but checking that this is the case still has a cost. If an existing GIL token can not be accessed, for example when implementing a pre-existing trait, but a GIL-bound reference is available, this cost can be avoided by exploiting that access to GIL-bound reference gives zero-cost access to a GIL token via Bound::py
.
For example, instead of writing
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyList;
struct Foo(Py<PyList>);
struct FooBound<'py>(Bound<'py, PyList>);
impl PartialEq<Foo> for FooBound<'_> {
fn eq(&self, other: &Foo) -> bool {
Python::with_gil(|py| {
let len = other.0.bind(py).len();
self.0.len() == len
})
}
}
use the more efficient
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyList;
struct Foo(Py<PyList>);
struct FooBound<'py>(Bound<'py, PyList>);
impl PartialEq<Foo> for FooBound<'_> {
fn eq(&self, other: &Foo) -> bool {
// Access to `&Bound<'py, PyAny>` implies access to `Python<'py>`.
let py = self.0.py();
let len = other.0.bind(py).len();
self.0.len() == len
}
}
Advanced topics
FFI
PyO3 exposes much of Python's C API through the ffi
module.
The C API is naturally unsafe and requires you to manage reference counts, errors and specific invariants yourself. Please refer to the C API Reference Manual and The Rustonomicon before using any function from that API.
Building and distribution
This chapter of the guide goes into detail on how to build and distribute projects using PyO3. The way to achieve this is very different depending on whether the project is a Python module implemented in Rust, or a Rust binary embedding Python. For both types of project there are also common problems such as the Python version to build for and the linker arguments to use.
The material in this chapter is intended for users who have already read the PyO3 README. It covers in turn the choices that can be made for Python modules and for Rust binaries. There is also a section at the end about cross-compiling projects using PyO3.
There is an additional sub-chapter dedicated to supporting multiple Python versions.
Configuring the Python version
PyO3 uses a build script (backed by the pyo3-build-config
crate) to determine the Python version and set the correct linker arguments. By default it will attempt to use the following in order:
- Any active Python virtualenv.
- The
python
executable (if it's a Python 3 interpreter). - The
python3
executable.
You can override the Python interpreter by setting the PYO3_PYTHON
environment variable, e.g. PYO3_PYTHON=python3.7
, PYO3_PYTHON=/usr/bin/python3.9
, or even a PyPy interpreter PYO3_PYTHON=pypy3
.
Once the Python interpreter is located, pyo3-build-config
executes it to query the information in the sysconfig
module which is needed to configure the rest of the compilation.
To validate the configuration which PyO3 will use, you can run a compilation with the environment variable PYO3_PRINT_CONFIG=1
set. An example output of doing this is shown below:
$ PYO3_PRINT_CONFIG=1 cargo build
Compiling pyo3 v0.14.1 (/home/david/dev/pyo3)
error: failed to run custom build command for `pyo3 v0.14.1 (/home/david/dev/pyo3)`
Caused by:
process didn't exit successfully: `/home/david/dev/pyo3/target/debug/build/pyo3-7a8cf4fe22e959b7/build-script-build` (exit status: 101)
--- stdout
cargo:rerun-if-env-changed=PYO3_CROSS
cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
-- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
implementation=CPython
version=3.8
shared=true
abi3=false
lib_name=python3.8
lib_dir=/usr/lib
executable=/usr/bin/python
pointer_width=64
build_flags=
suppress_build_script_link_lines=false
The PYO3_ENVIRONMENT_SIGNATURE
environment variable can be used to trigger rebuilds when its value changes, it has no other effect.
Advanced: config files
If you save the above output config from PYO3_PRINT_CONFIG
to a file, it is possible to manually override the contents and feed it back into PyO3 using the PYO3_CONFIG_FILE
env var.
If your build environment is unusual enough that PyO3's regular configuration detection doesn't work, using a config file like this will give you the flexibility to make PyO3 work for you. To see the full set of options supported, see the documentation for the InterpreterConfig
struct.
Building Python extension modules
Python extension modules need to be compiled differently depending on the OS (and architecture) that they are being compiled for. As well as multiple OSes (and architectures), there are also many different Python versions which are actively supported. Packages uploaded to PyPI usually want to upload prebuilt "wheels" covering many OS/arch/version combinations so that users on all these different platforms don't have to compile the package themselves. Package vendors can opt-in to the "abi3" limited Python API which allows their wheels to be used on multiple Python versions, reducing the number of wheels they need to compile, but restricts the functionality they can use.
There are many ways to go about this: it is possible to use cargo
to build the extension module (along with some manual work, which varies with OS). The PyO3 ecosystem has two packaging tools, maturin
and setuptools-rust
, which abstract over the OS difference and also support building wheels for PyPI upload.
PyO3 has some Cargo features to configure projects for building Python extension modules:
- The
extension-module
feature, which must be enabled when building Python extension modules. - The
abi3
feature and its version-specificabi3-pyXY
companions, which are used to opt-in to the limited Python API in order to support multiple Python versions in a single wheel.
This section describes each of these packaging tools before describing how to build manually without them. It then proceeds with an explanation of the extension-module
feature. Finally, there is a section describing PyO3's abi3
features.
Packaging tools
The PyO3 ecosystem has two main choices to abstract the process of developing Python extension modules:
maturin
is a command-line tool to build, package and upload Python modules. It makes opinionated choices about project layout meaning it needs very little configuration. This makes it a great choice for users who are building a Python extension from scratch and don't need flexibility.setuptools-rust
is an add-on forsetuptools
which adds extra keyword arguments to thesetup.py
configuration file. It requires more configuration thanmaturin
, however this gives additional flexibility for users adding Rust to an existing Python package that can't satisfymaturin
's constraints.
Consult each project's documentation for full details on how to get started using them and how to upload wheels to PyPI. It should be noted that while maturin
is able to build manylinux-compliant wheels out-of-the-box, setuptools-rust
requires a bit more effort, relying on Docker for this purpose.
There are also maturin-starter
and setuptools-rust-starter
examples in the PyO3 repository.
Manual builds
To build a PyO3-based Python extension manually, start by running cargo build
as normal in a library project which uses PyO3's extension-module
feature and has the cdylib
crate type.
Once built, symlink (or copy) and rename the shared library from Cargo's target/
directory to your desired output directory:
- on macOS, rename
libyour_module.dylib
toyour_module.so
. - on Windows, rename
libyour_module.dll
toyour_module.pyd
. - on Linux, rename
libyour_module.so
toyour_module.so
.
You can then open a Python shell in the output directory and you'll be able to run import your_module
.
If you're packaging your library for redistribution, you should indicated the Python interpreter your library is compiled for by including the platform tag in its name. This prevents incompatible interpreters from trying to import your library. If you're compiling for PyPy you must include the platform tag, or PyPy will ignore the module.
Bazel builds
To use PyO3 with bazel one needs to manually configure PyO3, PyO3-ffi and PyO3-macros. In particular, one needs to make sure that it is compiled with the right python flags for the version you intend to use. For example see:
- https://github.com/OliverFM/pytorch_with_gazelle -- for a minimal example of a repo that can use PyO3, PyTorch and Gazelle to generate python Build files.
- https://github.com/TheButlah/rules_pyo3 -- which has more extensive support, but is outdated.
Platform tags
Rather than using just the .so
or .pyd
extension suggested above (depending on OS), you can prefix the shared library extension with a platform tag to indicate the interpreter it is compatible with. You can query your interpreter's platform tag from the sysconfig
module. Some example outputs of this are seen below:
# CPython 3.10 on macOS
.cpython-310-darwin.so
# PyPy 7.3 (Python 3.8) on Linux
$ python -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))'
.pypy38-pp73-x86_64-linux-gnu.so
So, for example, a valid module library name on CPython 3.10 for macOS is your_module.cpython-310-darwin.so
, and its equivalent when compiled for PyPy 7.3 on Linux would be your_module.pypy38-pp73-x86_64-linux-gnu.so
.
See PEP 3149 for more background on platform tags.
macOS
On macOS, because the extension-module
feature disables linking to libpython
(see the next section), some additional linker arguments need to be set. maturin
and setuptools-rust
both pass these arguments for PyO3 automatically, but projects using manual builds will need to set these directly in order to support macOS.
The easiest way to set the correct linker arguments is to add a build.rs
with the following content:
fn main() {
pyo3_build_config::add_extension_module_link_args();
}
Remember to also add pyo3-build-config
to the build-dependencies
section in Cargo.toml
.
An alternative to using pyo3-build-config
is add the following to a cargo configuration file (e.g. .cargo/config.toml
):
[target.x86_64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
[target.aarch64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
Using the MacOS system python3 (/usr/bin/python3
, as opposed to python installed via homebrew, pyenv, nix, etc.) may result in runtime errors such as Library not loaded: @rpath/Python3.framework/Versions/3.8/Python3
. These can be resolved with another addition to .cargo/config.toml
:
[build]
rustflags = [
"-C", "link-args=-Wl,-rpath,/Library/Developer/CommandLineTools/Library/Frameworks",
]
Alternatively, on rust >= 1.56, one can include in build.rs
:
fn main() {
println!(
"cargo:rustc-link-arg=-Wl,-rpath,/Library/Developer/CommandLineTools/Library/Frameworks"
);
}
For more discussion on and workarounds for MacOS linking problems see this issue.
Finally, don't forget that on MacOS the extension-module
feature will cause cargo test
to fail without the --no-default-features
flag (see the FAQ).
The extension-module
feature
PyO3's extension-module
feature is used to disable linking to libpython
on Unix targets.
This is necessary because by default PyO3 links to libpython
. This makes binaries, tests, and examples "just work". However, Python extensions on Unix must not link to libpython for manylinux compliance.
The downside of not linking to libpython
is that binaries, tests, and examples (which usually embed Python) will fail to build. If you have an extension module as well as other outputs in a single project, you need to use optional Cargo features to disable the extension-module
when you're not building the extension module. See the FAQ for an example workaround.
Py_LIMITED_API
/abi3
By default, Python extension modules can only be used with the same Python version they were compiled against. For example, an extension module built for Python 3.5 can't be imported in Python 3.8. PEP 384 introduced the idea of the limited Python API, which would have a stable ABI enabling extension modules built with it to be used against multiple Python versions. This is also known as abi3
.
The advantage of building extension modules using the limited Python API is that package vendors only need to build and distribute a single copy (for each OS / architecture), and users can install it on all Python versions from the minimum version and up. The downside of this is that PyO3 can't use optimizations which rely on being compiled against a known exact Python version. It's up to you to decide whether this matters for your extension module. It's also possible to design your extension module such that you can distribute abi3
wheels but allow users compiling from source to benefit from additional optimizations - see the support for multiple python versions section of this guide, in particular the #[cfg(Py_LIMITED_API)]
flag.
There are three steps involved in making use of abi3
when building Python packages as wheels:
- Enable the
abi3
feature inpyo3
. This ensurespyo3
only calls Python C-API functions which are part of the stable API, and on Windows also ensures that the project links against the correct shared object (no special behavior is required on other platforms):
[dependencies]
pyo3 = { version = "0.21.1", features = ["abi3"] }
-
Ensure that the built shared objects are correctly marked as
abi3
. This is accomplished by telling your build system that you're using the limited API.maturin
>= 0.9.0 andsetuptools-rust
>= 0.11.4 supportabi3
wheels. See the corresponding PRs for more. -
Ensure that the
.whl
is correctly marked asabi3
. For projects usingsetuptools
, this is accomplished by passing--py-limited-api=cp3x
(wherex
is the minimum Python version supported by the wheel, e.g.--py-limited-api=cp35
for Python 3.5) tosetup.py bdist_wheel
.
Minimum Python version for abi3
Because a single abi3
wheel can be used with many different Python versions, PyO3 has feature flags abi3-py37
, abi3-py38
, abi3-py39
etc. to set the minimum required Python version for your abi3
wheel.
For example, if you set the abi3-py37
feature, your extension wheel can be used on all Python 3 versions from Python 3.7 and up. maturin
and setuptools-rust
will give the wheel a name like my-extension-1.0-cp37-abi3-manylinux2020_x86_64.whl
.
As your extension module may be run with multiple different Python versions you may occasionally find you need to check the Python version at runtime to customize behavior. See the relevant section of this guide on supporting multiple Python versions at runtime.
PyO3 is only able to link your extension module to abi3 version up to and including your host Python version. E.g., if you set abi3-py38
and try to compile the crate with a host of Python 3.7, the build will fail.
Note: If you set more that one of these
abi3
version feature flags the lowest version always wins. For example, with bothabi3-py37
andabi3-py38
set, PyO3 would build a wheel which supports Python 3.7 and up.
Building abi3
extensions without a Python interpreter
As an advanced feature, you can build PyO3 wheel without calling Python interpreter with the environment variable PYO3_NO_PYTHON
set.
Also, if the build host Python interpreter is not found or is too old or otherwise unusable,
PyO3 will still attempt to compile abi3
extension modules after displaying a warning message.
On Unix-like systems this works unconditionally; on Windows you must also set the RUSTFLAGS
environment variable
to contain -L native=/path/to/python/libs
so that the linker can find python3.lib
.
If the python3.dll
import library is not available, an experimental generate-import-lib
crate
feature may be enabled, and the required library will be created and used by PyO3 automatically.
Note: MSVC targets require LLVM binutils (llvm-dlltool
) to be available in PATH
for
the automatic import library generation feature to work.
Missing features
Due to limitations in the Python API, there are a few pyo3
features that do
not work when compiling for abi3
. These are:
#[pyo3(text_signature = "...")]
does not work on classes until Python 3.10 or greater.- The
dict
andweakref
options on classes are not supported until Python 3.9 or greater. - The buffer API is not supported until Python 3.11 or greater.
- Optimizations which rely on knowledge of the exact Python version compiled against.
Embedding Python in Rust
If you want to embed the Python interpreter inside a Rust program, there are two modes in which this can be done: dynamically and statically. We'll cover each of these modes in the following sections. Each of them affect how you must distribute your program. Instead of learning how to do this yourself, you might want to consider using a project like PyOxidizer to ship your application and all of its dependencies in a single file.
PyO3 automatically switches between the two linking modes depending on whether the Python distribution you have configured PyO3 to use (see above) contains a shared library or a static library. The static library is most often seen in Python distributions compiled from source without the --enable-shared
configuration option. For example, this is the default for pyenv
on macOS.
Dynamically embedding the Python interpreter
Embedding the Python interpreter dynamically is much easier than doing so statically. This is done by linking your program against a Python shared library (such as libpython.3.9.so
on UNIX, or python39.dll
on Windows). The implementation of the Python interpreter resides inside the shared library. This means that when the OS runs your Rust program it also needs to be able to find the Python shared library.
This mode of embedding works well for Rust tests which need access to the Python interpreter. It is also great for Rust software which is installed inside a Python virtualenv, because the virtualenv sets up appropriate environment variables to locate the correct Python shared library.
For distributing your program to non-technical users, you will have to consider including the Python shared library in your distribution as well as setting up wrapper scripts to set the right environment variables (such as LD_LIBRARY_PATH
on UNIX, or PATH
on Windows).
Note that PyPy cannot be embedded in Rust (or any other software). Support for this is tracked on the PyPy issue tracker.
Statically embedding the Python interpreter
Embedding the Python interpreter statically means including the contents of a Python static library directly inside your Rust binary. This means that to distribute your program you only need to ship your binary file: it contains the Python interpreter inside the binary!
On Windows static linking is almost never done, so Python distributions don't usually include a static library. The information below applies only to UNIX.
The Python static library is usually called libpython.a
.
Static linking has a lot of complications, listed below. For these reasons PyO3 does not yet have first-class support for this embedding mode. See issue 416 on PyO3's GitHub for more information and to discuss any issues you encounter.
The auto-initialize
feature is deliberately disabled when embedding the interpreter statically because this is often unintentionally done by new users to PyO3 running test programs. Trying out PyO3 is much easier using dynamic embedding.
The known complications are:
-
To import compiled extension modules (such as other Rust extension modules, or those written in C), your binary must have the correct linker flags set during compilation to export the original contents of
libpython.a
so that extensions can use them (e.g.-Wl,--export-dynamic
). -
The C compiler and flags which were used to create
libpython.a
must be compatible with your Rust compiler and flags, else you will experience compilation failures.Significantly different compiler versions may see errors like this:
lto1: fatal error: bytecode stream in file 'rust-numpy/target/release/deps/libpyo3-6a7fb2ed970dbf26.rlib' generated with LTO version 6.0 instead of the expected 6.2
Mismatching flags may lead to errors like this:
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libpython3.9.a(zlibmodule.o): relocation R_X86_64_32 against `.data' can not be used when making a PIE object; recompile with -fPIE
If you encounter these or other complications when linking the interpreter statically, discuss them on issue 416 on PyO3's GitHub. It is hoped that eventually that discussion will contain enough information and solutions that PyO3 can offer first-class support for static embedding.
Import your module when embedding the Python interpreter
When you run your Rust binary with an embedded interpreter, any #[pymodule]
created modules won't be accessible to import unless added to a table called PyImport_Inittab
before the embedded interpreter is initialized. This will cause Python statements in your embedded interpreter such as import your_new_module
to fail. You can call the macro append_to_inittab
with your module before initializing the Python interpreter to add the module function into that table. (The Python interpreter will be initialized by calling prepare_freethreaded_python
, with_embedded_python_interpreter
, or Python::with_gil
with the auto-initialize
feature enabled.)
Cross Compiling
Thanks to Rust's great cross-compilation support, cross-compiling using PyO3 is relatively straightforward. To get started, you'll need a few pieces of software:
- A toolchain for your target.
- The appropriate options in your Cargo
.config
for the platform you're targeting and the toolchain you are using. - A Python interpreter that's already been compiled for your target (optional when building "abi3" extension modules).
- A Python interpreter that is built for your host and available through the
PATH
or setting thePYO3_PYTHON
variable (optional when building "abi3" extension modules).
After you've obtained the above, you can build a cross-compiled PyO3 module by using Cargo's --target
flag. PyO3's build script will detect that you are attempting a cross-compile based on your host machine and the desired target.
When cross-compiling, PyO3's build script cannot execute the target Python interpreter to query the configuration, so there are a few additional environment variables you may need to set:
PYO3_CROSS
: If present this variable forces PyO3 to configure as a cross-compilation.PYO3_CROSS_LIB_DIR
: This variable can be set to the directory containing the target's libpython DSO and the associated_sysconfigdata*.py
file for Unix-like targets, or the Python DLL import libraries for the Windows target. This variable is only needed when the output binary must link to libpython explicitly (e.g. when targeting Windows and Android or embedding a Python interpreter), or when it is absolutely required to get the interpreter configuration from_sysconfigdata*.py
.PYO3_CROSS_PYTHON_VERSION
: Major and minor version (e.g. 3.9) of the target Python installation. This variable is only needed if PyO3 cannot determine the version to target fromabi3-py3*
features, or ifPYO3_CROSS_LIB_DIR
is not set, or if there are multiple versions of Python present inPYO3_CROSS_LIB_DIR
.PYO3_CROSS_PYTHON_IMPLEMENTATION
: Python implementation name ("CPython" or "PyPy") of the target Python installation. CPython is assumed by default when this variable is not set, unlessPYO3_CROSS_LIB_DIR
is set for a Unix-like target and PyO3 can get the interpreter configuration from_sysconfigdata*.py
.
An experimental pyo3
crate feature generate-import-lib
enables the user to cross-compile
extension modules for Windows targets without setting the PYO3_CROSS_LIB_DIR
environment
variable or providing any Windows Python library files. It uses an external python3-dll-a
crate
to generate import libraries for the Python DLL for MinGW-w64 and MSVC compile targets.
python3-dll-a
uses the binutils dlltool
program to generate DLL import libraries for MinGW-w64 targets.
It is possible to override the default dlltool
command name for the cross target
by setting PYO3_MINGW_DLLTOOL
environment variable.
Note: MSVC targets require LLVM binutils or MSVC build tools to be available on the host system.
More specifically, python3-dll-a
requires llvm-dlltool
or lib.exe
executable to be present in PATH
when
targeting *-pc-windows-msvc
. The Zig compiler executable can be used in place of llvm-dlltool
when the ZIG_COMMAND
environment variable is set to the installed Zig program name ("zig"
or "python -m ziglang"
).
An example might look like the following (assuming your target's sysroot is at /home/pyo3/cross/sysroot
and that your target is armv7
):
export PYO3_CROSS_LIB_DIR="/home/pyo3/cross/sysroot/usr/lib"
cargo build --target armv7-unknown-linux-gnueabihf
If there are multiple python versions at the cross lib directory and you cannot set a more precise location to include both the libpython
DSO and _sysconfigdata*.py
files, you can set the required version:
export PYO3_CROSS_PYTHON_VERSION=3.8
export PYO3_CROSS_LIB_DIR="/home/pyo3/cross/sysroot/usr/lib"
cargo build --target armv7-unknown-linux-gnueabihf
Or another example with the same sys root but building for Windows:
export PYO3_CROSS_PYTHON_VERSION=3.9
export PYO3_CROSS_LIB_DIR="/home/pyo3/cross/sysroot/usr/lib"
cargo build --target x86_64-pc-windows-gnu
Any of the abi3-py3*
features can be enabled instead of setting PYO3_CROSS_PYTHON_VERSION
in the above examples.
PYO3_CROSS_LIB_DIR
can often be omitted when cross compiling extension modules for Unix and macOS targets,
or when cross compiling extension modules for Windows and the experimental generate-import-lib
crate feature is enabled.
The following resources may also be useful for cross-compiling:
- github.com/japaric/rust-cross is a primer on cross compiling Rust.
- github.com/rust-embedded/cross uses Docker to make Rust cross-compilation easier.
Supporting multiple Python versions
PyO3 supports all actively-supported Python 3 and PyPy versions. As much as possible, this is done internally to PyO3 so that your crate's code does not need to adapt to the differences between each version. However, as Python features grow and change between versions, PyO3 cannot a completely identical API for every Python version. This may require you to add conditional compilation to your crate or runtime checks for the Python version.
This section of the guide first introduces the pyo3-build-config
crate, which you can use as a build-dependency
to add additional #[cfg]
flags which allow you to support multiple Python versions at compile-time.
Second, we'll show how to check the Python version at runtime. This can be useful when building for multiple versions with the abi3
feature, where the Python API compiled against is not always the same as the one in use.
Conditional compilation for different Python versions
The pyo3-build-config
exposes multiple #[cfg]
flags which can be used to conditionally compile code for a given Python version. PyO3 itself depends on this crate, so by using it you can be sure that you are configured correctly for the Python version PyO3 is building against.
This allows us to write code like the following
#[cfg(Py_3_7)]
fn function_only_supported_on_python_3_7_and_up() {}
#[cfg(not(Py_3_8))]
fn function_only_supported_before_python_3_8() {}
#[cfg(not(Py_LIMITED_API))]
fn function_incompatible_with_abi3_feature() {}
The following sections first show how to add these #[cfg]
flags to your build process, and then cover some common patterns flags in a little more detail.
To see a full reference of all the #[cfg]
flags provided, see the pyo3-build-cfg
docs.
Using pyo3-build-config
You can use the #[cfg]
flags in just two steps:
-
Add
pyo3-build-config
with theresolve-config
feature enabled to your crate's build dependencies inCargo.toml
:[build-dependencies] pyo3-build-config = { version = "0.21.1", features = ["resolve-config"] }
-
Add a
build.rs
file to your crate with the following contents:fn main() { // If you have an existing build.rs file, just add this line to it. pyo3_build_config::use_pyo3_cfgs(); }
After these steps you are ready to annotate your code!
Common usages of pyo3-build-cfg
flags
The #[cfg]
flags added by pyo3-build-cfg
can be combined with all of Rust's logic in the #[cfg]
attribute to create very precise conditional code generation. The following are some common patterns implemented using these flags:
#[cfg(Py_3_7)]
This #[cfg]
marks code that will only be present on Python 3.7 and upwards. There are similar options Py_3_8
, Py_3_9
, Py_3_10
and so on for each minor version.
#[cfg(not(Py_3_7))]
This #[cfg]
marks code that will only be present on Python versions before (but not including) Python 3.7.
#[cfg(not(Py_LIMITED_API))]
This #[cfg]
marks code that is only available when building for the unlimited Python API (i.e. PyO3's abi3
feature is not enabled). This might be useful if you want to ship your extension module as an abi3
wheel and also allow users to compile it from source to make use of optimizations only possible with the unlimited API.
#[cfg(any(Py_3_9, not(Py_LIMITED_API)))]
This #[cfg]
marks code which is available when running Python 3.9 or newer, or when using the unlimited API with an older Python version. Patterns like this are commonly seen on Python APIs which were added to the limited Python API in a specific minor version.
#[cfg(PyPy)]
This #[cfg]
marks code which is running on PyPy.
Checking the Python version at runtime
When building with PyO3's abi3
feature, your extension module will be compiled against a specific minimum version of Python, but may be running on newer Python versions.
For example with PyO3's abi3-py38
feature, your extension will be compiled as if it were for Python 3.8. If you were using pyo3-build-config
, #[cfg(Py_3_8)]
would be present. Your user could freely install and run your abi3 extension on Python 3.9.
There's no way to detect your user doing that at compile time, so instead you need to fall back to runtime checks.
PyO3 provides the APIs Python::version()
and Python::version_info()
to query the running Python version. This allows you to do the following, for example:
use pyo3::Python;
Python::with_gil(|py| {
// PyO3 supports Python 3.7 and up.
assert!(py.version_info() >= (3, 7));
assert!(py.version_info() >= (3, 7, 0));
});
The PyO3 ecosystem
This portion of the guide is dedicated to crates which are external to the main PyO3 project and provide additional functionality you might find useful.
Because these projects evolve independently of the PyO3 repository the content of these articles may fall out of date over time; please file issues on the PyO3 GitHub to alert maintainers when this is the case.
Logging
It is desirable if both the Python and Rust parts of the application end up logging using the same configuration into the same place.
This section of the guide briefly discusses how to connect the two languages'
logging ecosystems together. The recommended way for Python extension modules is
to configure Rust's logger to send log messages to Python using the pyo3-log
crate. For users who want to do the opposite and send Python log messages to
Rust, see the note at the end of this guide.
Using pyo3-log
to send Rust log messages to Python
The pyo3-log crate allows sending the messages from the Rust side to Python's logging system. This is mostly suitable for writing native extensions for Python programs.
Use pyo3_log::init
to install the logger in its default configuration.
It's also possible to tweak its configuration (mostly to tune its performance).
use log::info;
use pyo3::prelude::*;
#[pyfunction]
fn log_something() {
// This will use the logger installed in `my_module` to send the `info`
// message to the Python logging facilities.
info!("Something!");
}
#[pymodule]
fn my_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
// A good place to install the Rust -> Python logger.
pyo3_log::init();
m.add_function(wrap_pyfunction!(log_something, m)?)?;
Ok(())
}
Then it is up to the Python side to actually output the messages somewhere.
import logging
import my_module
FORMAT = '%(levelname)s %(name)s %(asctime)-15s %(filename)s:%(lineno)d %(message)s'
logging.basicConfig(format=FORMAT)
logging.getLogger().setLevel(logging.INFO)
my_module.log_something()
It is important to initialize the Python loggers first, before calling any Rust functions that may log. This limitation can be worked around if it is not possible to satisfy, read the documentation about caching.
The Python to Rust direction
To have python logs be handled by Rust, one need only register a rust function to handle logs emitted from the core python logging module.
This has been implemented within the pyo3-pylogger crate.
use log::{info, warn};
use pyo3::prelude::*;
fn main() -> PyResult<()> {
// register the host handler with python logger, providing a logger target
// set the name here to something appropriate for your application
pyo3_pylogger::register("example_application_py_logger");
// initialize up a logger
env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("trace")).init();
// Log some messages from Rust.
info!("Just some normal information!");
warn!("Something spooky happened!");
// Log some messages from Python
Python::with_gil(|py| {
py.run(
"
import logging
logging.error('Something bad happened')
",
None,
None,
)
})
}
Using async
and await
async
/await
support is currently being integrated in PyO3. See the dedicated documentation
If you are working with a Python library that makes use of async functions or wish to provide
Python bindings for an async Rust library, pyo3-asyncio
likely has the tools you need. It provides conversions between async functions in both Python and
Rust and was designed with first-class support for popular Rust runtimes such as
tokio
and async-std
. In addition, all async Python
code runs on the default asyncio
event loop, so pyo3-asyncio
should work just fine with existing
Python libraries.
In the following sections, we'll give a general overview of pyo3-asyncio
explaining how to call
async Python functions with PyO3, how to call async Rust functions from Python, and how to configure
your codebase to manage the runtimes of both.
Quickstart
Here are some examples to get you started right away! A more detailed breakdown of the concepts in these examples can be found in the following sections.
Rust Applications
Here we initialize the runtime, import Python's asyncio
library and run the given future to completion using Python's default EventLoop
and async-std
. Inside the future, we convert asyncio
sleep into a Rust future and await it.
# Cargo.toml dependencies
[dependencies]
pyo3 = { version = "0.14" }
pyo3-asyncio = { version = "0.14", features = ["attributes", "async-std-runtime"] }
async-std = "1.9"
//! main.rs
use pyo3::prelude::*;
#[pyo3_asyncio::async_std::main]
async fn main() -> PyResult<()> {
let fut = Python::with_gil(|py| {
let asyncio = py.import("asyncio")?;
// convert asyncio.sleep into a Rust Future
pyo3_asyncio::async_std::into_future(asyncio.call_method1("sleep", (1.into_py(py),))?)
})?;
fut.await?;
Ok(())
}
The same application can be written to use tokio
instead using the #[pyo3_asyncio::tokio::main]
attribute.
# Cargo.toml dependencies
[dependencies]
pyo3 = { version = "0.14" }
pyo3-asyncio = { version = "0.14", features = ["attributes", "tokio-runtime"] }
tokio = "1.4"
//! main.rs
use pyo3::prelude::*;
#[pyo3_asyncio::tokio::main]
async fn main() -> PyResult<()> {
let fut = Python::with_gil(|py| {
let asyncio = py.import("asyncio")?;
// convert asyncio.sleep into a Rust Future
pyo3_asyncio::tokio::into_future(asyncio.call_method1("sleep", (1.into_py(py),))?)
})?;
fut.await?;
Ok(())
}
More details on the usage of this library can be found in the API docs and the primer below.
PyO3 Native Rust Modules
PyO3 Asyncio can also be used to write native modules with async functions.
Add the [lib]
section to Cargo.toml
to make your library a cdylib
that Python can import.
[lib]
name = "my_async_module"
crate-type = ["cdylib"]
Make your project depend on pyo3
with the extension-module
feature enabled and select your
pyo3-asyncio
runtime:
For async-std
:
[dependencies]
pyo3 = { version = "0.14", features = ["extension-module"] }
pyo3-asyncio = { version = "0.14", features = ["async-std-runtime"] }
async-std = "1.9"
For tokio
:
[dependencies]
pyo3 = { version = "0.14", features = ["extension-module"] }
pyo3-asyncio = { version = "0.14", features = ["tokio-runtime"] }
tokio = "1.4"
Export an async function that makes use of async-std
:
//! lib.rs
use pyo3::{prelude::*, wrap_pyfunction};
#[pyfunction]
fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
pyo3_asyncio::async_std::future_into_py(py, async {
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
Ok(Python::with_gil(|py| py.None()))
})
}
#[pymodule]
fn my_async_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(rust_sleep, m)?)?;
Ok(())
}
If you want to use tokio
instead, here's what your module should look like:
//! lib.rs
use pyo3::{prelude::*, wrap_pyfunction};
#[pyfunction]
fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
pyo3_asyncio::tokio::future_into_py(py, async {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
Ok(Python::with_gil(|py| py.None()))
})
}
#[pymodule]
fn my_async_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(rust_sleep, m)?)?;
Ok(())
}
You can build your module with maturin (see the Using Rust in Python section in the PyO3 guide for setup instructions). After that you should be able to run the Python REPL to try it out.
maturin develop && python3
🔗 Found pyo3 bindings
🐍 Found CPython 3.8 at python3
Finished dev [unoptimized + debuginfo] target(s) in 0.04s
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>>
>>> from my_async_module import rust_sleep
>>>
>>> async def main():
>>> await rust_sleep()
>>>
>>> # should sleep for 1s
>>> asyncio.run(main())
>>>
Awaiting an Async Python Function in Rust
Let's take a look at a dead simple async Python function:
# Sleep for 1 second
async def py_sleep():
await asyncio.sleep(1)
Async functions in Python are simply functions that return a coroutine
object. For our purposes,
we really don't need to know much about these coroutine
objects. The key factor here is that calling
an async
function is just like calling a regular function, the only difference is that we have
to do something special with the object that it returns.
Normally in Python, that something special is the await
keyword, but in order to await this
coroutine in Rust, we first need to convert it into Rust's version of a coroutine
: a Future
.
That's where pyo3-asyncio
comes in.
pyo3_asyncio::async_std::into_future
performs this conversion for us.
The following example uses into_future
to call the py_sleep
function shown above and then await the
coroutine object returned from the call:
use pyo3::prelude::*;
#[pyo3_asyncio::tokio::main]
async fn main() -> PyResult<()> {
let future = Python::with_gil(|py| -> PyResult<_> {
// import the module containing the py_sleep function
let example = py.import("example")?;
// calling the py_sleep method like a normal function
// returns a coroutine
let coroutine = example.call_method0("py_sleep")?;
// convert the coroutine into a Rust future using the
// tokio runtime
pyo3_asyncio::tokio::into_future(coroutine)
})?;
// await the future
future.await?;
Ok(())
}
Alternatively, the below example shows how to write a #[pyfunction]
which uses into_future
to receive and await
a coroutine argument:
#[pyfunction]
fn await_coro(coro: &Bound<'_, PyAny>>) -> PyResult<()> {
// convert the coroutine into a Rust future using the
// async_std runtime
let f = pyo3_asyncio::async_std::into_future(coro)?;
pyo3_asyncio::async_std::run_until_complete(coro.py(), async move {
// await the future
f.await?;
Ok(())
})
}
This could be called from Python as:
import asyncio
async def py_sleep():
asyncio.sleep(1)
await_coro(py_sleep())
If for you wanted to pass a callable function to the #[pyfunction]
instead, (i.e. the last line becomes await_coro(py_sleep))
, then the above example needs to be tweaked to first call the callable to get the coroutine:
#[pyfunction]
fn await_coro(callable: &Bound<'_, PyAny>>) -> PyResult<()> {
// get the coroutine by calling the callable
let coro = callable.call0()?;
// convert the coroutine into a Rust future using the
// async_std runtime
let f = pyo3_asyncio::async_std::into_future(coro)?;
pyo3_asyncio::async_std::run_until_complete(coro.py(), async move {
// await the future
f.await?;
Ok(())
})
}
This can be particularly helpful where you need to repeatedly create and await a coroutine. Trying to await the same coroutine multiple times will raise an error:
RuntimeError: cannot reuse already awaited coroutine
If you're interested in learning more about
coroutines
andawaitables
in general, check out the Python 3asyncio
docs for more information.
Awaiting a Rust Future in Python
Here we have the same async function as before written in Rust using the
async-std
runtime:
/// Sleep for 1 second
async fn rust_sleep() {
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
}
Similar to Python, Rust's async functions also return a special object called a
Future
:
let future = rust_sleep();
We can convert this Future
object into Python to make it awaitable
. This tells Python that you
can use the await
keyword with it. In order to do this, we'll call
pyo3_asyncio::async_std::future_into_py
:
use pyo3::prelude::*;
async fn rust_sleep() {
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
}
#[pyfunction]
fn call_rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
pyo3_asyncio::async_std::future_into_py(py, async move {
rust_sleep().await;
Ok(Python::with_gil(|py| py.None()))
})
}
In Python, we can call this pyo3 function just like any other async function:
from example import call_rust_sleep
async def rust_sleep():
await call_rust_sleep()
Managing Event Loops
Python's event loop requires some special treatment, especially regarding the main thread. Some of
Python's asyncio
features, like proper signal handling, require control over the main thread, which
doesn't always play well with Rust.
Luckily, Rust's event loops are pretty flexible and don't need control over the main thread, so in
pyo3-asyncio
, we decided the best way to handle Rust/Python interop was to just surrender the main
thread to Python and run Rust's event loops in the background. Unfortunately, since most event loop
implementations prefer control over the main thread, this can still make some things awkward.
PyO3 Asyncio Initialization
Because Python needs to control the main thread, we can't use the convenient proc macros from Rust
runtimes to handle the main
function or #[test]
functions. Instead, the initialization for PyO3 has to be done from the main
function and the main
thread must block on pyo3_asyncio::async_std::run_until_complete
.
Because we have to block on one of those functions, we can't use #[async_std::main]
or #[tokio::main]
since it's not a good idea to make long blocking calls during an async function.
Internally, these
#[main]
proc macros are expanded to something like this:fn main() { // your async main fn async fn _main_impl() { /* ... */ } Runtime::new().block_on(_main_impl()); }
Making a long blocking call inside the
Future
that's being driven byblock_on
prevents that thread from doing anything else and can spell trouble for some runtimes (also this will actually deadlock a single-threaded runtime!). Many runtimes have some sort ofspawn_blocking
mechanism that can avoid this problem, but again that's not something we can use here since we need it to block on the main thread.
For this reason, pyo3-asyncio
provides its own set of proc macros to provide you with this
initialization. These macros are intended to mirror the initialization of async-std
and tokio
while also satisfying the Python runtime's needs.
Here's a full example of PyO3 initialization with the async-std
runtime:
use pyo3::prelude::*;
#[pyo3_asyncio::async_std::main]
async fn main() -> PyResult<()> {
// PyO3 is initialized - Ready to go
let fut = Python::with_gil(|py| -> PyResult<_> {
let asyncio = py.import("asyncio")?;
// convert asyncio.sleep into a Rust Future
pyo3_asyncio::async_std::into_future(
asyncio.call_method1("sleep", (1.into_py(py),))?
)
})?;
fut.await?;
Ok(())
}
A Note About asyncio.run
In Python 3.7+, the recommended way to run a top-level coroutine with asyncio
is with asyncio.run
. In v0.13
we recommended against using this function due to initialization issues, but in v0.14
it's perfectly valid to use this function... with a caveat.
Since our Rust <--> Python conversions require a reference to the Python event loop, this poses a problem. Imagine we have a PyO3 Asyncio module that defines
a rust_sleep
function like in previous examples. You might rightfully assume that you can call pass this directly into asyncio.run
like this:
import asyncio
from my_async_module import rust_sleep
asyncio.run(rust_sleep())
You might be surprised to find out that this throws an error:
Traceback (most recent call last):
File "example.py", line 5, in <module>
asyncio.run(rust_sleep())
RuntimeError: no running event loop
What's happening here is that we are calling rust_sleep
before the future is
actually running on the event loop created by asyncio.run
. This is counter-intuitive, but expected behaviour, and unfortunately there doesn't seem to be a good way of solving this problem within PyO3 Asyncio itself.
However, we can make this example work with a simple workaround:
import asyncio
from my_async_module import rust_sleep
# Calling main will just construct the coroutine that later calls rust_sleep.
# - This ensures that rust_sleep will be called when the event loop is running,
# not before.
async def main():
await rust_sleep()
# Run the main() coroutine at the top-level instead
asyncio.run(main())
Non-standard Python Event Loops
Python allows you to use alternatives to the default asyncio
event loop. One
popular alternative is uvloop
. In v0.13
using non-standard event loops was
a bit of an ordeal, but in v0.14
it's trivial.
Using uvloop
in a PyO3 Asyncio Native Extensions
# Cargo.toml
[lib]
name = "my_async_module"
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.14", features = ["extension-module"] }
pyo3-asyncio = { version = "0.14", features = ["tokio-runtime"] }
async-std = "1.9"
tokio = "1.4"
//! lib.rs
use pyo3::{prelude::*, wrap_pyfunction};
#[pyfunction]
fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
pyo3_asyncio::tokio::future_into_py(py, async {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
Ok(Python::with_gil(|py| py.None()))
})
}
#[pymodule]
fn my_async_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(rust_sleep, m)?)?;
Ok(())
}
$ maturin develop && python3
🔗 Found pyo3 bindings
🐍 Found CPython 3.8 at python3
Finished dev [unoptimized + debuginfo] target(s) in 0.04s
Python 3.8.8 (default, Apr 13 2021, 19:58:26)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> import uvloop
>>>
>>> import my_async_module
>>>
>>> uvloop.install()
>>>
>>> async def main():
... await my_async_module.rust_sleep()
...
>>> asyncio.run(main())
>>>
Using uvloop
in Rust Applications
Using uvloop
in Rust applications is a bit trickier, but it's still possible
with relatively few modifications.
Unfortunately, we can't make use of the #[pyo3_asyncio::<runtime>::main]
attribute with non-standard event loops. This is because the #[pyo3_asyncio::<runtime>::main]
proc macro has to interact with the Python
event loop before we can install the uvloop
policy.
[dependencies]
async-std = "1.9"
pyo3 = "0.14"
pyo3-asyncio = { version = "0.14", features = ["async-std-runtime"] }
//! main.rs
use pyo3::{prelude::*, types::PyType};
fn main() -> PyResult<()> {
pyo3::prepare_freethreaded_python();
Python::with_gil(|py| {
let uvloop = py.import("uvloop")?;
uvloop.call_method0("install")?;
// store a reference for the assertion
let uvloop = PyObject::from(uvloop);
pyo3_asyncio::async_std::run(py, async move {
// verify that we are on a uvloop.Loop
Python::with_gil(|py| -> PyResult<()> {
assert!(pyo3_asyncio::async_std::get_current_loop(py)?.is_instance(
uvloop
.as_ref(py)
.getattr("Loop")?
)?);
Ok(())
})?;
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
Ok(())
})
})
}
Additional Information
- Managing event loop references can be tricky with pyo3-asyncio. See Event Loop References in the API docs to get a better intuition for how event loop references are managed in this library.
- Testing pyo3-asyncio libraries and applications requires a custom test harness since Python requires control over the main thread. You can find a testing guide in the API docs for the
testing
module
Frequently Asked Questions and troubleshooting
Sorry that you're having trouble using PyO3. If you can't find the answer to your problem in the list below, you can also reach out for help on GitHub Discussions and on Discord.
I'm experiencing deadlocks using PyO3 with lazy_static or once_cell!
lazy_static
and once_cell::sync
both use locks to ensure that initialization is performed only by a single thread. Because the Python GIL is an additional lock this can lead to deadlocks in the following way:
- A thread (thread A) which has acquired the Python GIL starts initialization of a
lazy_static
value. - The initialization code calls some Python API which temporarily releases the GIL e.g.
Python::import
. - Another thread (thread B) acquires the Python GIL and attempts to access the same
lazy_static
value. - Thread B is blocked, because it waits for
lazy_static
's initialization to lock to release. - Thread A is blocked, because it waits to re-acquire the GIL which thread B still holds.
- Deadlock.
PyO3 provides a struct GILOnceCell
which works equivalently to OnceCell
but relies solely on the Python GIL for thread safety. This means it can be used in place of lazy_static
or once_cell
where you are experiencing the deadlock described above. See the documentation for GILOnceCell
for an example how to use it.
I can't run cargo test
; or I can't build in a Cargo workspace: I'm having linker issues like "Symbol not found" or "Undefined reference to _PyExc_SystemError"!
Currently, #340 causes cargo test
to fail with linking errors when the extension-module
feature is activated. Linking errors can also happen when building in a cargo workspace where a different crate also uses PyO3 (see #2521). For now, there are three ways we can work around these issues.
- Make the
extension-module
feature optional. Build withmaturin develop --features "extension-module"
[dependencies.pyo3]
version = "0.21.1"
[features]
extension-module = ["pyo3/extension-module"]
- Make the
extension-module
feature optional and default. Run tests withcargo test --no-default-features
:
[dependencies.pyo3]
version = "0.21.1"
[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]
- If you are using a
pyproject.toml
file to control maturin settings, add the following section:
[tool.maturin]
features = ["pyo3/extension-module"]
# Or for maturin 0.12:
# cargo-extra-args = ["--features", "pyo3/extension-module"]
I can't run cargo test
: my crate cannot be found for tests in tests/
directory!
The Rust book suggests to put integration tests inside a tests/
directory.
For a PyO3 extension-module
project where the crate-type
is set to "cdylib"
in your Cargo.toml
,
the compiler won't be able to find your crate and will display errors such as E0432
or E0463
:
error[E0432]: unresolved import `my_crate`
--> tests/test_my_crate.rs:1:5
|
1 | use my_crate;
| ^^^^^^^^^^^^ no external crate `my_crate`
The best solution is to make your crate types include both rlib
and cdylib
:
# Cargo.toml
[lib]
crate-type = ["cdylib", "rlib"]
Ctrl-C doesn't do anything while my Rust code is executing!
This is because Ctrl-C raises a SIGINT signal, which is handled by the calling Python process by simply setting a flag to action upon later. This flag isn't checked while Rust code called from Python is executing, only once control returns to the Python interpreter.
You can give the Python interpreter a chance to process the signal properly by calling Python::check_signals
. It's good practice to call this function regularly if you have a long-running Rust function so that your users can cancel it.
#[pyo3(get)]
clones my field!
You may have a nested struct similar to this:
use pyo3::prelude::*;
#[pyclass]
#[derive(Clone)]
struct Inner {/* fields omitted */}
#[pyclass]
struct Outer {
#[pyo3(get)]
inner: Inner,
}
#[pymethods]
impl Outer {
#[new]
fn __new__() -> Self {
Self { inner: Inner {} }
}
}
When Python code accesses Outer
's field, PyO3 will return a new object on every access (note that their addresses are different):
outer = Outer()
a = outer.inner
b = outer.inner
assert a is b, f"a: {a}\nb: {b}"
AssertionError: a: <builtins.Inner object at 0x00000238FFB9C7B0>
b: <builtins.Inner object at 0x00000238FFB9C830>
This can be especially confusing if the field is mutable, as getting the field and then mutating it won't persist - you'll just get a fresh clone of the original on the next access. Unfortunately Python and Rust don't agree about ownership - if PyO3 gave out references to (possibly) temporary Rust objects to Python code, Python code could then keep that reference alive indefinitely. Therefore returning Rust objects requires cloning.
If you don't want that cloning to happen, a workaround is to allocate the field on the Python heap and store a reference to that, by using Py<...>
:
use pyo3::prelude::*;
#[pyclass]
#[derive(Clone)]
struct Inner {/* fields omitted */}
#[pyclass]
struct Outer {
#[pyo3(get)]
inner: Py<Inner>,
}
#[pymethods]
impl Outer {
#[new]
fn __new__(py: Python<'_>) -> PyResult<Self> {
Ok(Self {
inner: Py::new(py, Inner {})?,
})
}
}
This time a
and b
are the same object:
outer = Outer()
a = outer.inner
b = outer.inner
assert a is b, f"a: {a}\nb: {b}"
print(f"a: {a}\nb: {b}")
a: <builtins.Inner object at 0x0000020044FCC670>
b: <builtins.Inner object at 0x0000020044FCC670>
The downside to this approach is that any Rust code working on the Outer
struct now has to acquire the GIL to do anything with its field.
I want to use the pyo3
crate re-exported from dependency but the proc-macros fail!
All PyO3 proc-macros (#[pyclass]
, #[pyfunction]
, #[derive(FromPyObject)]
and so on) expect the pyo3
crate to be available under that name in your crate
root, which is the normal situation when pyo3
is a direct dependency of your
crate.
However, when the dependency is renamed, or your crate only indirectly depends
on pyo3
, you need to let the macro code know where to find the crate. This is
done with the crate
attribute:
use pyo3::prelude::*;
pub extern crate pyo3;
mod reexported { pub use ::pyo3; }
#[pyclass]
#[pyo3(crate = "reexported::pyo3")]
struct MyClass;
I'm trying to call Python from Rust but I get STATUS_DLL_NOT_FOUND
or STATUS_ENTRYPOINT_NOT_FOUND
!
This happens on Windows when linking to the python DLL fails or the wrong one is linked. The Python DLL on Windows will usually be called something like:
python3X.dll
for Python 3.X, e.g.python310.dll
for Python 3.10python3.dll
when using PyO3'sabi3
feature
The DLL needs to be locatable using the Windows DLL search order. Some ways to achieve this are:
- Put the Python DLL in the same folder as your build artifacts
- Add the directory containing the Python DLL to your
PATH
environment variable, for exampleC:\Users\<You>\AppData\Local\Programs\Python\Python310
- If this happens when you are distributing your program, consider using PyOxidizer to package it with your binary.
If the wrong DLL is linked it is possible that this happened because another program added itself and its own Python DLLs to PATH
. Rearrange your PATH
variables to give the correct DLL priority.
Note: Changes to
PATH
(or any other environment variable) are not visible to existing shells. Restart it for changes to take effect.
For advanced troubleshooting, Dependency Walker can be used to diagnose linking errors.
Migrating from older PyO3 versions
This guide can help you upgrade code through breaking changes from one PyO3 version to the next. For a detailed list of all changes, see the CHANGELOG.
from 0.20.* to 0.21
Click to expand
PyO3 0.21 introduces a new Bound<'py, T>
smart pointer which replaces the existing "GIL Refs" API to interact with Python objects. For example, in PyO3 0.20 the reference &'py PyAny
would be used to interact with Python objects. In PyO3 0.21 the updated type is Bound<'py, PyAny>
. Making this change moves Rust ownership semantics out of PyO3's internals and into user code. This change fixes a known soundness edge case of interaction with gevent as well as improves CPU and memory performance. For a full history of discussion see https://github.com/PyO3/pyo3/issues/3382.
The "GIL Ref" &'py PyAny
and similar types such as &'py PyDict
continue to be available as a deprecated API. Due to the advantages of the new API it is advised that all users make the effort to upgrade as soon as possible.
In addition to the major API type overhaul, PyO3 has needed to make a few small breaking adjustments to other APIs to close correctness and soundness gaps.
The recommended steps to update to PyO3 0.21 is as follows:
- Enable the
gil-refs
feature to silence deprecations related to the API change - Fix all other PyO3 0.21 migration steps
- Disable the
gil-refs
feature and migrate off the deprecated APIs
The following sections are laid out in this order.
Enable the gil-refs
feature
Click to expand
To make the transition for the PyO3 ecosystem away from the GIL Refs API as smooth as possible, in PyO3 0.21 no APIs consuming or producing GIL Refs have been altered. Instead, variants using Bound<T>
smart pointers have been introduced, for example PyTuple::new_bound
which returns Bound<PyTuple>
is the replacement form of PyTuple::new
. The GIL Ref APIs have been deprecated, but to make migration easier it is possible to disable these deprecation warnings by enabling the gil-refs
feature.
The one single exception where an existing API was changed in-place is the
pyo3::intern!
macro. Almost all uses of this macro did not need to update code to account it changing to return&Bound<PyString>
immediately, and adding anintern_bound!
replacement was perceived as adding more work for users.
It is recommended that users do this as a first step of updating to PyO3 0.21 so that the deprecation warnings do not get in the way of resolving the rest of the migration steps.
Before:
# Cargo.toml
[dependencies]
pyo3 = "0.20"
After:
# Cargo.toml
[dependencies]
pyo3 = { version = "0.21", features = ["gil-refs"] }
PyTypeInfo
and PyTryFrom
have been adjusted
Click to expand
The PyTryFrom
trait has aged poorly, its try_from
method now conflicts with TryFrom::try_from
in the 2021 edition prelude. A lot of its functionality was also duplicated with PyTypeInfo
.
To tighten up the PyO3 traits as part of the deprecation of the GIL Refs API the PyTypeInfo
trait has had a simpler companion PyTypeCheck
. The methods PyAny::downcast
and PyAny::downcast_exact
no longer use PyTryFrom
as a bound, instead using PyTypeCheck
and PyTypeInfo
respectively.
To migrate, switch all type casts to use obj.downcast()
instead of try_from(obj)
(and similar for downcast_exact
).
Before:
#![allow(deprecated)]
use pyo3::prelude::*;
use pyo3::types::{PyInt, PyList};
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let list = PyList::new(py, 0..5);
let b = <PyInt as PyTryFrom>::try_from(list.get_item(0).unwrap())?;
Ok(())
})
}
After:
use pyo3::prelude::*;
use pyo3::types::{PyInt, PyList};
fn main() -> PyResult<()> {
Python::with_gil(|py| {
// Note that PyList::new is deprecated for PyList::new_bound as part of the GIL Refs API removal,
// see the section below on migration to Bound<T>.
#[allow(deprecated)]
let list = PyList::new(py, 0..5);
let b = list.get_item(0).unwrap().downcast::<PyInt>()?;
Ok(())
})
}
Iter(A)NextOutput
are deprecated
Click to expand
The __next__
and __anext__
magic methods can now return any type convertible into Python objects directly just like all other #[pymethods]
. The IterNextOutput
used by __next__
and IterANextOutput
used by __anext__
are subsequently deprecated. Most importantly, this change allows returning an awaitable from __anext__
without non-sensically wrapping it into Yield
or Some
. Only the return types Option<T>
and Result<Option<T>, E>
are still handled in a special manner where Some(val)
yields val
and None
stops iteration.
Starting with an implementation of a Python iterator using IterNextOutput
, e.g.
#![allow(deprecated)]
use pyo3::prelude::*;
use pyo3::iter::IterNextOutput;
#[pyclass]
struct PyClassIter {
count: usize,
}
#[pymethods]
impl PyClassIter {
fn __next__(&mut self) -> IterNextOutput<usize, &'static str> {
if self.count < 5 {
self.count += 1;
IterNextOutput::Yield(self.count)
} else {
IterNextOutput::Return("done")
}
}
}
If returning "done"
via StopIteration
is not really required, this should be written as
use pyo3::prelude::*;
#[pyclass]
struct PyClassIter {
count: usize,
}
#[pymethods]
impl PyClassIter {
fn __next__(&mut self) -> Option<usize> {
if self.count < 5 {
self.count += 1;
Some(self.count)
} else {
None
}
}
}
This form also has additional benefits: It has already worked in previous PyO3 versions, it matches the signature of Rust's Iterator
trait and it allows using a fast path in CPython which completely avoids the cost of raising a StopIteration
exception. Note that using Option::transpose
and the Result<Option<T>, E>
variant, this form can also be used to wrap fallible iterators.
Alternatively, the implementation can also be done as it would in Python itself, i.e. by "raising" a StopIteration
exception
use pyo3::prelude::*;
use pyo3::exceptions::PyStopIteration;
#[pyclass]
struct PyClassIter {
count: usize,
}
#[pymethods]
impl PyClassIter {
fn __next__(&mut self) -> PyResult<usize> {
if self.count < 5 {
self.count += 1;
Ok(self.count)
} else {
Err(PyStopIteration::new_err("done"))
}
}
}
Finally, an asynchronous iterator can directly return an awaitable without confusing wrapping
use pyo3::prelude::*;
#[pyclass]
struct PyClassAwaitable {
number: usize,
}
#[pymethods]
impl PyClassAwaitable {
fn __next__(&self) -> usize {
self.number
}
fn __await__(slf: Py<Self>) -> Py<Self> {
slf
}
}
#[pyclass]
struct PyClassAsyncIter {
number: usize,
}
#[pymethods]
impl PyClassAsyncIter {
fn __anext__(&mut self) -> PyClassAwaitable {
self.number += 1;
PyClassAwaitable {
number: self.number,
}
}
fn __aiter__(slf: Py<Self>) -> Py<Self> {
slf
}
}
PyType::name
has been renamed to PyType::qualname
Click to expand
PyType::name
has been renamed to PyType::qualname
to indicate that it does indeed return the qualified name, matching the __qualname__
attribute. The newly added PyType::name
yields the full name including the module name now which corresponds to __module__.__name__
on the level of attributes.
PyCell
has been deprecated
Click to expand
Interactions with Python objects implemented in Rust no longer need to go though PyCell<T>
. Instead iteractions with Python object now consistently go through Bound<T>
or Py<T>
independently of whether T
is native Python object or a #[pyclass]
implemented in Rust. Use Bound::new
or Py::new
respectively to create and Bound::borrow(_mut)
/ Py::borrow(_mut)
to borrow the Rust object.
Migrating from the GIL Refs API to Bound<T>
Click to expand
To minimise breakage of code using the GIL Refs API, the Bound<T>
smart pointer has been introduced by adding complements to all functions which accept or return GIL Refs. This allows code to migrate by replacing the deprecated APIs with the new ones.
To identify what to migrate, temporarily switch off the gil-refs
feature to see deprecation warnings on almost all uses of APIs accepting and producing GIL Refs . Over one or more PRs it should be possible to follow the deprecation hints to update code. Depending on your development environment, switching off the gil-refs
feature may introduce some very targeted breakages, so you may need to fixup those first.
For example, the following APIs have gained updated variants:
PyList::new
,PyTyple::new
and similar constructors have replacementsPyList::new_bound
,PyTuple::new_bound
etc.FromPyObject::extract
has a newFromPyObject::extract_bound
(see the section below)- The
PyTypeInfo
trait has had new_bound
methods added to accept / returnBound<T>
.
Because the new Bound<T>
API brings ownership out of the PyO3 framework and into user code, there are a few places where user code is expected to need to adjust while switching to the new API:
- Code will need to add the occasional
&
to borrow the new smart pointer as&Bound<T>
to pass these types around (or use.clone()
at the very small cost of increasing the Python reference count) Bound<PyList>
andBound<PyTuple>
cannot support indexing withlist[0]
, you should uselist.get_item(0)
instead.Bound<PyTuple>::iter_borrowed
is slightly more efficient thanBound<PyTuple>::iter
. The default iteration ofBound<PyTuple>
cannot return borrowed references because Rust does not (yet) have "lending iterators". SimilarlyBound<PyTuple>::get_borrowed_item
is more efficient thanBound<PyTuple>::get_item
for the same reason.&Bound<T>
does not implementFromPyObject
(although it might be possible to do this in the future once the GIL Refs API is completely removed). Usebound_any.downcast::<T>()
instead ofbound_any.extract::<&Bound<T>>()
.Bound<PyString>::to_str
now borrows from theBound<PyString>
rather than from the'py
lifetime, so code will need to store the smart pointer as a value in some cases where previously&PyString
was just used as a temporary. (There are some more details relating to this in the section below.).extract::<&str>()
now borrows from the source Python object. The simplest way to update is to change to.extract::<PyBackedStr>()
, which retains ownership of the Python reference. See more information in the section on deactivating thegil-refs
feature.
To convert between &PyAny
and &Bound<PyAny>
use the as_borrowed()
method:
let gil_ref: &PyAny = ...;
let bound: &Bound<PyAny> = &gil_ref.as_borrowed();
To convert between Py<T>
and Bound<T>
use the bind()
/ into_bound()
methods, and as_unbound()
/ unbind()
to go back from Bound<T>
to Py<T>
.
let obj: Py<PyList> = ...;
let bound: &Bound<'py, PyList> = obj.bind(py);
let bound: Bound<'py, PyList> = obj.into_bound(py);
let obj: &Py<PyList> = bound.as_unbound();
let obj: Py<PyList> = bound.unbind();
⚠️ Warning: dangling pointer trap 💣
Because of the ownership changes, code which uses
.as_ptr()
to convert&PyAny
and other GIL Refs to a*mut pyo3_ffi::PyObject
should take care to avoid creating dangling pointers now thatBound<PyAny>
carries ownership.For example, the following pattern with
Option<&PyAny>
can easily create a dangling pointer when migrating to theBound<PyAny>
smart pointer:let opt: Option<&PyAny> = ...; let p: *mut ffi::PyObject = opt.map_or(std::ptr::null_mut(), |any| any.as_ptr());
The correct way to migrate this code is to use
.as_ref()
to avoid dropping theBound<PyAny>
in themap_or
closure:let opt: Option<Bound<PyAny>> = ...; let p: *mut ffi::PyObject = opt.as_ref().map_or(std::ptr::null_mut(), Bound::as_ptr);
Migrating FromPyObject
implementations
FromPyObject
has had a new method extract_bound
which takes &Bound<'py, PyAny>
as an argument instead of &PyAny
. Both extract
and extract_bound
have been given default implementations in terms of the other, to avoid breaking code immediately on update to 0.21.
All implementations of FromPyObject
should be switched from extract
to extract_bound
.
Before:
impl<'py> FromPyObject<'py> for MyType {
fn extract(obj: &'py PyAny) -> PyResult<Self> {
/* ... */
}
}
After:
impl<'py> FromPyObject<'py> for MyType {
fn extract_bound(obj: &Bound<'py, PyAny>) -> PyResult<Self> {
/* ... */
}
}
The expectation is that in 0.22 extract_bound
will have the default implementation removed and in 0.23 extract
will be removed.
Cases where PyO3 cannot emit GIL Ref deprecation warnings
Despite a large amount of deprecations warnings produced by PyO3 to aid with the transition from GIL Refs to the Bound API, there are a few cases where PyO3 cannot automatically warn on uses of GIL Refs. It is worth checking for these cases manually after the deprecation warnings have all been addressed:
- Individual implementations of the
FromPyObject
trait cannot be deprecated, so PyO3 cannot warn about uses of code patterns like.extract<&PyAny>()
which produce a GIL Ref. - GIL Refs in
#[pyfunction]
arguments emit a warning, but if the GIL Ref is wrapped inside another container such asVec<&PyAny>
then PyO3 cannot warn against this. - The
wrap_pyfunction!(function)(py)
deferred argument form of thewrap_pyfunction
macro takingpy: Python<'py>
produces a GIL Ref, and due to limitations in type inference PyO3 cannot warn against this specific case.
Deactivating the gil-refs
feature
Click to expand
As a final step of migration, deactivating the gil-refs
feature will set up code for best performance and is intended to set up a forward-compatible API for PyO3 0.22.
At this point code that needed to manage GIL Ref memory can safely remove uses of GILPool
(which are constructed by calls to Python::new_pool
and Python::with_pool
). Deprecation warnings will highlight these cases.
There is just one case of code that changes upon disabling these features: FromPyObject
trait implementations for types that borrow directly from the input data cannot be implemented by PyO3 without GIL Refs (while the GIL Refs API is in the process of being removed). The main types affected are &str
, Cow<'_, str>
, &[u8]
, Cow<'_, u8>
.
To make PyO3's core functionality continue to work while the GIL Refs API is in the process of being removed, disabling the gil-refs
feature moves the implementations of FromPyObject
for &str
, Cow<'_, str>
, &[u8]
, Cow<'_, u8>
to a new temporary trait FromPyObjectBound
. This trait is the expected future form of FromPyObject
and has an additional lifetime 'a
to enable these types to borrow data from Python objects.
PyO3 0.21 has introduced the PyBackedStr
and PyBackedBytes
types to help with this case. The easiest way to avoid lifetime challenges from extracting &str
is to use these. For more complex types like Vec<&str>
, is now impossible to extract directly from a Python object and Vec<PyBackedStr>
is the recommended upgrade path.
A key thing to note here is because extracting to these types now ties them to the input lifetime, some extremely common patterns may need to be split into multiple Rust lines. For example, the following snippet of calling .extract::<&str>()
directly on the result of .getattr()
needs to be adjusted when deactivating the gil-refs-migration
feature.
Before:
#[cfg(feature = "gil-refs-migration")] {
use pyo3::prelude::*;
use pyo3::types::{PyList, PyType};
fn example<'py>(py: Python<'py>) -> PyResult<()> {
#[allow(deprecated)] // GIL Ref API
let obj: &'py PyType = py.get_type::<PyList>();
let name: &'py str = obj.getattr("__name__")?.extract()?;
assert_eq!(name, "list");
Ok(())
}
Python::with_gil(example).unwrap();
}
After:
#[cfg(any(not(Py_LIMITED_API), Py_3_10))] {
use pyo3::prelude::*;
use pyo3::types::{PyList, PyType};
fn example<'py>(py: Python<'py>) -> PyResult<()> {
let obj: Bound<'py, PyType> = py.get_type_bound::<PyList>();
let name_obj: Bound<'py, PyAny> = obj.getattr("__name__")?;
// the lifetime of the data is no longer `'py` but the much shorter
// lifetime of the `name_obj` smart pointer above
let name: &'_ str = name_obj.extract()?;
assert_eq!(name, "list");
Ok(())
}
Python::with_gil(example).unwrap();
}
To avoid needing to worry about lifetimes at all, it is also possible to use the new PyBackedStr
type, which stores a reference to the Python str
without a lifetime attachment. In particular, PyBackedStr
helps for abi3
builds for Python older than 3.10. Due to limitations in the abi3
CPython API for those older versions, PyO3 cannot offer a FromPyObjectBound
implementation for &str
on those versions. The easiest way to migrate for older abi3
builds is to replace any cases of .extract::<&str>()
with .extract::<PyBackedStr>()
. Alternatively, use .extract::<Cow<str>>()
, .extract::<String>()
to copy the data into Rust.
The following example uses the same snippet as those just above, but this time the final extracted type is PyBackedStr
:
use pyo3::prelude::*;
use pyo3::types::{PyList, PyType};
fn example<'py>(py: Python<'py>) -> PyResult<()> {
use pyo3::pybacked::PyBackedStr;
let obj: Bound<'py, PyType> = py.get_type_bound::<PyList>();
let name: PyBackedStr = obj.getattr("__name__")?.extract()?;
assert_eq!(&*name, "list");
Ok(())
}
Python::with_gil(example).unwrap();
from 0.19.* to 0.20
Drop support for older technologies
Click to expand
PyO3 0.20 has increased minimum Rust version to 1.56. This enables use of newer language features and simplifies maintenance of the project.
PyDict::get_item
now returns a Result
Click to expand
PyDict::get_item
in PyO3 0.19 and older was implemented using a Python API which would suppress all exceptions and return None
in those cases. This included errors in __hash__
and __eq__
implementations of the key being looked up.
Newer recommendations by the Python core developers advise against using these APIs which suppress exceptions, instead allowing exceptions to bubble upwards. PyDict::get_item_with_error
already implemented this recommended behavior, so that API has been renamed to PyDict::get_item
.
Before:
use pyo3::prelude::*;
use pyo3::exceptions::PyTypeError;
use pyo3::types::{PyDict, IntoPyDict};
fn main() {
let _ =
Python::with_gil(|py| {
let dict: &PyDict = [("a", 1)].into_py_dict(py);
// `a` is in the dictionary, with value 1
assert!(dict.get_item("a").map_or(Ok(false), |x| x.eq(1))?);
// `b` is not in the dictionary
assert!(dict.get_item("b").is_none());
// `dict` is not hashable, so this fails with a `TypeError`
assert!(dict
.get_item_with_error(dict)
.unwrap_err()
.is_instance_of::<PyTypeError>(py));
});
}
After:
use pyo3::prelude::*;
use pyo3::exceptions::PyTypeError;
use pyo3::types::{PyDict, IntoPyDict};
fn main() {
let _ =
Python::with_gil(|py| -> PyResult<()> {
let dict: &PyDict = [("a", 1)].into_py_dict(py);
// `a` is in the dictionary, with value 1
assert!(dict.get_item("a")?.map_or(Ok(false), |x| x.eq(1))?);
// `b` is not in the dictionary
assert!(dict.get_item("b")?.is_none());
// `dict` is not hashable, so this fails with a `TypeError`
assert!(dict
.get_item(dict)
.unwrap_err()
.is_instance_of::<PyTypeError>(py));
Ok(())
});
}
Required arguments are no longer accepted after optional arguments
Click to expand
Trailing Option<T>
arguments have an automatic default of None
. To avoid unwanted changes when modifying function signatures, in PyO3 0.18 it was deprecated to have a required argument after an Option<T>
argument without using #[pyo3(signature = (...))]
to specify the intended defaults. In PyO3 0.20, this becomes a hard error.
Before:
#[pyfunction]
fn x_or_y(x: Option<u64>, y: u64) -> u64 {
x.unwrap_or(y)
}
After:
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyfunction]
#[pyo3(signature = (x, y))] // both x and y have no defaults and are required
fn x_or_y(x: Option<u64>, y: u64) -> u64 {
x.unwrap_or(y)
}
Remove deprecated function forms
Click to expand
In PyO3 0.18 the #[args]
attribute for #[pymethods]
, and directly specifying the function signature in #[pyfunction]
, was deprecated. This functionality has been removed in PyO3 0.20.
Before:
#[pyfunction]
#[pyo3(a, b = "0", "/")]
fn add(a: u64, b: u64) -> u64 {
a + b
}
After:
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyfunction]
#[pyo3(signature = (a, b=0, /))]
fn add(a: u64, b: u64) -> u64 {
a + b
}
IntoPyPointer
trait removed
Click to expand
The trait IntoPyPointer
, which provided the into_ptr
method on many types, has been removed. into_ptr
is now available as an inherent method on all types that previously implemented this trait.
AsPyPointer
now unsafe
trait
Click to expand
The trait AsPyPointer
is now unsafe trait
, meaning any external implementation of it must be marked as unsafe impl
, and ensure that they uphold the invariant of returning valid pointers.
from 0.18.* to 0.19
Access to Python
inside __traverse__
implementations are now forbidden
Click to expand
During __traverse__
implementations for Python's Garbage Collection it is forbidden to do anything other than visit the members of the #[pyclass]
being traversed. This means making Python function calls or other API calls are forbidden.
Previous versions of PyO3 would allow access to Python
(e.g. via Python::with_gil
), which could cause the Python interpreter to crash or otherwise confuse the garbage collection algorithm.
Attempts to acquire the GIL will now panic. See #3165 for more detail.
use pyo3::prelude::*;
#[pyclass]
struct SomeClass {}
impl SomeClass {
fn __traverse__(&self, pyo3::class::gc::PyVisit<'_>) -> Result<(), pyo3::class::gc::PyTraverseError>` {
Python::with_gil(|| { /*...*/ }) // ERROR: this will panic
}
}
Smarter anyhow::Error
/ eyre::Report
conversion when inner error is "simple" PyErr
Click to expand
When converting from anyhow::Error
or eyre::Report
to PyErr
, if the inner error is a "simple" PyErr
(with no source error), then the inner error will be used directly as the PyErr
instead of wrapping it in a new PyRuntimeError
with the original information converted into a string.
#[cfg(feature = "anyhow")]
#[allow(dead_code)]
mod anyhow_only {
use pyo3::prelude::*;
use pyo3::exceptions::PyValueError;
#[pyfunction]
fn raise_err() -> anyhow::Result<()> {
Err(PyValueError::new_err("original error message").into())
}
fn main() {
Python::with_gil(|py| {
let rs_func = wrap_pyfunction!(raise_err, py).unwrap();
pyo3::py_run!(
py,
rs_func,
r"
try:
rs_func()
except Exception as e:
print(repr(e))
"
);
})
}
}
Before, the above code would have printed RuntimeError('ValueError: original error message')
, which might be confusing.
After, the same code will print ValueError: original error message
, which is more straightforward.
However, if the anyhow::Error
or eyre::Report
has a source, then the original exception will still be wrapped in a PyRuntimeError
.
The deprecated Python::acquire_gil
was removed and Python::with_gil
must be used instead
Click to expand
While the API provided by Python::acquire_gil
seems convenient, it is somewhat brittle as the design of the GIL token Python
relies on proper nesting and panics if not used correctly, e.g.
#![allow(dead_code, deprecated)]
use pyo3::prelude::*;
#[pyclass]
struct SomeClass {}
struct ObjectAndGuard {
object: Py<SomeClass>,
guard: GILGuard,
}
impl ObjectAndGuard {
fn new() -> Self {
let guard = Python::acquire_gil();
let object = Py::new(guard.python(), SomeClass {}).unwrap();
Self { object, guard }
}
}
let first = ObjectAndGuard::new();
let second = ObjectAndGuard::new();
// Panics because the guard within `second` is still alive.
drop(first);
drop(second);
The replacement is Python::with_gil
which is more cumbersome but enforces the proper nesting by design, e.g.
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyclass]
struct SomeClass {}
struct Object {
object: Py<SomeClass>,
}
impl Object {
fn new(py: Python<'_>) -> Self {
let object = Py::new(py, SomeClass {}).unwrap();
Self { object }
}
}
// It either forces us to release the GIL before aquiring it again.
let first = Python::with_gil(|py| Object::new(py));
let second = Python::with_gil(|py| Object::new(py));
drop(first);
drop(second);
// Or it ensure releasing the inner lock before the outer one.
Python::with_gil(|py| {
let first = Object::new(py);
let second = Python::with_gil(|py| Object::new(py));
drop(first);
drop(second);
});
Furthermore, Python::acquire_gil
provides ownership of a GILGuard
which can be freely stored and passed around. This is usually not helpful as it may keep the lock held for a long time thereby blocking progress in other parts of the program. Due to the generative lifetime attached to the GIL token supplied by Python::with_gil
, the problem is avoided as the GIL token can only be passed down the call chain. Often, this issue can also be avoided entirely as any GIL-bound reference &'py PyAny
implies access to a GIL token Python<'py>
via the PyAny::py
method.
from 0.17.* to 0.18
Required arguments after Option<_>
arguments will no longer be automatically inferred
Click to expand
In #[pyfunction]
and #[pymethods]
, if a "required" function input such as i32
came after an Option<_>
input, then the Option<_>
would be implicitly treated as required. (All trailing Option<_>
arguments were treated as optional with a default value of None
).
Starting with PyO3 0.18, this is deprecated and a future PyO3 version will require a #[pyo3(signature = (...))]
option to explicitly declare the programmer's intention.
Before, x in the below example would be required to be passed from Python code:
#![allow(dead_code)]
use pyo3::prelude::*;
#[pyfunction]
fn required_argument_after_option(x: Option<i32>, y: i32) {}
After, specify the intended Python signature explicitly:
#![allow(dead_code)]
use pyo3::prelude::*;
// If x really was intended to be required
#[pyfunction(signature = (x, y))]
fn required_argument_after_option_a(x: Option<i32>, y: i32) {}
// If x was intended to be optional, y needs a default too
#[pyfunction(signature = (x=None, y=0))]
fn required_argument_after_option_b(x: Option<i32>, y: i32) {}
__text_signature__
is now automatically generated for #[pyfunction]
and #[pymethods]
Click to expand
The #[pyo3(text_signature = "...")]
option was previously the only supported way to set the __text_signature__
attribute on generated Python functions.
PyO3 is now able to automatically populate __text_signature__
for all functions automatically based on their Rust signature (or the new #[pyo3(signature = (...))]
option). These automatically-generated __text_signature__
values will currently only render ...
for all default values. Many #[pyo3(text_signature = "...")]
options can be removed from functions when updating to PyO3 0.18, however in cases with default values a manual implementation may still be preferred for now.
As examples:
use pyo3::prelude::*;
// The `text_signature` option here is no longer necessary, as PyO3 will automatically
// generate exactly the same value.
#[pyfunction(text_signature = "(a, b, c)")]
fn simple_function(a: i32, b: i32, c: i32) {}
// The `text_signature` still provides value here as of PyO3 0.18, because the automatically
// generated signature would be "(a, b=..., c=...)".
#[pyfunction(signature = (a, b = 1, c = 2), text_signature = "(a, b=1, c=2)")]
fn function_with_defaults(a: i32, b: i32, c: i32) {}
fn main() {
Python::with_gil(|py| {
let simple = wrap_pyfunction_bound!(simple_function, py).unwrap();
assert_eq!(simple.getattr("__text_signature__").unwrap().to_string(), "(a, b, c)");
let defaulted = wrap_pyfunction_bound!(function_with_defaults, py).unwrap();
assert_eq!(defaulted.getattr("__text_signature__").unwrap().to_string(), "(a, b=1, c=2)");
})
}
from 0.16.* to 0.17
Type checks have been changed for PyMapping
and PySequence
types
Click to expand
Previously the type checks for PyMapping
and PySequence
(implemented in PyTryFrom
)
used the Python C-API functions PyMapping_Check
and PySequence_Check
.
Unfortunately these functions are not sufficient for distinguishing such types,
leading to inconsistent behavior (see
pyo3/pyo3#2072).
PyO3 0.17 changes these downcast checks to explicitly test if the type is a
subclass of the corresponding abstract base class collections.abc.Mapping
or
collections.abc.Sequence
. Note this requires calling into Python, which may
incur a performance penalty over the previous method. If this performance
penalty is a problem, you may be able to perform your own checks and use
try_from_unchecked
(unsafe).
Another side-effect is that a pyclass defined in Rust with PyO3 will need to
be registered with the corresponding Python abstract base class for
downcasting to succeed. PySequence::register
and PyMapping:register
have
been added to make it easy to do this from Rust code. These are equivalent to
calling collections.abc.Mapping.register(MappingPyClass)
or
collections.abc.Sequence.register(SequencePyClass)
from Python.
For example, for a mapping class defined in Rust:
use pyo3::prelude::*;
use std::collections::HashMap;
#[pyclass(mapping)]
struct Mapping {
index: HashMap<String, usize>,
}
#[pymethods]
impl Mapping {
#[new]
fn new(elements: Option<&PyList>) -> PyResult<Self> {
// ...
// truncated implementation of this mapping pyclass - basically a wrapper around a HashMap
}
You must register the class with collections.abc.Mapping
before the downcast will work:
let m = Py::new(py, Mapping { index }).unwrap();
assert!(m.as_ref(py).downcast::<PyMapping>().is_err());
PyMapping::register::<Mapping>(py).unwrap();
assert!(m.as_ref(py).downcast::<PyMapping>().is_ok());
Note that this requirement may go away in the future when a pyclass is able to inherit from the abstract base class directly (see pyo3/pyo3#991).
The multiple-pymethods
feature now requires Rust 1.62
Click to expand
Due to limitations in the inventory
crate which the multiple-pymethods
feature depends on, this feature now
requires Rust 1.62. For more information see dtolnay/inventory#32.
Added impl IntoPy<Py<PyString>> for &str
Click to expand
This may cause inference errors.
Before:
use pyo3::prelude::*;
fn main() {
Python::with_gil(|py| {
// Cannot infer either `Py<PyAny>` or `Py<PyString>`
let _test = "test".into_py(py);
});
}
After, some type annotations may be necessary:
use pyo3::prelude::*;
fn main() {
Python::with_gil(|py| {
let _test: Py<PyAny> = "test".into_py(py);
});
}
The pyproto
feature is now disabled by default
Click to expand
In preparation for removing the deprecated #[pyproto]
attribute macro in a future PyO3 version, it is now gated behind an opt-in feature flag. This also gives a slight saving to compile times for code which does not use the deprecated macro.
PyTypeObject
trait has been deprecated
Click to expand
The PyTypeObject
trait already was near-useless; almost all functionality was already on the PyTypeInfo
trait, which PyTypeObject
had a blanket implementation based upon. In PyO3 0.17 the final method, PyTypeObject::type_object
was moved to PyTypeInfo::type_object
.
To migrate, update trait bounds and imports from PyTypeObject
to PyTypeInfo
.
Before:
use pyo3::Python;
use pyo3::type_object::PyTypeObject;
use pyo3::types::PyType;
fn get_type_object<T: PyTypeObject>(py: Python<'_>) -> &PyType {
T::type_object(py)
}
After
use pyo3::{Python, PyTypeInfo};
use pyo3::types::PyType;
fn get_type_object<T: PyTypeInfo>(py: Python<'_>) -> &PyType {
T::type_object(py)
}
Python::with_gil(|py| { get_type_object::<pyo3::types::PyList>(py); });
impl<T, const N: usize> IntoPy<PyObject> for [T; N]
now requires T: IntoPy
rather than T: ToPyObject
Click to expand
If this leads to errors, simply implement IntoPy
. Because pyclasses already implement IntoPy
, you probably don't need to worry about this.
Each #[pymodule]
can now only be initialized once per process
Click to expand
To make PyO3 modules sound in the presence of Python sub-interpreters, for now it has been necessary to explicitly disable the ability to initialize a #[pymodule]
more than once in the same process. Attempting to do this will now raise an ImportError
.
from 0.15.* to 0.16
Drop support for older technologies
Click to expand
PyO3 0.16 has increased minimum Rust version to 1.48 and minimum Python version to 3.7. This enables use of newer language features (enabling some of the other additions in 0.16) and simplifies maintenance of the project.
#[pyproto]
has been deprecated
Click to expand
In PyO3 0.15, the #[pymethods]
attribute macro gained support for implementing "magic methods" such as __str__
(aka "dunder" methods). This implementation was not quite finalized at the time, with a few edge cases to be decided upon. The existing #[pyproto]
attribute macro was left untouched, because it covered these edge cases.
In PyO3 0.16, the #[pymethods]
implementation has been completed and is now the preferred way to implement magic methods. To allow the PyO3 project to move forward, #[pyproto]
has been deprecated (with expected removal in PyO3 0.18).
Migration from #[pyproto]
to #[pymethods]
is straightforward; copying the existing methods directly from the #[pyproto]
trait implementation is all that is needed in most cases.
Before:
use pyo3::prelude::*;
use pyo3::class::{PyObjectProtocol, PyIterProtocol};
use pyo3::types::PyString;
#[pyclass]
struct MyClass {}
#[pyproto]
impl PyObjectProtocol for MyClass {
fn __str__(&self) -> &'static [u8] {
b"hello, world"
}
}
#[pyproto]
impl PyIterProtocol for MyClass {
fn __iter__(slf: PyRef<self>) -> PyResult<&PyAny> {
PyString::new(slf.py(), "hello, world").iter()
}
}
After
use pyo3::prelude::*;
use pyo3::types::PyString;
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
fn __str__(&self) -> &'static [u8] {
b"hello, world"
}
fn __iter__(slf: PyRef<self>) -> PyResult<&PyAny> {
PyString::new(slf.py(), "hello, world").iter()
}
}
Removed PartialEq
for object wrappers
Click to expand
The Python object wrappers Py
and PyAny
had implementations of PartialEq
so that object_a == object_b
would compare the Python objects for pointer
equality, which corresponds to the is
operator, not the ==
operator in
Python. This has been removed in favor of a new method: use
object_a.is(object_b)
. This also has the advantage of not requiring the same
wrapper type for object_a
and object_b
; you can now directly compare a
Py<T>
with a &PyAny
without having to convert.
To check for Python object equality (the Python ==
operator), use the new
method eq()
.
Container magic methods now match Python behavior
Click to expand
In PyO3 0.15, __getitem__
, __setitem__
and __delitem__
in #[pymethods]
would generate only the mapping implementation for a #[pyclass]
. To match the Python behavior, these methods now generate both the mapping and sequence implementations.
This means that classes implementing these #[pymethods]
will now also be treated as sequences, same as a Python class
would be. Small differences in behavior may result:
- PyO3 will allow instances of these classes to be cast to
PySequence
as well asPyMapping
. - Python will provide a default implementation of
__iter__
(if the class did not have one) which repeatedly calls__getitem__
with integers (starting at 0) until anIndexError
is raised.
To explain this in detail, consider the following Python class:
class ExampleContainer:
def __len__(self):
return 5
def __getitem__(self, idx: int) -> int:
if idx < 0 or idx > 5:
raise IndexError()
return idx
This class implements a Python sequence.
The __len__
and __getitem__
methods are also used to implement a Python mapping. In the Python C-API, these methods are not shared: the sequence __len__
and __getitem__
are defined by the sq_length
and sq_item
slots, and the mapping equivalents are mp_length
and mp_subscript
. There are similar distinctions for __setitem__
and __delitem__
.
Because there is no such distinction from Python, implementing these methods will fill the mapping and sequence slots simultaneously. A Python class with __len__
implemented, for example, will have both the sq_length
and mp_length
slots filled.
The PyO3 behavior in 0.16 has been changed to be closer to this Python behavior by default.
wrap_pymodule!
and wrap_pyfunction!
now respect privacy correctly
Click to expand
Prior to PyO3 0.16 the wrap_pymodule!
and wrap_pyfunction!
macros could use modules and functions whose defining fn
was not reachable according Rust privacy rules.
For example, the following code was legal before 0.16, but in 0.16 is rejected because the wrap_pymodule!
macro cannot access the private_submodule
function:
mod foo {
use pyo3::prelude::*;
#[pymodule]
fn private_submodule(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
Ok(())
}
}
use pyo3::prelude::*;
use foo::*;
#[pymodule]
fn my_module(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pymodule!(private_submodule))?;
Ok(())
}
To fix it, make the private submodule visible, e.g. with pub
or pub(crate)
.
mod foo {
use pyo3::prelude::*;
#[pymodule]
pub(crate) fn private_submodule(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
Ok(())
}
}
use pyo3::prelude::*;
use pyo3::wrap_pymodule;
use foo::*;
#[pymodule]
fn my_module(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pymodule!(private_submodule))?;
Ok(())
}
from 0.14.* to 0.15
Changes in sequence indexing
Click to expand
For all types that take sequence indices (PyList
, PyTuple
and PySequence
),
the API has been made consistent to only take usize
indices, for consistency
with Rust's indexing conventions. Negative indices, which were only
sporadically supported even in APIs that took isize
, now aren't supported
anywhere.
Further, the get_item
methods now always return a PyResult
instead of
panicking on invalid indices. The Index
trait has been implemented instead,
and provides the same panic behavior as on Rust vectors.
Note that slice indices (accepted by PySequence::get_slice
and other) still
inherit the Python behavior of clamping the indices to the actual length, and
not panicking/returning an error on out of range indices.
An additional advantage of using Rust's indexing conventions for these types is that these types can now also support Rust's indexing operators as part of a consistent API:
#![allow(deprecated)]
use pyo3::{Python, types::PyList};
Python::with_gil(|py| {
let list = PyList::new(py, &[1, 2, 3]);
assert_eq!(list[0..2].to_string(), "[1, 2]");
});
from 0.13.* to 0.14
auto-initialize
feature is now opt-in
Click to expand
For projects embedding Python in Rust, PyO3 no longer automatically initializes a Python interpreter on the first call to Python::with_gil
(or Python::acquire_gil
) unless the auto-initialize
feature is enabled.
New multiple-pymethods
feature
Click to expand
#[pymethods]
have been reworked with a simpler default implementation which removes the dependency on the inventory
crate. This reduces dependencies and compile times for the majority of users.
The limitation of the new default implementation is that it cannot support multiple #[pymethods]
blocks for the same #[pyclass]
. If you need this functionality, you must enable the multiple-pymethods
feature which will switch #[pymethods]
to the inventory-based implementation.
Deprecated #[pyproto]
methods
Click to expand
Some protocol (aka __dunder__
) methods such as __bytes__
and __format__
have been possible to implement two ways in PyO3 for some time: via a #[pyproto]
(e.g. PyObjectProtocol
for the methods listed here), or by writing them directly in #[pymethods]
. This is only true for a handful of the #[pyproto]
methods (for technical reasons to do with the way PyO3 currently interacts with the Python C-API).
In the interest of having only one way to do things, the #[pyproto]
forms of these methods have been deprecated.
To migrate just move the affected methods from a #[pyproto]
to a #[pymethods]
block.
Before:
use pyo3::prelude::*;
use pyo3::class::basic::PyObjectProtocol;
#[pyclass]
struct MyClass {}
#[pyproto]
impl PyObjectProtocol for MyClass {
fn __bytes__(&self) -> &'static [u8] {
b"hello, world"
}
}
After:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
fn __bytes__(&self) -> &'static [u8] {
b"hello, world"
}
}
from 0.12.* to 0.13
Minimum Rust version increased to Rust 1.45
Click to expand
PyO3 0.13
makes use of new Rust language features stabilized between Rust 1.40 and Rust 1.45. If you are using a Rust compiler older than Rust 1.45, you will need to update your toolchain to be able to continue using PyO3.
Runtime changes to support the CPython limited API
Click to expand
In PyO3 0.13
support was added for compiling against the CPython limited API. This had a number of implications for all PyO3 users, described here.
The largest of these is that all types created from PyO3 are what CPython calls "heap" types. The specific implications of this are:
- If you wish to subclass one of these types from Rust you must mark it
#[pyclass(subclass)]
, as you would if you wished to allow subclassing it from Python code. - Type objects are now mutable - Python code can set attributes on them.
__module__
on types without#[pyclass(module="mymodule")]
no longer returnsbuiltins
, it now raisesAttributeError
.
from 0.11.* to 0.12
PyErr
has been reworked
Click to expand
In PyO3 0.12
the PyErr
type has been re-implemented to be significantly more compatible with
the standard Rust error handling ecosystem. Specifically PyErr
now implements
Error + Send + Sync
, which are the standard traits used for error types.
While this has necessitated the removal of a number of APIs, the resulting PyErr
type should now
be much more easier to work with. The following sections list the changes in detail and how to
migrate to the new APIs.
PyErr::new
and PyErr::from_type
now require Send + Sync
for their argument
Click to expand
For most uses no change will be needed. If you are trying to construct PyErr
from a value that is
not Send + Sync
, you will need to first create the Python object and then use
PyErr::from_instance
.
Similarly, any types which implemented PyErrArguments
will now need to be Send + Sync
.
PyErr
's contents are now private
Click to expand
It is no longer possible to access the fields .ptype
, .pvalue
and .ptraceback
of a PyErr
.
You should instead now use the new methods PyErr::ptype
, PyErr::pvalue
and PyErr::ptraceback
.
PyErrValue
and PyErr::from_value
have been removed
Click to expand
As these were part the internals of PyErr
which have been reworked, these APIs no longer exist.
If you used this API, it is recommended to use PyException::new_err
(see the section on
Exception types).
Into<PyResult<T>>
for PyErr
has been removed
Click to expand
This implementation was redundant. Just construct the Result::Err
variant directly.
Before:
let result: PyResult<()> = PyErr::new::<TypeError, _>("error message").into();
After (also using the new reworked exception types; see the following section):
use pyo3::{PyResult, exceptions::PyTypeError};
let result: PyResult<()> = Err(PyTypeError::new_err("error message"));
Exception types have been reworked
Click to expand
Previously exception types were zero-sized marker types purely used to construct PyErr
. In PyO3
0.12, these types have been replaced with full definitions and are usable in the same way as PyAny
, PyDict
etc. This
makes it possible to interact with Python exception objects.
The new types also have names starting with the "Py" prefix. For example, before:
let err: PyErr = TypeError::py_err("error message");
After:
use pyo3::{PyErr, PyResult, Python, type_object::PyTypeObject};
use pyo3::exceptions::{PyBaseException, PyTypeError};
Python::with_gil(|py| -> PyResult<()> {
let err: PyErr = PyTypeError::new_err("error message");
// Uses Display for PyErr, new for PyO3 0.12
assert_eq!(err.to_string(), "TypeError: error message");
// Now possible to interact with exception instances, new for PyO3 0.12
let instance: &PyBaseException = err.instance(py);
assert_eq!(
instance.getattr("__class__")?,
PyTypeError::type_object(py).as_ref()
);
Ok(())
}).unwrap();
FromPy
has been removed
Click to expand
To simplify the PyO3 conversion traits, the FromPy
trait has been removed. Previously there were
two ways to define the to-Python conversion for a type:
FromPy<T> for PyObject
and IntoPy<PyObject> for T
.
Now there is only one way to define the conversion, IntoPy
, so downstream crates may need to
adjust accordingly.
Before:
use pyo3::prelude::*;
struct MyPyObjectWrapper(PyObject);
impl FromPy<MyPyObjectWrapper> for PyObject {
fn from_py(other: MyPyObjectWrapper, _py: Python<'_>) -> Self {
other.0
}
}
After
use pyo3::prelude::*;
#[allow(dead_code)]
struct MyPyObjectWrapper(PyObject);
impl IntoPy<PyObject> for MyPyObjectWrapper {
fn into_py(self, _py: Python<'_>) -> PyObject {
self.0
}
}
Similarly, code which was using the FromPy
trait can be trivially rewritten to use IntoPy
.
Before:
use pyo3::prelude::*;
Python::with_gil(|py| {
let obj = PyObject::from_py(1.234, py);
})
After:
use pyo3::prelude::*;
Python::with_gil(|py| {
let obj: PyObject = 1.234.into_py(py);
})
PyObject
is now a type alias of Py<PyAny>
Click to expand
This should change very little from a usage perspective. If you implemented traits for both
PyObject
and Py<T>
, you may find you can just remove the PyObject
implementation.
AsPyRef
has been removed
Click to expand
As PyObject
has been changed to be just a type alias, the only remaining implementor of AsPyRef
was Py<T>
. This removed the need for a trait, so the AsPyRef::as_ref
method has been moved to
Py::as_ref
.
This should require no code changes except removing use pyo3::AsPyRef
for code which did not use
pyo3::prelude::*
.
Before:
use pyo3::{AsPyRef, Py, types::PyList};
pyo3::Python::with_gil(|py| {
let list_py: Py<PyList> = PyList::empty(py).into();
let list_ref: &PyList = list_py.as_ref(py);
})
After:
use pyo3::{Py, types::PyList};
pyo3::Python::with_gil(|py| {
let list_py: Py<PyList> = PyList::empty(py).into();
let list_ref: &PyList = list_py.as_ref(py);
})
from 0.10.* to 0.11
Stable Rust
Click to expand
PyO3 now supports the stable Rust toolchain. The minimum required version is 1.39.0.
#[pyclass]
structs must now be Send
or unsendable
Click to expand
Because #[pyclass]
structs can be sent between threads by the Python interpreter, they must implement
Send
or declared as unsendable
(by #[pyclass(unsendable)]
).
Note that unsendable
is added in PyO3 0.11.1
and Send
is always required in PyO3 0.11.0
.
This may "break" some code which previously was accepted, even though it could be unsound. There can be two fixes:
-
If you think that your
#[pyclass]
actually must beSend
able, then let's implementSend
. A common, safer way is using thread-safe types. E.g.,Arc
instead ofRc
,Mutex
instead ofRefCell
, andBox<dyn Send + T>
instead ofBox<dyn T>
.Before:
use pyo3::prelude::*; use std::rc::Rc; use std::cell::RefCell; #[pyclass] struct NotThreadSafe { shared_bools: Rc<RefCell<Vec<bool>>>, closure: Box<dyn Fn()>, }
After:
#![allow(dead_code)] use pyo3::prelude::*; use std::sync::{Arc, Mutex}; #[pyclass] struct ThreadSafe { shared_bools: Arc<Mutex<Vec<bool>>>, closure: Box<dyn Fn() + Send>, }
In situations where you cannot change your
#[pyclass]
to automatically implementSend
(e.g., when it contains a raw pointer), you can useunsafe impl Send
. In such cases, care should be taken to ensure the struct is actually thread safe. See the Rustonomicon for more. -
If you think that your
#[pyclass]
should not be accessed by another thread, you can useunsendable
flag. A class marked withunsendable
panics when accessed by another thread, making it thread-safe to expose an unsendable object to the Python interpreter.Before:
use pyo3::prelude::*; #[pyclass] struct Unsendable { pointers: Vec<*mut std::os::raw::c_char>, }
After:
#![allow(dead_code)] use pyo3::prelude::*; #[pyclass(unsendable)] struct Unsendable { pointers: Vec<*mut std::os::raw::c_char>, }
All PyObject
and Py<T>
methods now take Python
as an argument
Click to expand
Previously, a few methods such as Object::get_refcnt
did not take Python
as an argument (to
ensure that the Python GIL was held by the current thread). Technically, this was not sound.
To migrate, just pass a py
argument to any calls to these methods.
Before:
pyo3::Python::with_gil(|py| {
py.None().get_refcnt();
})
After:
pyo3::Python::with_gil(|py| {
py.None().get_refcnt(py);
})
from 0.9.* to 0.10
ObjectProtocol
is removed
Click to expand
All methods are moved to PyAny
.
And since now all native types (e.g., PyList
) implements Deref<Target=PyAny>
,
all you need to do is remove ObjectProtocol
from your code.
Or if you use ObjectProtocol
by use pyo3::prelude::*
, you have to do nothing.
Before:
use pyo3::ObjectProtocol;
pyo3::Python::with_gil(|py| {
let obj = py.eval("lambda: 'Hi :)'", None, None).unwrap();
let hi: &pyo3::types::PyString = obj.call0().unwrap().downcast().unwrap();
assert_eq!(hi.len().unwrap(), 5);
})
After:
pyo3::Python::with_gil(|py| {
let obj = py.eval("lambda: 'Hi :)'", None, None).unwrap();
let hi: &pyo3::types::PyString = obj.call0().unwrap().downcast().unwrap();
assert_eq!(hi.len().unwrap(), 5);
})
No #![feature(specialization)]
in user code
Click to expand
While PyO3 itself still requires specialization and nightly Rust,
now you don't have to use #![feature(specialization)]
in your crate.
from 0.8.* to 0.9
#[new]
interface
Click to expand
PyRawObject
is now removed and our syntax for constructors has changed.
Before:
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
#[new]
fn new(obj: &PyRawObject) {
obj.init(MyClass {})
}
}
After:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
#[pymethods]
impl MyClass {
#[new]
fn new() -> Self {
MyClass {}
}
}
Basically you can return Self
or Result<Self>
directly.
For more, see the constructor section of this guide.
PyCell
Click to expand
PyO3 0.9 introduces PyCell
, which is a RefCell
-like object wrapper
for ensuring Rust's rules regarding aliasing of references are upheld.
For more detail, see the
Rust Book's section on Rust's rules of references
For #[pymethods]
or #[pyfunction]
s, your existing code should continue to work without any change.
Python exceptions will automatically be raised when your functions are used in a way which breaks Rust's
rules of references.
Here is an example.
use pyo3::prelude::*;
#[pyclass]
struct Names {
names: Vec<String>,
}
#[pymethods]
impl Names {
#[new]
fn new() -> Self {
Names { names: vec![] }
}
fn merge(&mut self, other: &mut Names) {
self.names.append(&mut other.names)
}
}
Python::with_gil(|py| {
let names = Py::new(py, Names::new()).unwrap();
pyo3::py_run!(py, names, r"
try:
names.merge(names)
assert False, 'Unreachable'
except RuntimeError as e:
assert str(e) == 'Already borrowed'
");
})
Names
has a merge
method, which takes &mut self
and another argument of type &mut Self
.
Given this #[pyclass]
, calling names.merge(names)
in Python raises
a PyBorrowMutError
exception, since it requires two mutable borrows of names
.
However, for #[pyproto]
and some functions, you need to manually fix the code.
Object creation
In 0.8 object creation was done with PyRef::new
and PyRefMut::new
.
In 0.9 these have both been removed.
To upgrade code, please use
PyCell::new
instead.
If you need PyRef
or PyRefMut
, just call .borrow()
or .borrow_mut()
on the newly-created PyCell
.
Before:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
Python::with_gil(|py| {
let obj_ref = PyRef::new(py, MyClass {}).unwrap();
})
After:
use pyo3::prelude::*;
#[pyclass]
struct MyClass {}
Python::with_gil(|py| {
let obj = PyCell::new(py, MyClass {}).unwrap();
let obj_ref = obj.borrow();
})
Object extraction
For PyClass
types T
, &T
and &mut T
no longer have FromPyObject
implementations.
Instead you should extract PyRef<T>
or PyRefMut<T>
, respectively.
If T
implements Clone
, you can extract T
itself.
In addition, you can also extract &PyCell<T>
, though you rarely need it.
Before:
let obj: &PyAny = create_obj();
let obj_ref: &MyClass = obj.extract().unwrap();
let obj_ref_mut: &mut MyClass = obj.extract().unwrap();
After:
use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
#[pyclass] #[derive(Clone)] struct MyClass {}
#[pymethods] impl MyClass { #[new]fn new() -> Self { MyClass {} }}
Python::with_gil(|py| {
let typeobj = py.get_type::<MyClass>();
let d = [("c", typeobj)].into_py_dict(py);
let create_obj = || py.eval("c()", None, Some(d)).unwrap();
let obj: &PyAny = create_obj();
let obj_cell: &PyCell<MyClass> = obj.extract().unwrap();
let obj_cloned: MyClass = obj.extract().unwrap(); // extracted by cloning the object
{
let obj_ref: PyRef<'_, MyClass> = obj.extract().unwrap();
// we need to drop obj_ref before we can extract a PyRefMut due to Rust's rules of references
}
let obj_ref_mut: PyRefMut<'_, MyClass> = obj.extract().unwrap();
})
#[pyproto]
Most of the arguments to methods in #[pyproto]
impls require a
FromPyObject
implementation.
So if your protocol methods take &T
or &mut T
(where T: PyClass
),
please use PyRef
or PyRefMut
instead.
Before:
use pyo3::prelude::*;
use pyo3::class::PySequenceProtocol;
#[pyclass]
struct ByteSequence {
elements: Vec<u8>,
}
#[pyproto]
impl PySequenceProtocol for ByteSequence {
fn __concat__(&self, other: &Self) -> PyResult<Self> {
let mut elements = self.elements.clone();
elements.extend_from_slice(&other.elements);
Ok(Self { elements })
}
}
After:
use pyo3::prelude::*;
use pyo3::class::PySequenceProtocol;
#[pyclass]
struct ByteSequence {
elements: Vec<u8>,
}
#[pyproto]
impl PySequenceProtocol for ByteSequence {
fn __concat__(&self, other: PyRef<'p, Self>) -> PyResult<Self> {
let mut elements = self.elements.clone();
elements.extend_from_slice(&other.elements);
Ok(Self { elements })
}
}
Using in Python a Rust function with trait bounds
PyO3 allows for easy conversion from Rust to Python for certain functions and classes (see the conversion table. However, it is not always straightforward to convert Rust code that requires a given trait implementation as an argument.
This tutorial explains how to convert a Rust function that takes a trait as argument for use in Python with classes implementing the same methods as the trait.
Why is this useful?
Pros
- Make your Rust code available to Python users
- Code complex algorithms in Rust with the help of the borrow checker
Cons
- Not as fast as native Rust (type conversion has to be performed and one part of the code runs in Python)
- You need to adapt your code to expose it
Example
Let's work with the following basic example of an implementation of a optimization solver operating on a given model.
Let's say we have a function solve
that operates on a model and mutates its state.
The argument of the function can be any model that implements the Model
trait :
#![allow(dead_code)]
pub trait Model {
fn set_variables(&mut self, inputs: &Vec<f64>);
fn compute(&mut self);
fn get_results(&self) -> Vec<f64>;
}
pub fn solve<T: Model>(model: &mut T) {
println!("Magic solver that mutates the model into a resolved state");
}
Let's assume we have the following constraints:
- We cannot change that code as it runs on many Rust models.
- We also have many Python models that cannot be solved as this solver is not available in that language. Rewriting it in Python would be cumbersome and error-prone, as everything is already available in Rust.
How could we expose this solver to Python thanks to PyO3 ?
Implementation of the trait bounds for the Python class
If a Python class implements the same three methods as the Model
trait, it seems logical it could be adapted to use the solver.
However, it is not possible to pass a PyObject
to it as it does not implement the Rust trait (even if the Python model has the required methods).
In order to implement the trait, we must write a wrapper around the calls in Rust to the Python model. The method signatures must be the same as the trait, keeping in mind that the Rust trait cannot be changed for the purpose of making the code available in Python.
The Python model we want to expose is the following one, which already contains all the required methods:
class Model:
def set_variables(self, inputs):
self.inputs = inputs
def compute(self):
self.results = [elt**2 - 3 for elt in self.inputs]
def get_results(self):
return self.results
The following wrapper will call the Python model from Rust, using a struct to hold the model as a PyAny
object:
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyList;
pub trait Model {
fn set_variables(&mut self, inputs: &Vec<f64>);
fn compute(&mut self);
fn get_results(&self) -> Vec<f64>;
}
struct UserModel {
model: Py<PyAny>,
}
impl Model for UserModel {
fn set_variables(&mut self, var: &Vec<f64>) {
println!("Rust calling Python to set the variables");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("set_variables", (PyList::new_bound(py, var),), None)
.unwrap();
})
}
fn get_results(&self) -> Vec<f64> {
println!("Rust calling Python to get the results");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("get_results", (), None)
.unwrap()
.extract()
.unwrap()
})
}
fn compute(&mut self) {
println!("Rust calling Python to perform the computation");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("compute", (), None)
.unwrap();
})
}
}
Now that this bit is implemented, let's expose the model wrapper to Python. Let's add the PyO3 annotations and add a constructor:
#![allow(dead_code)]
pub trait Model {
fn set_variables(&mut self, inputs: &Vec<f64>);
fn compute(&mut self);
fn get_results(&self) -> Vec<f64>;
}
use pyo3::prelude::*;
#[pyclass]
struct UserModel {
model: Py<PyAny>,
}
#[pymodule]
fn trait_exposure(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<UserModel>()?;
Ok(())
}
#[pymethods]
impl UserModel {
#[new]
pub fn new(model: Py<PyAny>) -> Self {
UserModel { model }
}
}
Now we add the PyO3 annotations to the trait implementation:
#[pymethods]
impl Model for UserModel {
// the previous trait implementation
}
However, the previous code will not compile. The compilation error is the following one:
error: #[pymethods] cannot be used on trait impl blocks
That's a bummer! However, we can write a second wrapper around these functions to call them directly. This wrapper will also perform the type conversions between Python and Rust.
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyList;
pub trait Model {
fn set_variables(&mut self, inputs: &Vec<f64>);
fn compute(&mut self);
fn get_results(&self) -> Vec<f64>;
}
#[pyclass]
struct UserModel {
model: Py<PyAny>,
}
impl Model for UserModel {
fn set_variables(&mut self, var: &Vec<f64>) {
println!("Rust calling Python to set the variables");
Python::with_gil(|py| {
self.model.bind(py)
.call_method("set_variables", (PyList::new_bound(py, var),), None)
.unwrap();
})
}
fn get_results(&self) -> Vec<f64> {
println!("Rust calling Python to get the results");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("get_results", (), None)
.unwrap()
.extract()
.unwrap()
})
}
fn compute(&mut self) {
println!("Rust calling Python to perform the computation");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("compute", (), None)
.unwrap();
})
}
}
#[pymethods]
impl UserModel {
pub fn set_variables(&mut self, var: Vec<f64>) {
println!("Set variables from Python calling Rust");
Model::set_variables(self, &var)
}
pub fn get_results(&mut self) -> Vec<f64> {
println!("Get results from Python calling Rust");
Model::get_results(self)
}
pub fn compute(&mut self) {
println!("Compute from Python calling Rust");
Model::compute(self)
}
}
This wrapper handles the type conversion between the PyO3 requirements and the trait. In order to meet PyO3 requirements, this wrapper must:
- return an object of type
PyResult
- use only values, not references in the method signatures
Let's run the file python file:
class Model:
def set_variables(self, inputs):
self.inputs = inputs
def compute(self):
self.results = [elt**2 - 3 for elt in self.inputs]
def get_results(self):
return self.results
if __name__=="__main__":
import trait_exposure
myModel = Model()
my_rust_model = trait_exposure.UserModel(myModel)
my_rust_model.set_variables([2.0])
print("Print value from Python: ", myModel.inputs)
my_rust_model.compute()
print("Print value from Python through Rust: ", my_rust_model.get_results())
print("Print value directly from Python: ", myModel.get_results())
This outputs:
Set variables from Python calling Rust
Set variables from Rust calling Python
Print value from Python: [2.0]
Compute from Python calling Rust
Compute from Rust calling Python
Get results from Python calling Rust
Get results from Rust calling Python
Print value from Python through Rust: [1.0]
Print value directly from Python: [1.0]
We have now successfully exposed a Rust model that implements the Model
trait to Python!
We will now expose the solve
function, but before, let's talk about types errors.
Type errors in Python
What happens if you have type errors when using Python and how can you improve the error messages?
Wrong types in Python function arguments
Let's assume in the first case that you will use in your Python file my_rust_model.set_variables(2.0)
instead of my_rust_model.set_variables([2.0])
.
The Rust signature expects a vector, which corresponds to a list in Python. What happens if instead of a vector, we pass a single value ?
At the execution of Python, we get :
File "main.py", line 15, in <module>
my_rust_model.set_variables(2)
TypeError
It is a type error and Python points to it, so it's easy to identify and solve.
Wrong types in Python method signatures
Let's assume now that the return type of one of the methods of our Model class is wrong, for example the get_results
method that is expected to return a Vec<f64>
in Rust, a list in Python.
class Model:
def set_variables(self, inputs):
self.inputs = inputs
def compute(self):
self.results = [elt**2 -3 for elt in self.inputs]
def get_results(self):
return self.results[0]
#return self.results <-- this is the expected output
This call results in the following panic:
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: PyErr { type: Py(0x10dcf79f0, PhantomData) }
This error code is not helpful for a Python user that does not know anything about Rust, or someone that does not know PyO3 was used to interface the Rust code.
However, as we are responsible for making the Rust code available to Python, we can do something about it.
The issue is that we called unwrap
anywhere we could, and therefore any panic from PyO3 will be directly forwarded to the end user.
Let's modify the code performing the type conversion to give a helpful error message to the Python user:
We used in our get_results
method the following call that performs the type conversion:
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyList;
pub trait Model {
fn set_variables(&mut self, inputs: &Vec<f64>);
fn compute(&mut self);
fn get_results(&self) -> Vec<f64>;
}
#[pyclass]
struct UserModel {
model: Py<PyAny>,
}
impl Model for UserModel {
fn get_results(&self) -> Vec<f64> {
println!("Rust calling Python to get the results");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("get_results", (), None)
.unwrap()
.extract()
.unwrap()
})
}
fn set_variables(&mut self, var: &Vec<f64>) {
println!("Rust calling Python to set the variables");
Python::with_gil(|py| {
self.model.bind(py)
.call_method("set_variables", (PyList::new_bound(py, var),), None)
.unwrap();
})
}
fn compute(&mut self) {
println!("Rust calling Python to perform the computation");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("compute", (), None)
.unwrap();
})
}
}
Let's break it down in order to perform better error handling:
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyList;
pub trait Model {
fn set_variables(&mut self, inputs: &Vec<f64>);
fn compute(&mut self);
fn get_results(&self) -> Vec<f64>;
}
#[pyclass]
struct UserModel {
model: Py<PyAny>,
}
impl Model for UserModel {
fn get_results(&self) -> Vec<f64> {
println!("Get results from Rust calling Python");
Python::with_gil(|py| {
let py_result: Bound<'_, PyAny> = self
.model
.bind(py)
.call_method("get_results", (), None)
.unwrap();
if py_result.get_type().name().unwrap() != "list" {
panic!(
"Expected a list for the get_results() method signature, got {}",
py_result.get_type().name().unwrap()
);
}
py_result.extract()
})
.unwrap()
}
fn set_variables(&mut self, var: &Vec<f64>) {
println!("Rust calling Python to set the variables");
Python::with_gil(|py| {
let py_model = self.model.bind(py)
.call_method("set_variables", (PyList::new_bound(py, var),), None)
.unwrap();
})
}
fn compute(&mut self) {
println!("Rust calling Python to perform the computation");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("compute", (), None)
.unwrap();
})
}
}
By doing so, you catch the result of the Python computation and check its type in order to be able to deliver a better error message before performing the unwrapping.
Of course, it does not cover all the possible wrong outputs: the user could return a list of strings instead of a list of floats. In this case, a runtime panic would still occur due to PyO3, but with an error message much more difficult to decipher for non-rust user.
It is up to the developer exposing the rust code to decide how much effort to invest into Python type error handling and improved error messages.
The final code
Now let's expose the solve()
function to make it available from Python.
It is not possible to directly expose the solve
function to Python, as the type conversion cannot be performed.
It requires an object implementing the Model
trait as input.
However, the UserModel
already implements this trait.
Because of this, we can write a function wrapper that takes the UserModel
--which has already been exposed to Python--as an argument in order to call the core function solve
.
It is also required to make the struct public.
#![allow(dead_code)]
use pyo3::prelude::*;
use pyo3::types::PyList;
pub trait Model {
fn set_variables(&mut self, var: &Vec<f64>);
fn get_results(&self) -> Vec<f64>;
fn compute(&mut self);
}
pub fn solve<T: Model>(model: &mut T) {
println!("Magic solver that mutates the model into a resolved state");
}
#[pyfunction]
#[pyo3(name = "solve")]
pub fn solve_wrapper(model: &mut UserModel) {
solve(model);
}
#[pyclass]
pub struct UserModel {
model: Py<PyAny>,
}
#[pymodule]
fn trait_exposure(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<UserModel>()?;
m.add_function(wrap_pyfunction!(solve_wrapper, m)?)?;
Ok(())
}
#[pymethods]
impl UserModel {
#[new]
pub fn new(model: Py<PyAny>) -> Self {
UserModel { model }
}
pub fn set_variables(&mut self, var: Vec<f64>) {
println!("Set variables from Python calling Rust");
Model::set_variables(self, &var)
}
pub fn get_results(&mut self) -> Vec<f64> {
println!("Get results from Python calling Rust");
Model::get_results(self)
}
pub fn compute(&mut self) {
Model::compute(self)
}
}
impl Model for UserModel {
fn set_variables(&mut self, var: &Vec<f64>) {
println!("Rust calling Python to set the variables");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("set_variables", (PyList::new_bound(py, var),), None)
.unwrap();
})
}
fn get_results(&self) -> Vec<f64> {
println!("Get results from Rust calling Python");
Python::with_gil(|py| {
let py_result: Bound<'_, PyAny> = self
.model
.bind(py)
.call_method("get_results", (), None)
.unwrap();
if py_result.get_type().name().unwrap() != "list" {
panic!(
"Expected a list for the get_results() method signature, got {}",
py_result.get_type().name().unwrap()
);
}
py_result.extract()
})
.unwrap()
}
fn compute(&mut self) {
println!("Rust calling Python to perform the computation");
Python::with_gil(|py| {
self.model
.bind(py)
.call_method("compute", (), None)
.unwrap();
})
}
}
Typing and IDE hints for your Python package
PyO3 provides an easy to use interface to code native Python libraries in Rust. The accompanying Maturin allows you to build and publish them as a package. Yet, for a better user experience, Python libraries should provide typing hints and documentation for all public entities, so that IDEs can show them during development and type analyzing tools such as mypy
can use them to properly verify the code.
Currently the best solution for the problem is to manually maintain *.pyi
files and ship them along with the package.
There is a sketch of a roadmap towards completing the experimental-inspect
feature which may eventually lead to automatic type annotations generated by PyO3. This needs more testing and implementation, please see issue #2454.
Introduction to pyi
files
pyi
files (an abbreviation for Python Interface
) are called "stub files" in most of the documentation related to them. A very good definition of what it is can be found in old MyPy documentation:
A stubs file only contains a description of the public interface of the module without any implementations.
There is also extensive documentation on type stubs on the official Python typing documentation.
Most Python developers probably already encountered them when trying to use their IDE's "Go to Definition" function on any builtin type. For example, the definitions of a few standard exceptions look like this:
class BaseException(object):
args: Tuple[Any, ...]
__cause__: BaseException | None
__context__: BaseException | None
__suppress_context__: bool
__traceback__: TracebackType | None
def __init__(self, *args: object) -> None: ...
def __str__(self) -> str: ...
def __repr__(self) -> str: ...
def with_traceback(self: _TBE, tb: TracebackType | None) -> _TBE: ...
class SystemExit(BaseException):
code: int
class Exception(BaseException): ...
class StopIteration(Exception):
value: Any
As we can see, those are not full definitions containing implementation, but just a description of the interface. It is usually all that the user of the library needs.
What do the PEPs say?
At the time of writing this documentation, the pyi
files are referenced in three PEPs.
PEP8 - Style Guide for Python Code - #Function Annotations (last point) recommends all third party library creators to provide stub files as the source of knowledge about the package for type checker tools.
(...) it is expected that users of third party library packages may want to run type checkers over those packages. For this purpose PEP 484 recommends the use of stub files: .pyi files that are read by the type checker in preference of the corresponding .py files. (...)
PEP484 - Type Hints - #Stub Files defines stub files as follows.
Stub files are files containing type hints that are only for use by the type checker, not at runtime.
It contains a specification for them (highly recommended reading, since it contains at least one thing that is not used in normal Python code) and also some general information about where to store the stub files.
PEP561 - Distributing and Packaging Type Information describes in detail how to build packages that will enable type checking. In particular it contains information about how the stub files must be distributed in order for type checkers to use them.
How to do it?
PEP561 recognizes three ways of distributing type information:
inline
- the typing is placed directly in source (py
) files;separate package with stub files
- the typing is placed inpyi
files distributed in their own, separate package;in-package stub files
- the typing is placed inpyi
files distributed in the same package as source files.
The first way is tricky with PyO3 since we do not have py
files. When it has been investigated and necessary changes are implemented, this document will be updated.
The second way is easy to do, and the whole work can be fully separated from the main library code. The example repo for the package with stub files can be found in PEP561 references section: Stub package repository
The third way is described below.
Including pyi
files in your PyO3/Maturin build package
When source files are in the same package as stub files, they should be placed next to each other. We need a way to do that with Maturin. Also, in order to mark our package as typing-enabled we need to add an empty file named py.typed
to the package.
If you do not have other Python files
If you do not need to add any other Python files apart from pyi
to the package, Maturin provides a way to do most of the work for you. As documented in the Maturin Guide, the only thing you need to do is to create a stub file for your module named <module_name>.pyi
in your project root and Maturin will do the rest.
my-rust-project/
├── Cargo.toml
├── my_project.pyi # <<< add type stubs for Rust functions in the my_project module here
├── pyproject.toml
└── src
└── lib.rs
For an example pyi
file see the my_project.pyi
content section.
If you need other Python files
If you need to add other Python files apart from pyi
to the package, you can do it also, but that requires some more work. Maturin provides an easy way to add files to a package (documentation). You just need to create a folder with the name of your module next to the Cargo.toml
file (for customization see documentation linked above).
The folder structure would be:
my-project
├── Cargo.toml
├── my_project
│ ├── __init__.py
│ ├── my_project.pyi
│ ├── other_python_file.py
│ └── py.typed
├── pyproject.toml
├── Readme.md
└── src
└── lib.rs
Let's go a little bit more into detail regarding the files inside the package folder.
__init__.py
content
As we now specify our own package content, we have to provide the __init__.py
file, so the folder is treated as a package and we can import things from it. We can always use the same content that Maturin creates for us if we do not specify a Python source folder. For PyO3 bindings it would be:
from .my_project import *
That way everything that is exposed by our native module can be imported directly from the package.
py.typed
requirement
As stated in PEP561:
Package maintainers who wish to support type checking of their code MUST add a marker file named py.typed to their package supporting typing. This marker applies recursively: if a top-level package includes it, all its sub-packages MUST support type checking as well.
If we do not include that file, some IDEs might still use our pyi
files to show hints, but the type checkers might not. MyPy will raise an error in this situation:
error: Skipping analyzing "my_project": found module but no type hints or library stubs
The file is just a marker file, so it should be empty.
my_project.pyi
content
Our module stub file. This document does not aim at describing how to write them, since you can find a lot of documentation on it, starting from the already quoted PEP484.
The example can look like this:
class Car:
"""
A class representing a car.
:param body_type: the name of body type, e.g. hatchback, sedan
:param horsepower: power of the engine in horsepower
"""
def __init__(self, body_type: str, horsepower: int) -> None: ...
@classmethod
def from_unique_name(cls, name: str) -> 'Car':
"""
Creates a Car based on unique name
:param name: model name of a car to be created
:return: a Car instance with default data
"""
def best_color(self) -> str:
"""
Gets the best color for the car.
:return: the name of the color our great algorithm thinks is the best for this car
"""
Changelog
All notable changes to this project will be documented in this file. For help with updating to new PyO3 versions, please see the migration guide.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
To see unreleased changes, please see the CHANGELOG on the main branch guide.
0.21.1 - 2024-04-01
Added
- Implement
Send
andSync
forPyBackedStr
andPyBackedBytes
. #4007 - Implement
Clone
,Debug
,PartialEq
,Eq
,PartialOrd
,Ord
andHash
implementation forPyBackedBytes
andPyBackedStr
, andDisplay
forPyBackedStr
. #4020 - Add
import_exception_bound!
macro to import exception types without generating GIL Ref functionality for them. #4027
Changed
- Emit deprecation warning for uses of GIL Refs as
#[setter]
function arguments. #3998 - Add
#[inline]
hints on manyBound
andBorrowed
methods. #4024
Fixed
- Handle
#[pyo3(from_py_with = "")]
in#[setter]
methods #3995 - Allow extraction of
&Bound
in#[setter]
methods. #3998 - Fix some uncovered code blocks emitted by
#[pymodule]
,#[pyfunction]
and#[pyclass]
macros. #4009 - Fix typo in the panic message when a class referenced in
pyo3::import_exception!
does not exist. #4012 - Fix compile error when using an async
#[pymethod]
with a receiver and additional arguments. #4015
0.21.0 - 2024-03-25
Added
- Add support for GraalPy (24.0 and up). #3247
- Add
PyMemoryView
type. #3514 - Allow
async fn
in for#[pyfunction]
and#[pymethods]
, with theexperimental-async
feature. #3540 #3588 #3599 #3931 - Implement
PyTypeInfo
forPyEllipsis
,PyNone
andPyNotImplemented
. #3577 - Support
#[pyclass]
on enums that have non-unit variants. #3582 - Support
chrono
feature withabi3
feature. #3664 FromPyObject
,IntoPy<PyObject>
andToPyObject
are implemented onstd::duration::Duration
#3670- Add
PyString::to_cow
. AddPy<PyString>::to_str
,Py<PyString>::to_cow
, andPy<PyString>::to_string_lossy
, as ways to access Python string data safely beyond the GIL lifetime. #3677 - Add
Bound<T>
andBorrowed<T>
smart pointers as a new API for accessing Python objects. #3686 - Add
PyNativeType::as_borrowed
to convert "GIL refs" to the newBound
smart pointer. #3692 - Add
FromPyObject::extract_bound
method, to migrateFromPyObject
implementations to the Bound API. #3706 - Add
gil-refs
feature to allow continued use of the deprecated GIL Refs APIs. #3707 - Add methods to
PyAnyMethods
for binary operators (add
,sub
, etc.) #3712 - Add
chrono-tz
feature allowing conversion betweenchrono_tz::Tz
andzoneinfo.ZoneInfo
#3730 - Add FFI definition
PyType_GetModuleByDef
. #3734 - Conversion between
std::time::SystemTime
anddatetime.datetime
#3736 - Add
Py::as_any
andPy::into_any
. #3785 - Add
PyStringMethods::encode_utf8
. #3801 - Add
PyBackedStr
andPyBackedBytes
, as alternatives to&str
and&bytes
where a Python object owns the data. #3802 #3991 - Allow
#[pymodule]
macro on Rustmod
blocks, with theexperimental-declarative-modules
feature. #3815 - Implement
ExactSizeIterator
forset
andfrozenset
iterators onabi3
feature. #3849 - Add
Py::drop_ref
to explicitly drop a `Py`` and immediately decrease the Python reference count if the GIL is already held. #3871 - Allow
#[pymodule]
macro on single argument functions that take&Bound<'_, PyModule>
. #3905 - Implement
FromPyObject
forCow<str>
. #3928 - Implement
Default
forGILOnceCell
. #3971 - Add
PyDictMethods::into_mapping
,PyListMethods::into_sequence
andPyTupleMethods::into_sequence
. #3982
Changed
PyDict::from_sequence
now takes a single argument of type&PyAny
(previously took two argumentsPython
andPyObject
). #3532- Deprecate
Py::is_ellipsis
andPyAny::is_ellipsis
in favour ofany.is(py.Ellipsis())
. #3577 - Split some
PyTypeInfo
functionality into new traitsHasPyGilRef
andPyTypeCheck
. #3600 - Deprecate
PyTryFrom
andPyTryInto
traits in favor ofany.downcast()
via thePyTypeCheck
andPyTypeInfo
traits. #3601 - Allow async methods to accept
&self
/&mut self
#3609 FromPyObject
for set types now also acceptfrozenset
objects as input. #3632FromPyObject
forbool
now also accepts NumPy'sbool_
as input. #3638- Add
AsRefSource
associated type toPyNativeType
. #3653 - Rename
.is_true
to.is_truthy
onPyAny
andPy<PyAny>
to clarify that the test is not based on identity with or equality to the True singleton. #3657 PyType::name
is nowPyType::qualname
whereasPyType::name
efficiently accesses the full name which includes the module name. #3660- The
Iter(A)NextOutput
types are now deprecated and__(a)next__
can directly return anything which can be converted into Python objects, i.e. awaitables do not need to be wrapped intoIterANextOutput
orOption
any more.Option
can still be used as well and returningNone
will trigger the fast path for__next__
, stopping iteration without having to raise aStopIteration
exception. #3661 - Implement
FromPyObject
onchrono::DateTime<Tz>
for allTz
, not justFixedOffset
andUtc
. #3663 - Add lifetime parameter to
PyTzInfoAccess
trait. For the deprecated gil-ref API, the trait is now implemented for&'py PyTime
and&'py PyDateTime
instead ofPyTime
andPyDate
. #3679 - Calls to
__traverse__
become no-ops for unsendable pyclasses if on the wrong thread, thereby avoiding hard aborts at the cost of potential leakage. #3689 - Include
PyNativeType
inpyo3::prelude
. #3692 - Improve performance of
extract::<i64>
(and other integer types) by avoiding call to__index__()
converting the value to an integer for 3.10+. Gives performance improvement of around 30% for successful extraction. #3742 - Relax bound of
FromPyObject
forPy<T>
to justT: PyTypeCheck
. #3776 PySet
andPyFrozenSet
iterators now always iterate the equivalent ofiter(set)
. (A "fast path" with no noticeable performance benefit was removed.) #3849- Move implementations of
FromPyObject
for&str
,Cow<str>
,&[u8]
andCow<[u8]>
onto a temporary traitFromPyObjectBound
whengil-refs
feature is deactivated. #3928 - Deprecate
GILPool
,Python::with_pool
, andPython::new_pool
. #3947
Removed
- Remove all functionality deprecated in PyO3 0.19. #3603
Fixed
- Match PyPy 7.3.14 in removing PyPy-only symbol
Py_MAX_NDIMS
in favour ofPyBUF_MAX_NDIM
. #3757 - Fix segmentation fault using
datetime
types when an invaliddatetime
module is on sys.path. #3818 - Fix
non_local_definitions
lint warning triggered by many PyO3 macros. #3901 - Disable
PyCode
andPyCode_Type
on PyPy:PyCode_Type
is not exposed by PyPy. #3934
0.21.0-beta.0 - 2024-03-10
Prerelease of PyO3 0.21. See the GitHub diff for what changed between 0.21.0-beta.0 and the final release.
0.20.3 - 2024-02-23
Packaging
- Add
portable-atomic
dependency. #3619 - Check maximum version of Python at build time and for versions not yet supported require opt-in to the
abi3
stable ABI by the environment variablePYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
. #3821
Fixed
- Use
portable-atomic
to support platforms without 64-bit atomics. #3619 - Fix compilation failure with
either
feature enabled withoutexperimental-inspect
enabled. #3834
0.20.2 - 2024-01-04
Packaging
- Pin
pyo3
andpyo3-ffi
dependencies onpyo3-build-config
to require the same patch version, i.e.pyo3
0.20.2 requires exactlypyo3-build-config
0.20.2. #3721
Fixed
- Fix compile failure when building
pyo3
0.20.0 with latestpyo3-build-config
0.20.X. #3724 - Fix docs.rs build. #3722
0.20.1 - 2023-12-30
Added
- Add optional
either
feature to add conversions foreither::Either<L, R>
sum type. #3456 - Add optional
smallvec
feature to add conversions forsmallvec::SmallVec
. #3507 - Add
take
andinto_inner
methods toGILOnceCell
#3556 #[classmethod]
methods can now also receivePy<PyType>
as their first argument. #3587#[pyfunction(pass_module)]
can now also receivePy<PyModule>
as their first argument. #3587- Add
traverse
method toGILProtected
. #3616 - Added
abi3-py312
feature #3687
Fixed
- Fix minimum version specification for optional
chrono
dependency. #3512 - Silenced new
clippy::unnecessary_fallible_conversions
warning when using aPy<Self>
self
receiver. #3564
0.20.0 - 2023-10-11
Packaging
- Dual-license PyO3 under either the Apache 2.0 OR the MIT license. This makes the project GPLv2 compatible. #3108
- Update MSRV to Rust 1.56. #3208
- Bump
indoc
dependency to 2.0 andunindent
dependency to 0.2. #3237 - Bump
syn
dependency to 2.0. #3239 - Drop support for debug builds of Python 3.7. #3387
- Bump
chrono
optional dependency to require 0.4.25 or newer. #3427 - Support Python 3.12. #3488
Added
- Support
__lt__
,__le__
,__eq__
,__ne__
,__gt__
and__ge__
in#[pymethods]
. #3203 - Add FFI definition
Py_GETENV
. #3336 - Add
as_ptr
andinto_ptr
inherent methods forPy
,PyAny
,PyRef
, andPyRefMut
. #3359 - Implement
DoubleEndedIterator
forPyTupleIterator
andPyListIterator
. #3366 - Add
#[pyclass(rename_all = "...")]
option: this allows renaming all getters and setters of a struct, or all variants of an enum. Available renaming rules are:"camelCase"
,"kebab-case"
,"lowercase"
,"PascalCase"
,"SCREAMING-KEBAB-CASE"
,"SCREAMING_SNAKE_CASE"
,"snake_case"
,"UPPERCASE"
. #3384 - Add FFI definitions
PyObject_GC_IsTracked
andPyObject_GC_IsFinalized
on Python 3.9 and up (PyPy 3.10 and up). #3403 - Add types for
None
,Ellipsis
, andNotImplemented
. #3408 - Add FFI definitions for the
Py_mod_multiple_interpreters
constant and its possible values. #3494 - Add FFI definitions for
PyInterpreterConfig
struct, its constants andPy_NewInterpreterFromConfig
. #3502
Changed
- Change
PySet::discard
to returnPyResult<bool>
(previously returned nothing). #3281 - Optimize implmentation of
IntoPy
for Rust tuples to Python tuples. #3321 - Change
PyDict::get_item
to no longer suppress arbitrary exceptions (the return type is nowPyResult<Option<&PyAny>>
instead ofOption<&PyAny>
), and deprecatePyDict::get_item_with_error
. #3330 - Deprecate FFI definitions which are deprecated in Python 3.12. #3336
AsPyPointer
is now anunsafe trait
. #3358- Accept all
os.PathLike
values in implementation ofFromPyObject
forPathBuf
. #3374 - Add
__builtins__
to globals inpy.run()
andpy.eval()
if they're missing. #3378 - Optimize implementation of
FromPyObject
forBigInt
andBigUint
. #3379 PyIterator::from_object
andPyByteArray::from
now take a single argument of type&PyAny
(previously took two argumentsPython
andAsPyPointer
). #3389- Replace
AsPyPointer
withAsRef<PyAny>
as a bound in the blanket implementation ofFrom<&T> for PyObject
. #3391 - Replace blanket
impl IntoPy<PyObject> for &T where T: AsPyPointer
with implementations ofimpl IntoPy<PyObject>
for&PyAny
,&T where T: AsRef<PyAny>
, and&Py<T>
. #3393 - Preserve
std::io::Error
kind in implementation ofFrom<std::io::IntoInnerError>
forPyErr
#3396 - Try to select a relevant
ErrorKind
in implementation ofFrom<PyErr>
forOSError
subclass. #3397 - Retrieve the original
PyErr
in implementation ofFrom<std::io::Error>
forPyErr
if thestd::io::Error
has been built using a Python exception (previously would create a new exception wrapping thestd::io::Error
). #3402 #[pymodule]
will now return the same module object on repeated import by the same Python interpreter, on Python 3.9 and up. #3446- Truncate leap-seconds and warn when converting
chrono
types to Pythondatetime
types (datetime
cannot represent leap-seconds). #3458 Err
returned from#[pyfunction]
will now have a non-None__context__
if called from inside acatch
block. #3455- Deprecate undocumented
#[__new__]
form of#[new]
attribute. #3505
Removed
- Remove all functionality deprecated in PyO3 0.18, including
#[args]
attribute for#[pymethods]
. #3232 - Remove
IntoPyPointer
trait in favour ofinto_ptr
inherent methods. #3385
Fixed
- Handle exceptions properly in
PySet::discard
. #3281 - The
PyTupleIterator
type returned byPyTuple::iter
is now public and hence can be named by downstream crates. #3366 - Linking of
PyOS_FSPath
on PyPy. #3374 - Fix memory leak in
PyTypeBuilder::build
. #3401 - Disable removed FFI definitions
_Py_GetAllocatedBlocks
,_PyObject_GC_Malloc
, and_PyObject_GC_Calloc
on Python 3.11 and up. #3403 - Fix
ResourceWarning
and crashes related to GC when running with debug builds of CPython. #3404 - Some-wrapping of
Option<T>
default arguments will no longer re-wrapSome(T)
or expressions evaluating toNone
. #3461 - Fix
IterNextOutput::Return
not returning a value on PyPy. #3471 - Emit compile errors instead of ignoring macro invocations inside
#[pymethods]
blocks. #3491 - Emit error on invalid arguments to
#[new]
,#[classmethod]
,#[staticmethod]
, and#[classattr]
. #3484 - Disable
PyMarshal_WriteObjectToString
fromPyMarshal_ReadObjectFromString
with theabi3
feature. #3490 - Fix FFI definitions for
_PyFrameEvalFunction
on Python 3.11 and up (it now receives a_PyInterpreterFrame
opaque struct). #3500
0.19.2 - 2023-08-01
Added
- Add FFI definitions
PyState_AddModule
,PyState_RemoveModule
andPyState_FindModule
for PyPy 3.9 and up. #3295 - Add FFI definitions
_PyObject_CallFunction_SizeT
and_PyObject_CallMethod_SizeT
. #3297 - Add a "performance" section to the guide collecting performance-related tricks and problems. #3304
- Add
PyErr::Display
for all Python versions, and FFI symbolPyErr_DisplayException
for Python 3.12. #3334 - Add FFI definition
PyType_GetDict()
for Python 3.12. #3339 - Add
PyAny::downcast_exact
. #3346 - Add
PySlice::full()
to construct a full slice (::
). #3353
Changed
- Update
PyErr
for 3.12 betas to avoid deprecated ffi methods. #3306 - Update FFI definitions of
object.h
for Python 3.12.0b4. #3335 - Update
pyo3::ffi
struct definitions to be compatible with 3.12.0b4. #3342 - Optimize conversion of
float
tof64
(andPyFloat::value
) on non-abi3 builds. #3345
Fixed
- Fix timezone conversion bug for FixedOffset datetimes that were being incorrectly converted to and from UTC. #3269
- Fix
SystemError
raised inPyUnicodeDecodeError_Create
on PyPy 3.10. #3297 - Correct FFI definition
Py_EnterRecursiveCall
to returnc_int
(was incorrectly returning()
). #3300 - Fix case where
PyErr::matches
andPyErr::is_instance
returned results inconsistent withPyErr::get_type
. #3313 - Fix loss of panic message in
PanicException
when unwinding after the exception was "normalized". #3326 - Fix
PyErr::from_value
andPyErr::into_value
losing traceback on conversion. #3328 - Fix reference counting of immortal objects on Python 3.12.0b4. #3335
0.19.1 - 2023-07-03
Packaging
- Extend range of supported versions of
hashbrown
optional dependency to include version 0.14 #3258 - Extend range of supported versions of
indexmap
optional dependency to include version 2. #3277 - Support PyPy 3.10. #3289
Added
- Add
pyo3::types::PyFrozenSetBuilder
to allow building aPyFrozenSet
item by item. #3156 - Add support for converting to and from Python's
ipaddress.IPv4Address
/ipaddress.IPv6Address
andstd::net::IpAddr
. #3197 - Add support for
num-bigint
feature in combination withabi3
. #3198 - Add
PyErr_GetRaisedException()
,PyErr_SetRaisedException()
to FFI definitions for Python 3.12 and later. #3248 - Add
Python::with_pool
which is a safer but more limited alternative toPython::new_pool
. #3263 - Add
PyDict::get_item_with_error
on PyPy. #3270 - Allow
#[new]
methods may to returnPy<Self>
in order to return existing instances. #3287
Fixed
- Fix conversion of classes implementing
__complex__
toComplex
when usingabi3
or PyPy. #3185 - Stop suppressing unrelated exceptions in
PyAny::hasattr
. #3271 - Fix memory leak when creating
PySet
orPyFrozenSet
or returning types converted into these internally, e.g.HashSet
orBTreeSet
. #3286
0.19.0 - 2023-05-31
Packaging
- Correct dependency on syn to version 1.0.85 instead of the incorrect version 1.0.56. #3152
Added
- Accept
text_signature
option (and automatically generate signature) for#[new]
in#[pymethods]
. #2980 - Add support for converting to and from Python's
decimal.Decimal
andrust_decimal::Decimal
. #3016 - Add
#[pyo3(from_item_all)]
when derivingFromPyObject
to specifyget_item
as getter for all fields. #3120 - Add
pyo3::exceptions::PyBaseExceptionGroup
for Python 3.11, and corresponding FFI definitionPyExc_BaseExceptionGroup
. #3141 - Accept
#[new]
with#[classmethod]
to create a constructor which receives a (subtype's) class/PyType
as its first argument. #3157 - Add
PyClass::get
andPy::get
for GIL-indepedent access to classes with#[pyclass(frozen)]
. #3158 - Add
PyAny::is_exact_instance
andPyAny::is_exact_instance_of
. #3161
Changed
PyAny::is_instance_of::<T>(obj)
is now equivalent toT::is_type_of(obj)
, and now returnsbool
instead ofPyResult<bool>
. #2881- Deprecate
text_signature
option on#[pyclass]
structs. #2980 - No longer wrap
anyhow::Error
/eyre::Report
containing a basicPyErr
without a chain in aPyRuntimeError
. #3004 -
- Change
#[getter]
and#[setter]
to use a common call "trampoline" to slightly reduce generated code size and compile times. #3029
- Change
- Improve default values for str, numbers and bool in automatically-generated
text_signature
. #3050 - Improve default value for
None
in automatically-generatedtext_signature
. #3066 - Rename
PySequence::list
andPySequence::tuple
toPySequence::to_list
andPySequence::to_tuple
. (The old names continue to exist as deprecated forms.) #3111 - Extend the lifetime of the GIL token returned by
PyRef::py
andPyRefMut::py
to match the underlying borrow. #3131 - Safe access to the GIL, for example via
Python::with_gil
, is now locked inside of implementations of the__traverse__
slot. #3168
Removed
- Remove all functionality deprecated in PyO3 0.17, most prominently
Python::acquire_gil
is replaced byPython::with_gil
. #2981
Fixed
- Correct FFI definitions
PyGetSetDef
,PyMemberDef
,PyStructSequence_Field
andPyStructSequence_Desc
to have*const c_char
members forname
anddoc
(not*mut c_char
). #3036 - Fix panic on
fmt::Display
, instead return"<unprintable object>"
string and report error viasys.unraisablehook()
#3062 - Fix a compile error of "temporary value dropped while borrowed" when
#[pyfunction]
s take references into#[pyclass]
es #3142 - Fix crashes caused by PyO3 applying deferred reference count updates when entering a
__traverse__
implementation. #3168 - Forbid running the
Drop
implementations of unsendable classes on other threads. #3176 - Fix a compile error when
#[pymethods]
items come from somewhere else (for example, as a macro argument) and a custom receiver likePy<Self>
is used. #3178
0.18.3 - 2023-04-13
Added
- Add
GILProtected<T>
to mediate concurrent access to a value using Python's global interpreter lock (GIL). #2975 - Support
PyASCIIObject
/PyUnicode
and associated methods on big-endian architectures. #3015 - Add FFI definition
_PyDict_Contains_KnownHash()
for CPython 3.10 and up. #3088
Fixed
- Fix compile error for
#[pymethods]
and#[pyfunction]
called "output". #3022 - Fix compile error in generated code for magic methods implemented as a
#[staticmethod]
. #3055 - Fix
is_instance
forPyDateTime
(would incorrectly check for aPyDate
). #3071 - Fix upstream deprecation of
PyUnicode_InternImmortal
since Python 3.10. #3071
0.18.2 - 2023-03-24
Packaging
- Disable default features of
chrono
to avoid depending ontime
v0.1.x. #2939
Added
- Implement
IntoPy<PyObject>
,ToPyObject
andFromPyObject
forCow<[u8]>
to efficiently handle bothbytes
andbytearray
objects. #2899 - Implement
IntoPy<PyObject>
,ToPyObject
andFromPyObject
forCell<T>
. #3014 - Add
PyList::to_tuple()
, as a convenient and efficient conversion from lists to tuples. #3042 - Add
PyTuple::to_list()
, as a convenient and efficient conversion from tuples to lists. #3044
Changed
- Optimize
PySequence
conversion forlist
andtuple
inputs. #2944 - Improve exception raised when creating
#[pyclass]
type object fails during module import. #2947 - Optimize
PyMapping
conversion fordict
inputs. #2954 - Allow
create_exception!
to take adotted.module
to place the exception in a submodule. #2979
Fixed
- Fix a reference counting race condition affecting
PyObject
s cloned inallow_threads
blocks. #2952 - Fix
clippy::redundant_closure
lint on default arguments in#[pyo3(signature = (...))]
annotations. #2990 - Fix
non_snake_case
lint on generated code in#[pyfunction]
macro. #2993 - Fix some FFI definitions for the upcoming PyPy 3.10 release. #3031
0.18.1 - 2023-02-07
Added
- Add
PyErr::write_unraisable()
. #2889 - Add
Python::Ellipsis()
andPyAny::is_ellipsis()
methods. #2911 - Add
PyDict::update()
andPyDict::update_if_missing()
methods. #2912
Changed
- FFI definition
PyIter_Check
on CPython 3.7 is now implemented ashasattr(type(obj), "__next__")
, which works correctly on all platforms and adds support forabi3
. #2914 - Warn about unknown config keys in
PYO3_CONFIG_FILE
instead of denying. #2926
Fixed
- Send errors returned by
__releasebuffer__
tosys.unraisablehook
rather than causingSystemError
. #2886 - Fix downcast to
PyIterator
succeeding for Python classes which did not implement__next__
. #2914 - Fix segfault in
__traverse__
when visitingNone
fields ofOption<T: AsPyPointer>
. #2921 - Fix
#[pymethods(crate = "...")]
option being ignored. #2923 - Link against
pythonXY_d.dll
for debug Python builds on Windows. #2937
0.18.0 - 2023-01-17
Packaging
- Relax
indexmap
optional depecency to allow>= 1.6, < 2
. #2849 - Relax
hashbrown
optional dependency to allow>= 0.9, < 0.14
. #2875 - Update
memoffset
dependency to 0.8. #2875
Added
- Add
GILOnceCell::get_or_try_init
for fallibleGILOnceCell
initialization. #2398 - Add experimental feature
experimental-inspect
withtype_input()
andtype_output()
helpers to get the Python type of any Python-compatible object. #2490 #2882 - The
#[pyclass]
macro can now takeget_all
andset_all
to create getters and setters for every field. #2692 - Add
#[pyo3(signature = (...))]
option for#[pyfunction]
and#[pymethods]
. #2702 pyo3-build-config
: rebuild whenPYO3_ENVIRONMENT_SIGNATURE
environment variable value changes. #2727- Add conversions between non-zero int types in
std::num
and Pythonint
. #2730 - Add
Py::downcast()
as a companion toPyAny::downcast()
, as well asdowncast_unchecked()
for both types. #2734 - Add types for all built-in
Warning
classes as well asPyErr::warn_explicit
. #2742 - Add
abi3-py311
feature. #2776 - Add FFI definition
_PyErr_ChainExceptions()
for CPython. #2788 - Add FFI definitions
PyVectorcall_NARGS
andPY_VECTORCALL_ARGUMENTS_OFFSET
for PyPy 3.8 and up. #2811 - Add
PyList::get_item_unchecked
for PyPy. #2827
Changed
- PyO3's macros now emit a much nicer error message if function return values don't implement the required trait(s). #2664
- Use a TypeError, rather than a ValueError, when refusing to treat a str as a Vec. #2685
- Change
PyCFunction::new_closure
to takename
anddoc
arguments. #2686 PyType::is_subclass
,PyErr::is_instance
andPyAny::is_instance
now take&PyAny
instead of&PyType
arguments, so that they work with objects that pretend to be types using__subclasscheck__
and__instancecheck__
. #2695- Deprecate
#[args]
attribute and passing "args" specification directly to#[pyfunction]
in favor of the new#[pyo3(signature = (...))]
option. #2702 - Deprecate required arguments after
Option<T>
arguments to#[pyfunction]
and#[pymethods]
without also using#[pyo3(signature)]
to specify whether the arguments should be required or have defaults. #2703 - Change
#[pyfunction]
and#[pymethods]
to use a common call "trampoline" to slightly reduce generated code size and compile times. #2705 PyAny::cast_as()
andPy::cast_as()
are now deprecated in favor ofPyAny::downcast()
and the newPy::downcast()
. #2734- Relax lifetime bounds on
PyAny::downcast()
. #2734 - Automatically generate
__text_signature__
for all Python functions created using#[pyfunction]
and#[pymethods]
. #2784 - Accept any iterator in
PySet::new
andPyFrozenSet::new
. #2795 - Mixing
#[cfg(...)]
and#[pyo3(...)]
attributes on#[pyclass]
struct fields will now work. #2796 - Re-enable
PyFunction
on when building for abi3 or PyPy. #2838 - Improve
derive(FromPyObject)
to useintern!
when applicable for#[pyo3(item)]
. #2879
Removed
- Remove the deprecated
pyproto
feature,#[pyproto]
macro, and all accompanying APIs. #2587 - Remove all functionality deprecated in PyO3 0.16. #2843
Fixed
- Disable
PyModule::filename
on PyPy. #2715 PyCodeObject
is now once again defined with fields on Python 3.7. #2726- Raise a
TypeError
if#[new]
pymethods with no arguments receive arguments when called from Python. #2749 - Use the
NOARGS
argument calling convention for methods that have a singlepy: Python
argument (as a performance optimization). #2760 - Fix truncation of
isize
values toc_long
inPySlice::new
. #2769 - Fix soundness issue with FFI definition
PyUnicodeDecodeError_Create
on PyPy leading to indeterminate behavior (typically aTypeError
). #2772 - Allow functions taking
**kwargs
to accept keyword arguments which share a name with a positional-only argument (as permitted by PEP 570). #2800 - Fix unresolved symbol for
PyObject_Vectorcall
on PyPy 3.9 and up. #2811 - Fix memory leak in
PyCFunction::new_closure
. #2842
0.17.3 - 2022-11-01
Packaging
- Support Python 3.11. (Previous versions of PyO3 0.17 have been tested against Python 3.11 release candidates and are expected to be compatible, this is the first version tested against Python 3.11.0.) #2708
Added
- Implemented
ExactSizeIterator
forPyListIterator
,PyDictIterator
,PySetIterator
andPyFrozenSetIterator
. #2676
Fixed
- Fix regression of
impl FromPyObject for [T; N]
no longer accepting types passingPySequence_Check
, e.g. NumPy arrays, since version 0.17.0. This the same fix that was appliedimpl FromPyObject for Vec<T>
in version 0.17.1 extended to fixed-size arrays. #2675 - Fix UB in
FunctionDescription::extract_arguments_fastcall
due to creating slices from a null pointer. #2687
0.17.2 - 2022-10-04
Packaging
- Added optional
chrono
feature to convertchrono
types into types in thedatetime
module. #2612
Added
- Add support for
num-bigint
feature onPyPy
. #2626
Fixed
- Correctly implement
__richcmp__
for enums, fixing__ne__
returning always returningTrue
. #2622 - Fix compile error since 0.17.0 with
Option<&SomePyClass>
argument with a default. #2630 - Fix regression of
impl FromPyObject for Vec<T>
no longer accepting types passingPySequence_Check
, e.g. NumPy arrays, since 0.17.0. #2631
0.17.1 - 2022-08-28
Fixed
- Fix visibility of
PyDictItems
,PyDictKeys
, andPyDictValues
types added in PyO3 0.17.0. - Fix compile failure when using
#[pyo3(from_py_with = "...")]
attribute on an argument of typeOption<T>
. #2592 - Fix clippy
redundant-closure
lint on**kwargs
arguments for#[pyfunction]
and#[pymethods]
. #2595
0.17.0 - 2022-08-23
Packaging
- Update inventory dependency to
0.3
(themultiple-pymethods
feature now requires Rust 1.62 for correctness). #2492
Added
- Add
timezone_utc
. #1588 - Implement
ToPyObject
for[T; N]
. #2313 - Add
PyDictKeys
,PyDictValues
andPyDictItems
Rust types. #2358 - Add
append_to_inittab
. #2377 - Add FFI definition
PyFrame_GetCode
. #2406 - Add
PyCode
andPyFrame
high level objects. #2408 - Add FFI definitions
Py_fstring_input
,sendfunc
, and_PyErr_StackItem
. #2423 - Add
PyDateTime::new_with_fold
,PyTime::new_with_fold
,PyTime::get_fold
, andPyDateTime::get_fold
for PyPy. #2428 - Add
#[pyclass(frozen)]
. #2448 - Accept
#[pyo3(name)]
on enum variants. #2457 - Add
CompareOp::matches
to implement__richcmp__
as the result of a Ruststd::cmp::Ordering
comparison. #2460 - Add
PySuper
type. #2486 - Support PyPy on Windows with the
generate-import-lib
feature. #2506 - Add FFI definitions
Py_EnterRecursiveCall
andPy_LeaveRecursiveCall
. #2511 - Add
PyDict::get_item_with_error
. #2536 - Add
#[pyclass(sequence)]
option. #2567
Changed
- Change datetime constructors taking a
tzinfo
to takeOption<&PyTzInfo>
instead ofOption<&PyObject>
:PyDateTime::new
,PyDateTime::new_with_fold
,PyTime::new
, andPyTime::new_with_fold
. #1588 - Move
PyTypeObject::type_object
method to thePyTypeInfo
trait, and deprecate thePyTypeObject
trait. #2287 - Methods of
Py
andPyAny
now acceptimpl IntoPy<Py<PyString>>
rather than just&str
to allow use of theintern!
macro. #2312 - Change the deprecated
pyproto
feature to be opt-in instead of opt-out. #2322 - Emit better error messages when
#[pyfunction]
return types do not implementIntoPy
. #2326 - Require
T: IntoPy
forimpl<T, const N: usize> IntoPy<PyObject> for [T; N]
instead ofT: ToPyObject
. #2326 - Deprecate the
ToBorrowedObject
trait. #2333 - Iterators over
PySet
andPyDict
will now panic if the underlying collection is mutated during the iteration. #2380 - Iterators over
PySet
andPyDict
will now panic if the underlying collection is mutated during the iteration. #2380 - Allow
#[classattr]
methods to be fallible. #2385 - Prevent multiple
#[pymethods]
with the same name for a single#[pyclass]
. #2399 - Fixup
lib_name
when usingPYO3_CONFIG_FILE
. #2404 - Add a message to the
ValueError
raised by the#[derive(FromPyObject)]
implementation for a tuple struct. #2414 - Allow
#[classattr]
methods to takePython
argument. #2456 - Rework
PyCapsule
type to resolve soundness issues: #2485PyCapsule::new
andPyCapsule::new_with_destructor
now takename: Option<CString>
instead of&CStr
.- The destructor
F
inPyCapsule::new_with_destructor
must now beSend
. PyCapsule::get_context
deprecated in favor ofPyCapsule::context
which doesn't take apy: Python<'_>
argument.PyCapsule::set_context
no longer takes apy: Python<'_>
argument.PyCapsule::name
now returnsPyResult<Option<&CStr>>
instead of&CStr
.
FromPyObject::extract
forVec<T>
no longer accepts Pythonstr
inputs. #2500- Ensure each
#[pymodule]
is only initialized once. #2523 pyo3_build_config::add_extension_module_link_args
now also emits linker arguments forwasm32-unknown-emscripten
. #2538- Type checks for
PySequence
andPyMapping
now require inputs to inherit from (or register with)collections.abc.Sequence
andcollections.abc.Mapping
respectively. #2477 - Disable
PyFunction
on when building for abi3 or PyPy. #2542 - Deprecate
Python::acquire_gil
. #2549
Removed
- Remove all functionality deprecated in PyO3 0.15. #2283
- Make the
Dict
,WeakRef
andBaseNativeType
members of thePyClass
private implementation details. #2572
Fixed
- Enable incorrectly disabled FFI definition
PyThreadState_DeleteCurrent
. #2357 - Fix
wrap_pymodule
interactions with name resolution rules: it no longer "sees through" glob imports ofuse submodule::*
whensubmodule::submodule
is a#[pymodule]
. #2363 - Correct FFI definition
PyEval_EvalCodeEx
to take*const *mut PyObject
array arguments instead of*mut *mut PyObject
. #2368 - Fix "raw-ident" structs (e.g.
#[pyclass] struct r#RawName
) incorrectly havingr#
at the start of the class name created in Python. #2395 - Correct FFI definition
Py_tracefunc
to beunsafe extern "C" fn
(was previously safe). #2407 - Fix compile failure with
#[pyo3(from_py_with = "...")]
annotations on a field in a#[derive(FromPyObject)]
struct. #2414 - Fix FFI definitions
_PyDateTime_BaseTime
and_PyDateTime_BaseDateTime
lacking leading underscores in their names. #2421 - Remove FFI definition
PyArena
on Python 3.10 and up. #2421 - Fix FFI definition
PyCompilerFlags
missing membercf_feature_version
on Python 3.8 and up. #2423 - Fix FFI definition
PyAsyncMethods
missing memberam_send
on Python 3.10 and up. #2423 - Fix FFI definition
PyGenObject
having multiple incorrect members on various Python versions. #2423 - Fix FFI definition
PySyntaxErrorObject
missing membersend_lineno
andend_offset
on Python 3.10 and up. #2423 - Fix FFI definition
PyHeapTypeObject
missing memberht_module
on Python 3.9 and up. #2423 - Fix FFI definition
PyFrameObject
having multiple incorrect members on various Python versions. #2424 #2434 - Fix FFI definition
PyTypeObject
missing deprecated fieldtp_print
on Python 3.8. #2428 - Fix FFI definitions
PyDateTime_CAPI
.PyDateTime_Date
,PyASCIIObject
,PyBaseExceptionObject
,PyListObject
, andPyTypeObject
on PyPy. #2428 - Fix FFI definition
_inittab
fieldinitfunc
typo'd asinitfun
. #2431 - Fix FFI definitions
_PyDateTime_BaseTime
and_PyDateTime_BaseDateTime
incorrectly havingfold
member. #2432 - Fix FFI definitions
PyTypeObject
.PyHeapTypeObject
, andPyCFunctionObject
having incorrect members on PyPy 3.9. #2433 - Fix FFI definition
PyGetSetDef
to have*const c_char
fordoc
member (not*mut c_char
). #2439 - Fix
#[pyo3(from_py_with = "...")]
being ignored for 1-element tuple structs and transparent structs. #2440 - Use
memoffset
to avoid UB when computingPyCell
layout. #2450 - Fix incorrect enum names being returned by the generated
repr
for enums renamed by#[pyclass(name = "...")]
#2457 - Fix
PyObject_CallNoArgs
incorrectly being available when building for abi3 on Python 3.9. #2476 - Fix several clippy warnings generated by
#[pyfunction]
arguments. #2503
0.16.6 - 2022-08-23
Changed
- Fix soundness issues with
PyCapsule
type with select workarounds. Users are encourage to upgrade to PyO3 0.17 at their earliest convenience which contains API breakages which fix the issues in a long-term fashion. #2522PyCapsule::new
andPyCapsule::new_with_destructor
now take ownership of a copy of thename
to resolve a possible use-after-free.PyCapsule::name
now returns an emptyCStr
instead of dereferencing a null pointer if the capsule has no name.- The destructor
F
inPyCapsule::new_with_destructor
will never be called if the capsule is deleted from a thread other than the one which the capsule was created in (a warning will be emitted).
- Panics during drop of panic payload caught by PyO3 will now abort. #2544
0.16.5 - 2022-05-15
Added
- Add an experimental
generate-import-lib
feature to support auto-generating non-abi3 python import libraries for Windows targets. #2364 - Add FFI definition
Py_ExitStatusException
. #2374
Changed
- Deprecate experimental
generate-abi3-import-lib
feature in favor of the newgenerate-import-lib
feature. #2364
Fixed
- Added missing
warn_default_encoding
field toPyConfig
on 3.10+. The previously missing field could result in incorrect behavior or crashes. #2370 - Fixed order of
pathconfig_warnings
andprogram_name
fields ofPyConfig
on 3.10+. Previously, the order of the fields was swapped and this could lead to incorrect behavior or crashes. #2370
0.16.4 - 2022-04-14
Added
- Add
PyTzInfoAccess
trait for safe access to time zone information. #2263 - Add an experimental
generate-abi3-import-lib
feature to auto-generatepython3.dll
import libraries for Windows. #2282 - Add FFI definitions for
PyDateTime_BaseTime
andPyDateTime_BaseDateTime
. #2294
Changed
- Improved performance of failing calls to
FromPyObject::extract
which is common when functions accept multiple distinct types. #2279 - Default to "m" ABI tag when choosing
libpython
link name for CPython 3.7 on Unix. #2288 - Allow to compile "abi3" extensions without a working build host Python interpreter. #2293
Fixed
- Crates depending on PyO3 can collect code coverage via LLVM instrumentation using stable Rust. #2286
- Fix segfault when calling FFI methods
PyDateTime_DATE_GET_TZINFO
orPyDateTime_TIME_GET_TZINFO
ondatetime
ortime
without a tzinfo. #2289 - Fix directory names starting with the letter
n
breaking serialization of the interpreter configuration on Windows since PyO3 0.16.3. #2299
0.16.3 - 2022-04-05
Packaging
- Extend
parking_lot
dependency supported versions to include 0.12. #2239
Added
- Add methods to
pyo3_build_config::InterpreterConfig
to run Python scripts using the configured executable. #2092 - Add
as_bytes
method toPy<PyBytes>
. #2235 - Add FFI definitions for
PyType_FromModuleAndSpec
,PyType_GetModule
,PyType_GetModuleState
andPyModule_AddType
. #2250 - Add
pyo3_build_config::cross_compiling_from_to
as a helper to detect when PyO3 is cross-compiling. #2253 - Add
#[pyclass(mapping)]
option to leave sequence slots empty in container implementations. #2265 - Add
PyString::intern
to enable usage of the Python's built-in string interning. #2268 - Add
intern!
macro which can be used to amortize the cost of creating Python strings by storing them inside aGILOnceCell
. #2269 - Add
PYO3_CROSS_PYTHON_IMPLEMENTATION
environment variable for selecting the default cross Python implementation. #2272
Changed
- Allow
#[pyo3(crate = "...", text_signature = "...")]
options to be used directly in#[pyclass(crate = "...", text_signature = "...")]
. #2234 - Make
PYO3_CROSS_LIB_DIR
environment variable optional when cross compiling. #2241 - Mark
METH_FASTCALL
calling convention as limited API on Python 3.10. #2250 - Deprecate
pyo3_build_config::cross_compiling
in favor ofpyo3_build_config::cross_compiling_from_to
. #2253
Fixed
- Fix
abi3-py310
feature: use Python 3.10 ABI when available instead of silently falling back to the 3.9 ABI. #2242 - Use shared linking mode when cross compiling against a Framework bundle for macOS. #2233
- Fix panic during compilation when
PYO3_CROSS_LIB_DIR
is set for some host/target combinations. #2232 - Correct dependency version for
syn
to require minimal patch version 1.0.56. #2240
0.16.2 - 2022-03-15
Packaging
- Warn when modules are imported on PyPy 3.7 versions older than PyPy 7.3.8, as they are known to have binary compatibility issues. #2217
- Ensure build script of
pyo3-ffi
runs before that ofpyo3
to fix cross compilation. #2224
0.16.1 - 2022-03-05
Packaging
- Extend
hashbrown
optional dependency supported versions to include 0.12. #2197
Fixed
- Fix incorrect platform detection for Windows in
pyo3-build-config
. #2198 - Fix regression from 0.16 preventing cross compiling to aarch64 macOS. #2201
0.16.0 - 2022-02-27
Packaging
- Update MSRV to Rust 1.48. #2004
- Update
indoc
optional dependency to 1.0. #2004 - Drop support for Python 3.6, remove
abi3-py36
feature. #2006 pyo3-build-config
no longer enables theresolve-config
feature by default. #2008- Update
inventory
optional dependency to 0.2. #2019 - Drop
paste
dependency. #2081 - The bindings found in
pyo3::ffi
are now a re-export of a separatepyo3-ffi
crate. #2126 - Support PyPy 3.9. #2143
Added
- Add
PyCapsule
type exposing the Capsule API. #1980 - Add
pyo3_build_config::Sysconfigdata
and supporting APIs. #1996 - Add
Py::setattr
method. #2009 - Add
#[pyo3(crate = "some::path")]
option to all attribute macros (except the deprecated#[pyproto]
). #2022 - Enable
create_exception!
macro to take an optional docstring. #2027 - Enable
#[pyclass]
for fieldless (aka C-like) enums. #2034 - Add buffer magic methods
__getbuffer__
and__releasebuffer__
to#[pymethods]
. #2067 - Add support for paths in
wrap_pyfunction
andwrap_pymodule
. #2081 - Enable
wrap_pyfunction!
to wrap a#[pyfunction]
implemented in a different Rust module or crate. #2091 - Add
PyAny::contains
method (in
operator forPyAny
). #2115 - Add
PyMapping::contains
method (in
operator forPyMapping
). #2133 - Add garbage collection magic magic methods
__traverse__
and__clear__
to#[pymethods]
. #2159 - Add support for
from_py_with
on struct tuples and enums to override the default from-Python conversion. #2181 - Add
eq
,ne
,lt
,le
,gt
,ge
methods toPyAny
that wraprich_compare
. #2175 - Add
Py::is
andPyAny::is
methods to check for object identity. #2183 - Add support for the
__getattribute__
magic method. #2187
Changed
PyType::is_subclass
,PyErr::is_instance
andPyAny::is_instance
now operate run-time type object instead of a type known at compile-time. The old behavior is still available asPyType::is_subclass_of
,PyErr::is_instance_of
andPyAny::is_instance_of
. #1985- Rename some methods on
PyErr
(the old names are just marked deprecated for now): #2026pytype
->get_type
pvalue
->value
(and deprecate equivalentinstance
)ptraceback
->traceback
from_instance
->from_value
into_instance
->into_value
PyErr::new_type
now takes an optional docstring and now returnsPyResult<Py<PyType>>
rather than affi::PyTypeObject
pointer. #2027- Deprecate
PyType::is_instance
; it is inconsistent with otheris_instance
methods in PyO3. Instead oftyp.is_instance(obj)
, useobj.is_instance(typ)
. #2031 __getitem__
,__setitem__
and__delitem__
in#[pymethods]
now implement both a Python mapping and sequence by default. #2065- Improve performance and error messages for
#[derive(FromPyObject)]
for enums. #2068 - Reduce generated LLVM code size (to improve compile times) for:
- Respect Rust privacy rules for items wrapped with
wrap_pyfunction
andwrap_pymodule
. #2081 - Add modulo argument to
__ipow__
magic method. #2083 - Fix FFI definition for
_PyCFunctionFast
. #2126 PyDateTimeAPI
andPyDateTime_TimeZone_UTC
are now unsafe functions instead of statics. #2126PyDateTimeAPI
does not implicitly callPyDateTime_IMPORT
anymore to reflect the original Python API more closely. Before the first call toPyDateTime_IMPORT
a null pointer is returned. Therefore before calling any of the following FFI functionsPyDateTime_IMPORT
must be called to avoid undefined behavior: #2126PyDateTime_TimeZone_UTC
PyDate_Check
PyDate_CheckExact
PyDateTime_Check
PyDateTime_CheckExact
PyTime_Check
PyTime_CheckExact
PyDelta_Check
PyDelta_CheckExact
PyTZInfo_Check
PyTZInfo_CheckExact
PyDateTime_FromTimestamp
PyDate_FromTimestamp
- Deprecate the
gc
option forpyclass
(e.g.#[pyclass(gc)]
). Just implement a__traverse__
#[pymethod]
. #2159 - The
ml_meth
field ofPyMethodDef
is now represented by thePyMethodDefPointer
union. 2166 - Deprecate the
#[pyproto]
traits. #2173
Removed
- Remove all functionality deprecated in PyO3 0.14. #2007
- Remove
Default
impl forPyMethodDef
. #2166 - Remove
PartialEq
impl forPy
andPyAny
(use the newis
instead). #2183
Fixed
- Fix undefined symbol for
PyObject_HasAttr
on PyPy. #2025 - Fix memory leak in
PyErr::into_value
. #2026 - Fix clippy warning
needless-option-as-deref
in code generated by#[pyfunction]
and#[pymethods]
. #2040 - Fix undefined behavior in
PySlice::indices
. #2061 - Fix the
wrap_pymodule!
macro using the wrong name for a#[pymodule]
with a#[pyo3(name = "..")]
attribute. #2081 - Fix magic methods in
#[pymethods]
accepting implementations with the wrong number of arguments. #2083 - Fix panic in
#[pyfunction]
generated code when a required argument following anOption
was not provided. #2093 - Fixed undefined behavior caused by incorrect
ExactSizeIterator
implementations. #2124 - Fix missing FFI definition
PyCMethod_New
on Python 3.9 and up. #2143 - Add missing FFI definitions
_PyLong_NumBits
and_PyLong_AsByteArray
on PyPy. #2146 - Fix memory leak in implementation of
AsPyPointer
forOption<T>
. #2160 - Fix FFI definition of
_PyLong_NumBits
to returnsize_t
instead ofc_int
. #2161 - Fix
TypeError
thrown when argument parsing failed missing the originating causes. 2177
0.15.2 - 2022-04-14
Packaging
- Backport of PyPy 3.9 support from PyO3 0.16. #2262
0.15.1 - 2021-11-19
Added
- Add implementations for
Py::as_ref
andPy::into_ref
forPy<PySequence>
,Py<PyIterator>
andPy<PyMapping>
. #1682 - Add
PyTraceback
type to represent and format Python tracebacks. #1977
Changed
#[classattr]
constants with a known magic method name (which is lowercase) no longer trigger lint warnings expecting constants to be uppercase. #1969
Fixed
- Fix creating
#[classattr]
by functions with the name of a known magic method. #1969 - Fix use of
catch_unwind
inallow_threads
which can cause fatal crashes. #1989 - Fix build failure on PyPy when abi3 features are activated. #1991
- Fix mingw platform detection. #1993
- Fix panic in
__get__
implementation when accessing descriptor on type object. #1997
0.15.0 - 2021-11-03
Packaging
pyo3
'sCargo.toml
now advertiseslinks = "python"
to inform Cargo that it links against libpython. #1819- Added optional
anyhow
feature to convertanyhow::Error
intoPyErr
. #1822 - Support Python 3.10. #1889
- Added optional
eyre
feature to converteyre::Report
intoPyErr
. #1893 - Support PyPy 3.8. #1948
Added
- Add
PyList::get_item_unchecked
andPyTuple::get_item_unchecked
to get items without bounds checks. #1733 - Support
#[doc = include_str!(...)]
attributes on Rust 1.54 and up. #1746 - Add
PyAny::py
as a convenience forPyNativeType::py
. #1751 - Add implementation of
std::ops::Index<usize>
forPyList
,PyTuple
andPySequence
. #1825 - Add range indexing implementations of
std::ops::Index
forPyList
,PyTuple
andPySequence
. #1829 - Add
PyMapping
type to represent the Python mapping protocol. #1844 - Add commonly-used sequence methods to
PyList
andPyTuple
. #1849 - Add
as_sequence
methods toPyList
andPyTuple
. #1860 - Add support for magic methods in
#[pymethods]
, intended as a replacement for#[pyproto]
. #1864 - Add
abi3-py310
feature. #1889 - Add
PyCFunction::new_closure
to create a Python function from a Rust closure. #1901 - Add support for positional-only arguments in
#[pyfunction]
. #1925 - Add
PyErr::take
to attempt to fetch a Python exception if present. #1957
Changed
PyList
,PyTuple
andPySequence
's APIs now accepts onlyusize
indices instead ofisize
. #1733, #1802, #1803PyList::get_item
andPyTuple::get_item
now returnPyResult<&PyAny>
instead of panicking. #1733PySequence::in_place_repeat
andPySequence::in_place_concat
now returnPyResult<&PySequence>
instead ofPyResult<()>
, which is needed in case of immutable sequences such as tuples. #1803PySequence::get_slice
now returnsPyResult<&PySequence>
instead ofPyResult<&PyAny>
. #1829- Deprecate
PyTuple::split_from
. #1804 - Deprecate
PyTuple::slice
, new methodPyTuple::get_slice
added withusize
indices. #1828 - Deprecate FFI definitions
PyParser_SimpleParseStringFlags
,PyParser_SimpleParseStringFlagsFilename
,PyParser_SimpleParseFileFlags
when building for Python 3.9. #1830 - Mark FFI definitions removed in Python 3.10
PyParser_ASTFromString
,PyParser_ASTFromStringObject
,PyParser_ASTFromFile
,PyParser_ASTFromFileObject
,PyParser_SimpleParseStringFlags
,PyParser_SimpleParseStringFlagsFilename
,PyParser_SimpleParseFileFlags
,PyParser_SimpleParseString
,PyParser_SimpleParseFile
,Py_SymtableString
, andPy_SymtableStringObject
. #1830 #[pymethods]
now handles magic methods similarly to#[pyproto]
. In the future,#[pyproto]
may be deprecated. #1864- Deprecate FFI definitions
PySys_AddWarnOption
,PySys_AddWarnOptionUnicode
andPySys_HasWarnOptions
. #1887 - Deprecate
#[call]
attribute in favor of usingfn __call__
. #1929 - Fix missing FFI definition
_PyImport_FindExtensionObject
on Python 3.10. #1942 - Change
PyErr::fetch
to panic in debug mode if no exception is present. #1957
Fixed
- Fix building with a conda environment on Windows. #1873
- Fix panic on Python 3.6 when calling
Python::with_gil
with Python initialized but threading not initialized. #1874 - Fix incorrect linking to version-specific DLL instead of
python3.dll
when cross-compiling to Windows withabi3
. #1880 - Fix FFI definition for
PyTuple_ClearFreeList
incorrectly being present for Python 3.9 and up. #1887 - Fix panic in generated
#[derive(FromPyObject)]
for enums. #1888 - Fix cross-compiling to Python 3.7 builds with the "m" abi flag. #1908
- Fix
__mod__
magic method fallback to__rmod__
. #1934. - Fix missing FFI definition
_PyImport_FindExtensionObject
on Python 3.10. #1942
0.14.5 - 2021-09-05
Added
- Make
pyo3_build_config::InterpreterConfig
and subfields public. #1848 - Add
resolve-config
feature to thepyo3-build-config
to control whether its build script does anything. #1856
Fixed
- Fix 0.14.4 compile regression on
s390x-unknown-linux-gnu
target. #1850
0.14.4 - 2021-08-29
Changed
- Mark
PyString::data
asunsafe
and disable it and some supporting PyUnicode FFI APIs (which depend on a C bitfield) on big-endian targets. #1834
0.14.3 - 2021-08-22
Added
- Add
PyString::data
to access the raw bytes stored in a Python string. #1794
Fixed
- Raise
AttributeError
to avoid panic when callingdel
on a#[setter]
defined class property. #1779 - Restrict FFI definitions
PyGILState_Check
andPy_tracefunc
to the unlimited API. #1787 - Add missing
_type
field toPyStatus
struct definition. #1791 - Reduce lower bound
num-complex
optional dependency to support interop withrust-numpy
andndarray
when building with the MSRV of 1.41 #1799 - Fix memory leak in
Python::run_code
. #1806 - Fix memory leak in
PyModule::from_code
. #1810 - Remove use of
pyo3::
inpyo3::types::datetime
which broke builds using-Z avoid-dev-deps
#1811
0.14.2 - 2021-08-09
Added
- Add
indexmap
feature to addToPyObject
,IntoPy
andFromPyObject
implementations forindexmap::IndexMap
. #1728 - Add
pyo3_build_config::add_extension_module_link_args
to use in build scripts to set linker arguments (for macOS). #1755 - Add
Python::with_gil_unchecked
unsafe variation ofPython::with_gil
to allow obtaining aPython
in scenarios wherePython::with_gil
would fail. #1769
Changed
PyErr::new
no longer acquires the Python GIL internally. #1724- Reverted PyO3 0.14.0's use of
cargo:rustc-cdylib-link-arg
in its build script, as Cargo unintentionally allowed crates to pass linker args to downstream crates in this way. Projects supporting macOS may need to restore.cargo/config.toml
files. #1755
Fixed
- Fix regression in 0.14.0 rejecting usage of
#[doc(hidden)]
on structs and functions annotated with PyO3 macros. #1722 - Fix regression in 0.14.0 leading to incorrect code coverage being computed for
#[pyfunction]
s. #1726 - Fix incorrect FFI definition of
Py_Buffer
on PyPy. #1737 - Fix incorrect calculation of
dictoffset
on 32-bit Windows. #1475 - Fix regression in 0.13.2 leading to linking to incorrect Python library on Windows "gnu" targets. #1759
- Fix compiler warning: deny trailing semicolons in expression macro. #1762
- Fix incorrect FFI definition of
Py_DecodeLocale
. The 2nd argument is now*mut Py_ssize_t
instead ofPy_ssize_t
. #1766
0.14.1 - 2021-07-04
Added
- Implement
IntoPy<PyObject>
for&PathBuf
and&OsString
. #1712
Fixed
- Fix crashes on PyPy due to incorrect definitions of
PyList_SET_ITEM
. #1713
0.14.0 - 2021-07-03
Packaging
- Update
num-bigint
optional dependency to 0.4. #1481 - Update
num-complex
optional dependency to 0.4. #1482 - Extend
hashbrown
optional dependency supported versions to include 0.11. #1496 - Support PyPy 3.7. #1538
Added
- Extend conversions for
[T; N]
to allN
using const generics (on Rust 1.51 and up). #1128 - Add conversions between
OsStr
/OsString
and Python strings. #1379 - Add conversions between
Path
/PathBuf
and Python strings (andpathlib.Path
objects). #1379 #1654 - Add a new set of
#[pyo3(...)]
attributes to control various PyO3 macro functionality: - Add FFI definition
PyCFunction_CheckExact
for Python 3.9 and later. #1425 - Add FFI definition
Py_IS_TYPE
. #1429 - Add FFI definition
_Py_InitializeMain
. #1473 - Add FFI definitions from
cpython/import.h
.#1475 - Add tuple and unit struct support for
#[pyclass]
macro. #1504 - Add FFI definition
PyDateTime_TimeZone_UTC
. #1572 - Add support for
#[pyclass(extends=Exception)]
. #1591 - Add
PyErr::cause
andPyErr::set_cause
. #1679 - Add FFI definitions from
cpython/pystate.h
. #1687 - Add
wrap_pyfunction!
macro topyo3::prelude
. #1695
Changed
- Allow only one
#[pymethods]
block per#[pyclass]
by default, to remove the dependency oninventory
. Add amultiple-pymethods
feature to opt-in the original behavior and dependency oninventory
. #1457 - Change
PyTimeAccess::get_fold
to return abool
instead of au8
. #1397 - Deprecate FFI definition
PyCFunction_Call
for Python 3.9 and up. #1425 - Deprecate FFI definition
PyModule_GetFilename
. #1425 - The
auto-initialize
feature is no longer enabled by default. #1443 - Change
PyCFunction::new
andPyCFunction::new_with_keywords
to take&'static str
arguments rather than implicitly copying (and leaking) them. #1450 - Deprecate
PyModule::call
,PyModule::call0
,PyModule::call1
andPyModule::get
. #1492 - Add length information to
PyBufferError
s raised fromPyBuffer::copy_to_slice
andPyBuffer::copy_from_slice
. #1534 - Automatically set
-undefined
anddynamic_lookup
linker arguments on macOS with theextension-module
feature. #1539 - Deprecate
#[pyproto]
methods which are easier to implement as#[pymethods]
: #1560PyBasicProtocol::__bytes__
andPyBasicProtocol::__format__
PyContextProtocol::__enter__
andPyContextProtocol::__exit__
PyDescrProtocol::__delete__
andPyDescrProtocol::__set_name__
PyMappingProtocol::__reversed__
PyNumberProtocol::__complex__
andPyNumberProtocol::__round__
PyAsyncProtocol::__aenter__
andPyAsyncProtocol::__aexit__
- Deprecate several attributes in favor of the new
#[pyo3(...)]
options: - Reduce LLVM line counts to improve compilation times. #1604
- No longer call
PyEval_InitThreads
in#[pymodule]
init code. #1630 - Use
METH_FASTCALL
argument passing convention, when possible, to improve#[pyfunction]
and method performance. #1619, #1660 - Filter sysconfigdata candidates by architecture when cross-compiling. #1626
Removed
- Remove deprecated exception names
BaseException
etc. #1426 - Remove deprecated methods
Python::is_instance
,Python::is_subclass
,Python::release
,Python::xdecref
, andPy::from_owned_ptr_or_panic
. #1426 - Remove many FFI definitions which never existed in the Python C-API:
- Remove pyclass implementation details from
PyTypeInfo
: - Remove
PYO3_CROSS_INCLUDE_DIR
environment variable and the associated C header parsing functionality. #1521 - Remove
raw_pycfunction!
macro. #1619 - Remove
PyClassAlloc
trait. #1657 - Remove
PyList::get_parked_item
. #1664
Fixed
- Remove FFI definition
PyCFunction_ClearFreeList
for Python 3.9 and later. #1425 PYO3_CROSS_LIB_DIR
environment variable no long required when compiling for x86-64 Python from macOS arm64 and reverse. #1428- Fix FFI definition
_PyEval_RequestCodeExtraIndex
, which took an argument of the wrong type. #1429 - Fix FFI definition
PyIndex_Check
missing with theabi3
feature. #1436 - Fix incorrect
TypeError
raised when keyword-only argument passed along with a positional argument in*args
. #1440 - Fix inability to use a named lifetime for
&PyTuple
of*args
in#[pyfunction]
. #1440 - Fix use of Python argument for
#[pymethods]
inside macro expansions. #1505 - No longer include
__doc__
in__all__
generated for#[pymodule]
. #1509 - Always use cross-compiling configuration if any of the
PYO3_CROSS
family of environment variables are set. #1514 - Support
EnvironmentError
,IOError
, andWindowsError
on PyPy. #1533 - Fix unnecessary rebuilds when cycling between
cargo check
andcargo clippy
in a Python virtualenv. #1557 - Fix segfault when dereferencing
ffi::PyDateTimeAPI
without the GIL. #1563 - Fix memory leak in
FromPyObject
implementations foru128
andi128
. #1638 - Fix
#[pyclass(extends=PyDict)]
leaking the dict contents on drop. #1657 - Fix segfault when calling
PyList::get_item
with negative indices. #1668 - Fix FFI definitions of
PyEval_SetProfile
/PyEval_SetTrace
to takeOption<Py_tracefunc>
parameters. #1692 - Fix
ToPyObject
impl forHashSet
to accept non-default hashers. #1702
0.13.2 - 2021-02-12
Packaging
- Lower minimum supported Rust version to 1.41. #1421
Added
- Add unsafe API
with_embedded_python_interpreter
to initialize a Python interpreter, execute a closure, and finalize the interpreter. #1355 - Add
serde
feature which provides implementations ofSerialize
andDeserialize
forPy<T>
. #1366 - Add FFI definition
_PyCFunctionFastWithKeywords
on Python 3.7 and up. #1384 - Add
PyDateTime::new_with_fold
method. #1398 - Add
size_hint
impls for{PyDict,PyList,PySet,PyTuple}Iterator
s. #1699
Changed
prepare_freethreaded_python
will no longer register anatexit
handler to callPy_Finalize
. This resolves a number of issues with incompatible C extensions causing crashes at finalization. #1355- Mark
PyLayout::py_init
,PyClassDict::clear_dict
, andopt_to_pyobj
safe, as they do not perform any unsafe operations. #1404
Fixed
- Fix support for using
r#raw_idents
as argument names in pyfunctions. #1383 - Fix typo in FFI definition for
PyFunction_GetCode
(was incorrectlyPyFunction_Code
). #1387 - Fix FFI definitions
PyMarshal_WriteObjectToString
andPyMarshal_ReadObjectFromString
as available in limited API. #1387 - Fix FFI definitions
PyListObject
and those fromfuncobject.h
as requiring non-limited API. #1387 - Fix unqualified
Result
usage inpyobject_native_type_base
. #1402 - Fix build on systems where the default Python encoding is not UTF-8. #1405
- Fix build on mingw / MSYS2. #1423
0.13.1 - 2021-01-10
Added
- Add support for
#[pyclass(dict)]
and#[pyclass(weakref)]
with theabi3
feature on Python 3.9 and up. #1342 - Add FFI definitions
PyOS_BeforeFork
,PyOS_AfterFork_Parent
,PyOS_AfterFork_Child
for Python 3.7 and up. #1348 - Add an
auto-initialize
feature to control whether PyO3 should automatically initialize an embedded Python interpreter. For compatibility this feature is enabled by default in PyO3 0.13.1, but is planned to become opt-in from PyO3 0.14.0. #1347 - Add support for cross-compiling to Windows without needing
PYO3_CROSS_INCLUDE_DIR
. #1350
Deprecated
- Deprecate FFI definitions
PyEval_CallObjectWithKeywords
,PyEval_CallObject
,PyEval_CallFunction
,PyEval_CallMethod
when building for Python 3.9. #1338 - Deprecate FFI definitions
PyGetSetDef_DICT
andPyGetSetDef_INIT
which have never been in the Python API. #1341 - Deprecate FFI definitions
PyGen_NeedsFinalizing
,PyImport_Cleanup
(removed in 3.9), andPyOS_InitInterrupts
(3.10). #1348 - Deprecate FFI definition
PyOS_AfterFork
for Python 3.7 and up. #1348 - Deprecate FFI definitions
PyCoro_Check
,PyAsyncGen_Check
, andPyCoroWrapper_Check
, which have never been in the Python API (for the first two, it is possible to usePyCoro_CheckExact
andPyAsyncGen_CheckExact
instead; these are the actual functions provided by the Python API). #1348 - Deprecate FFI definitions for
PyUnicode_FromUnicode
,PyUnicode_AsUnicode
andPyUnicode_AsUnicodeAndSize
, which will be removed from 3.12 and up due to PEP 623. #1370
Removed
- Remove FFI definition
PyFrame_ClearFreeList
when building for Python 3.9. #1341 - Remove FFI definition
_PyDict_Contains
when building for Python 3.10. #1341 - Remove FFI definitions
PyGen_NeedsFinalizing
andPyImport_Cleanup
(for 3.9 and up), andPyOS_InitInterrupts
(3.10). #1348
Fixed
- Stop including
Py_TRACE_REFS
config setting automatically ifPy_DEBUG
is set on Python 3.8 and up. #1334 - Remove
#[deny(warnings)]
attribute (and instead refuse warnings only in CI). #1340 - Fix deprecation warning for missing
__module__
with#[pyclass]
. #1343 - Correct return type of
PyFrozenSet::empty
to&PyFrozenSet
(was incorrectly&PySet
). #1351 - Fix missing
Py_INCREF
on heap type objects on Python versions before 3.8. #1365
0.13.0 - 2020-12-22
Packaging
- Drop support for Python 3.5 (as it is now end-of-life). #1250
- Bump minimum supported Rust version to 1.45. #1272
- Bump indoc dependency to 1.0. #1272
- Bump paste dependency to 1.0. #1272
- Rename internal crates
pyo3cls
andpyo3-derive-backend
topyo3-macros
andpyo3-macros-backend
respectively. #1317
Added
- Add support for building for CPython limited API. Opting-in to the limited API enables a single extension wheel built with PyO3 to be installable on multiple Python versions. This required a few minor changes to runtime behavior of of PyO3
#[pyclass]
types. See the migration guide for full details. #1152- Add feature flags
abi3-py36
,abi3-py37
,abi3-py38
etc. to set the minimum Python version when using the limited API. #1263
- Add feature flags
- Add argument names to
TypeError
messages generated by pymethod wrappers. #1212 - Add FFI definitions for PEP 587 "Python Initialization Configuration". #1247
- Add FFI definitions for
PyEval_SetProfile
andPyEval_SetTrace
. #1255 - Add FFI definitions for context.h functions (
PyContext_New
, etc). #1259 - Add
PyAny::is_instance
method. #1276 - Add support for conversion between
char
andPyString
. #1282 - Add FFI definitions for
PyBuffer_SizeFromFormat
,PyObject_LengthHint
,PyObject_CallNoArgs
,PyObject_CallOneArg
,PyObject_CallMethodNoArgs
,PyObject_CallMethodOneArg
,PyObject_VectorcallDict
, andPyObject_VectorcallMethod
. #1287 - Add conversions between
u128
/i128
andPyLong
for PyPy. #1310 - Add
Python::version
andPython::version_info
to get the running interpreter version. #1322 - Add conversions for tuples of length 10, 11, and 12. #1454
Changed
- Change return type of
PyType::name
fromCow<str>
toPyResult<&str>
. #1152 #[pyclass(subclass)]
is now required for subclassing from Rust (was previously just required for subclassing from Python). #1152- Change
PyIterator
to be consistent with other native types: it is now used as&PyIterator
instead ofPyIterator<'a>
. #1176 - Change formatting of
PyDowncastError
messages to be closer to Python's builtin error messages. #1212 - Change
Debug
andDisplay
impls forPyException
to be consistent withPyAny
. #1275 - Change
Debug
impl ofPyErr
to output more helpful information (acquiring the GIL if necessary). #1275 - Rename
PyTypeInfo::is_instance
andPyTypeInfo::is_exact_instance
toPyTypeInfo::is_type_of
andPyTypeInfo::is_exact_type_of
. #1278 - Optimize
PyAny::call0
,Py::call0
andPyAny::call_method0
andPy::call_method0
on Python 3.9 and up. #1287 - Require double-quotes for pyclass name argument e.g
#[pyclass(name = "MyClass")]
. #1303
Deprecated
- Deprecate
Python::is_instance
,Python::is_subclass
,Python::release
, andPython::xdecref
. #1292
Removed
- Remove deprecated ffi definitions
PyUnicode_AsUnicodeCopy
,PyUnicode_GetMax
,_Py_CheckRecursionLimit
,PyObject_AsCharBuffer
,PyObject_AsReadBuffer
,PyObject_CheckReadBuffer
andPyObject_AsWriteBuffer
, which will be removed in Python 3.10. #1217 - Remove unused
python3
feature. #1235
Fixed
- Fix missing field in
PyCodeObject
struct (co_posonlyargcount
) - caused invalid access to other fields in Python >3.7. #1260 - Fix building for
x86_64-unknown-linux-musl
target fromx86_64-unknown-linux-gnu
host. #1267 - Fix
#[text_signature]
interacting badly with rustr#raw_identifiers
. #1286 - Fix FFI definitions for
PyObject_Vectorcall
andPyVectorcall_Call
. #1287 - Fix building with Anaconda python inside a virtualenv. #1290
- Fix definition of opaque FFI types. #1312
- Fix using custom error type in pyclass
#[new]
methods. #1319
0.12.4 - 2020-11-28
Fixed
- Fix reference count bug in implementation of
From<Py<T>>
forPyObject
, a regression introduced in PyO3 0.12. #1297
0.12.3 - 2020-10-12
Fixed
- Fix support for Rust versions 1.39 to 1.44, broken by an incorrect internal update to paste 1.0 which was done in PyO3 0.12.2. #1234
0.12.2 - 2020-10-12
Added
- Add support for keyword-only arguments without default values in
#[pyfunction]
. #1209 - Add
Python::check_signals
as a safe a wrapper forPyErr_CheckSignals
. #1214
Fixed
- Fix invalid document for protocol methods. #1169
- Hide docs of PyO3 private implementation details in
pyo3::class::methods
. #1169 - Fix unnecessary rebuild on PATH changes when the python interpreter is provided by PYO3_PYTHON. #1231
0.12.1 - 2020-09-16
Fixed
- Fix building for a 32-bit Python on 64-bit Windows with a 64-bit Rust toolchain. #1179
- Fix building on platforms where
c_char
isu8
. #1182
0.12.0 - 2020-09-12
Added
- Add FFI definitions
Py_FinalizeEx
,PyOS_getsig
, andPyOS_setsig
. #1021 - Add
PyString::to_str
for accessingPyString
as&str
. #1023 - Add
Python::with_gil
for executing a closure with the Python GIL. #1037 - Add type information to failures in
PyAny::downcast
. #1050 - Implement
Debug
forPyIterator
. #1051 - Add
PyBytes::new_with
andPyByteArray::new_with
for initialisingbytes
andbytearray
objects using a closure. #1074 - Add
#[derive(FromPyObject)]
macro for enums and structs. #1065 - Add
Py::as_ref
andPy::into_ref
for convertingPy<T>
to&T
. #1098 - Add ability to return
Result
types other thanPyResult
from#[pyfunction]
,#[pymethod]
and#[pyproto]
functions. #1106. - Implement
ToPyObject
,IntoPy
, andFromPyObject
for hashbrown'sHashMap
andHashSet
types (requires thehashbrown
feature). #1114 - Add
#[pyfunction(pass_module)]
and#[pyfn(pass_module)]
to pass the module object as the first function argument. #1143 - Add
PyModule::add_function
andPyModule::add_submodule
as typed alternatives toPyModule::add_wrapped
. #1143 - Add native
PyCFunction
andPyFunction
types. #1163
Changed
- Rework exception types: #1024 #1115
- Rename exception types from e.g.
RuntimeError
toPyRuntimeError
. The old names continue to exist but are deprecated. - Exception objects are now accessible as
&T
orPy<T>
, just like other Python-native types. - Rename
PyException::py_err
toPyException::new_err
. - Rename
PyUnicodeDecodeErr::new_err
toPyUnicodeDecodeErr::new
. - Remove
PyStopIteration::stop_iteration
.
- Rename exception types from e.g.
- Require
T: Send
for the return valueT
ofPython::allow_threads
. #1036 - Rename
PYTHON_SYS_EXECUTABLE
toPYO3_PYTHON
. The old name will continue to work (undocumented) but will be removed in a future release. #1039 - Remove
unsafe
from signature ofPyType::as_type_ptr
. #1047 - Change return type of
PyIterator::from_object
toPyResult<PyIterator>
(wasResult<PyIterator, PyDowncastError>
). #1051 IntoPy
is no longer implied byFromPy
. #1063- Change
PyObject
to be a type alias forPy<PyAny>
. #1063 - Rework
PyErr
to be compatible with thestd::error::Error
trait: #1067 #1115- Implement
Display
,Error
,Send
andSync
forPyErr
andPyErrArguments
. - Add
PyErr::instance
for accessingPyErr
as&PyBaseException
. PyErr
's fields are now an implementation detail. The equivalent values can be accessed withPyErr::ptype
,PyErr::pvalue
andPyErr::ptraceback
.- Change receiver of
PyErr::print
andPyErr::print_and_set_sys_last_vars
to&self
(wasself
). - Remove
PyErrValue
,PyErr::from_value
,PyErr::into_normalized
, andPyErr::normalize
. - Remove
PyException::into
. - Remove
Into<PyResult<T>>
forPyErr
andPyException
.
- Implement
- Change methods generated by
#[pyproto]
to returnNotImplemented
if Python should try a reversed operation. #1072 - Change argument to
PyModule::add
toimpl IntoPy<PyObject>
(wasimpl ToPyObject
). #1124
Removed
- Remove many exception and
PyErr
APIs; see the "changed" section above. #1024 #1067 #1115 - Remove
PyString::to_string
(use newPyString::to_str
). #1023 - Remove
PyString::as_bytes
. #1023 - Remove
Python::register_any
. #1023 - Remove
GILGuard::acquire
from the public API. UsePython::acquire_gil
orPython::with_gil
. #1036 - Remove the
FromPy
trait. #1063 - Remove the
AsPyRef
trait. #1098
Fixed
- Correct FFI definitions
Py_SetProgramName
andPy_SetPythonHome
to take*const
arguments (was*mut
). #1021 - Fix
FromPyObject
fornum_bigint::BigInt
for Python objects with an__index__
method. #1027 - Correct FFI definition
_PyLong_AsByteArray
to take*mut c_uchar
argument (was*const c_uchar
). #1029 - Fix segfault with
#[pyclass(dict, unsendable)]
. #1058 #1059 - Fix using
&Self
as an argument type for functions in a#[pymethods]
block. #1071 - Fix best-effort build against PyPy 3.6. #1092
- Fix many cases of lifetime elision in
#[pyproto]
implementations. #1093 - Fix detection of Python build configuration when cross-compiling. #1095
- Always link against libpython on android with the
extension-module
feature. #1095 - Fix the
+
operator not trying__radd__
when both__add__
and__radd__
are defined inPyNumberProtocol
(and similar for all other reversible operators). #1107 - Fix building with Anaconda python. #1175
0.11.1 - 2020-06-30
Added
#[pyclass(unsendable)]
. #1009
Changed
- Update
parking_lot
dependency to0.11
. #1010
0.11.0 - 2020-06-28
Added
- Support stable versions of Rust (>=1.39). #969
- Add FFI definition
PyObject_AsFileDescriptor
. #938 - Add
PyByteArray::data
,PyByteArray::as_bytes
, andPyByteArray::as_bytes_mut
. #967 - Add
GILOnceCell
to use in situations wherelazy_static
oronce_cell
can deadlock. #975 - Add
Py::borrow
,Py::borrow_mut
,Py::try_borrow
, andPy::try_borrow_mut
for accessing#[pyclass]
values. #976 - Add
IterNextOutput
andIterANextOutput
for returning from__next__
/__anext__
. #997
Changed
- Simplify internals of
#[pyo3(get)]
attribute. (Remove the hidden APIGetPropertyValue
.) #934 - Call
Py_Finalize
at exit to flush buffers, etc. #943 - Add type parameter to PyBuffer. #951
- Require
Send
bound for#[pyclass]
. #966 - Add
Python
argument to most methods onPyObject
andPy<T>
to ensure GIL safety. #970 - Change signature of
PyTypeObject::type_object
- now takesPython
argument and returns&PyType
. #970 - Change return type of
PyTuple::slice
andPyTuple::split_from
fromPy<PyTuple>
to&PyTuple
. #970 - Change return type of
PyTuple::as_slice
to&[&PyAny]
. #971 - Rename
PyTypeInfo::type_object
totype_object_raw
, and addPython
argument. #975 - Update
num-complex
optional dependendency from0.2
to0.3
. #977 - Update
num-bigint
optional dependendency from0.2
to0.3
. #978 #[pyproto]
is re-implemented without specialization. #961PyClassAlloc::alloc
is renamed toPyClassAlloc::new
. #990#[pyproto]
methods can now have return valueT
orPyResult<T>
(previously onlyPyResult<T>
was supported). #996#[pyproto]
methods can now skip annotating the return type if it is()
. #998
Removed
- Remove
ManagedPyRef
(unused, and needs specialization) #930
Fixed
- Fix passing explicit
None
toOption<T>
argument#[pyfunction]
with a default value. #936 - Fix
PyClass.__new__
's not respecting subclasses when inherited by a Python class. #990 - Fix returning
Option<T>
from#[pyproto]
methods. #996 - Fix accepting
PyRef<Self>
andPyRefMut<Self>
to#[getter]
and#[setter]
methods. #999
0.10.1 - 2020-05-14
Fixed
- Fix deadlock in
Python::acquire_gil
after dropping aPyObject
orPy<T>
. #924
0.10.0 - 2020-05-13
Added
- Add FFI definition
_PyDict_NewPresized
. #849 - Implement
IntoPy<PyObject>
forHashSet
andBTreeSet
. #864 - Add
PyAny::dir
method. #886 - Gate macros behind a
macros
feature (enabled by default). #897 - Add ability to define class attributes using
#[classattr]
on functions in#[pymethods]
. #905 - Implement
Clone
forPyObject
andPy<T>
. #908 - Implement
Deref<Target = PyAny>
for all builtin types. (PyList
,PyTuple
,PyDict
etc.) #911 - Implement
Deref<Target = PyAny>
forPyCell<T>
. #911 - Add
#[classattr]
support for associated constants in#[pymethods]
. #914
Changed
- Panics will now be raised as a Python
PanicException
. #797 - Change
PyObject
andPy<T>
reference counts to decrement immediately upon drop when the GIL is held. #851 - Allow
PyIterProtocol
methods to use eitherPyRef
orPyRefMut
as the receiver type. #856 - Change the implementation of
FromPyObject
forPy<T>
to apply to a wider range ofT
, including allT: PyClass
. #880 - Move all methods from the
ObjectProtocol
trait to thePyAny
struct. #911 - Remove need for
#![feature(specialization)]
in crates depending on PyO3. #917
Removed
- Remove
PyMethodsProtocol
trait. #889 - Remove
num-traits
dependency. #895 - Remove
ObjectProtocol
trait. #911 - Remove
PyAny::None
. Users should usePython::None
instead. #911 - Remove all
*ProtocolImpl
traits. #917
Fixed
- Fix support for
__radd__
and other__r*__
methods as implementations for Python mathematical operators. #839 - Fix panics during garbage collection when traversing objects that were already mutably borrowed. #855
- Prevent
&'static
references to Python objects as arguments to#[pyfunction]
and#[pymethods]
. #869 - Fix lifetime safety bug with
AsPyRef::as_ref
. #876 - Fix
#[pyo3(get)]
attribute onPy<T>
fields. #880 - Fix segmentation faults caused by functions such as
PyList::get_item
returning borrowed objects when it was not safe to do so. #890 - Fix segmentation faults caused by nested
Python::acquire_gil
calls creating dangling references. #893 - Fix segmentatation faults when a panic occurs during a call to
Python::allow_threads
. #912
0.9.2 - 2020-04-09
Added
FromPyObject
implementations forHashSet
andBTreeSet
. #842
Fixed
- Correctly detect 32bit architecture. #830
0.9.1 - 2020-03-23
Fixed
0.9.0 - 2020-03-19
Added
PyCell
, which has RefCell-like features. #770PyClass
,PyLayout
,PyClassInitializer
. #683- Implemented
IntoIterator
forPySet
andPyFrozenSet
. #716 FromPyObject
is now automatically implemented forT: Clone
pyclasses. #730#[pyo3(get)]
and#[pyo3(set)]
will now use the Rust doc-comment from the field for the Python property. #755#[setter]
functions may now take an argument ofPyo3::Python
. #760PyTypeInfo::BaseLayout
andPyClass::BaseNativeType
. #770PyDowncastImpl
. #770- Implement
FromPyObject
andIntoPy<PyObject>
traits for arrays (up to 32). #778 migration.md
andtypes.md
in the guide. #795, #802ffi::{_PyBytes_Resize, _PyDict_Next, _PyDict_Contains, _PyDict_GetDictPtr}
. #820
Changed
#[new]
does not takePyRawObject
and can returnSelf
. #683- The blanket implementations for
FromPyObject
for&T
and&mut T
are no longer specializable. ImplementPyTryFrom
for your type to control the behavior ofFromPyObject::extract
for your types. #713 - The implementation for
IntoPy<U> for T
whereU: FromPy<T>
is no longer specializable. Control the behavior of this via the implementation ofFromPy
. #713 - Use
parking_lot::Mutex
instead ofspin::Mutex
. #734 - Bumped minimum Rust version to
1.42.0-nightly 2020-01-21
. #761 PyRef
andPyRefMut
are renewed forPyCell
. #770- Some new FFI functions for Python 3.8. #784
PyAny
is now on the top level module and prelude. #816
Removed
PyRawObject
. #683PyNoArgsFunction
. #741initialize_type
. To set the module name for a#[pyclass]
, use themodule
argument to the macro. #751AsPyRef::as_mut/with/with_mut/into_py/into_mut_py
. #770PyTryFrom::try_from_mut/try_from_mut_exact/try_from_mut_unchecked
. #770Python::mut_from_owned_ptr/mut_from_borrowed_ptr
. #770ObjectProtocol::get_base/get_mut_base
. #770
Fixed
- Fixed unsoundness of subclassing. #683.
- Clear error indicator when the exception is handled on the Rust side. #719
- Usage of raw identifiers with
#[pyo3(set)]
. #745 - Usage of
PyObject
with#[pyo3(get)]
. #760 #[pymethods]
used in conjunction with#[cfg]
. #769"*"
in a#[pyfunction()]
argument list incorrectly accepting any number of positional arguments (useargs = "*"
when this behavior is desired). #792PyModule::dict
. #809- Fix the case where
DESCRIPTION
is not null-terminated. #822
0.8.5 - 2020-01-05
Added
- Implemented
FromPyObject
forHashMap
andBTreeMap
- Support for
#[name = "foo"]
attribute for#[pyfunction]
and in#[pymethods]
. #692
0.8.4 - 2019-12-14
Added
- Support for
#[text_signature]
attribute. #675
0.8.3 - 2019-11-23
Removed
#[init]
is removed. #658
Fixed
- Now all
&Py~
types have!Send
bound. #655 - Fix a compile error raised by the stabilization of
!
type. #672.
0.8.2 - 2019-10-27
Added
- FFI compatibility for PEP 590 Vectorcall. #641
Fixed
- Fix PySequenceProtocol::set_item. #624
- Fix a corner case of BigInt::FromPyObject. #630
- Fix index errors in parameter conversion. #631
- Fix handling of invalid utf-8 sequences in
PyString::as_bytes
. #639 andPyString::to_string_lossy
#642. - Remove
__contains__
and__iter__
from PyMappingProtocol. #644 - Fix proc-macro definition of PySetAttrProtocol. #645
0.8.1 - 2019-10-08
Added
- Conversion between num-bigint and Python int. #608
Fixed
- Make sure the right Python interpreter is used in OSX builds. #604
- Patch specialization being broken by Rust 1.40. #614
- Fix a segfault around PyErr. #597
0.8.0 - 2019-09-16
Added
module
argument topyclass
macro. #499py_run!
macro #512- Use existing fields and methods before calling custom getattr. #505
PyBytes
can now be indexed just likeVec<u8>
- Implement
IntoPy<PyObject>
forPyRef
andPyRefMut
.
Changed
- Implementing the Using the
gc
parameter forpyclass
(e.g.#[pyclass(gc)]
) without implementing theclass::PyGCProtocol
trait is now a compile-time error. Failing to implement this trait could lead to segfaults. #532 PyByteArray::data
has been replaced withPyDataArray::to_vec
because returning a&[u8]
is unsound. (See this comment for a great write-up for why that was unsound)- Replace
mashup
withpaste
. GILPool
gained aPython
marker to prevent it from being misused to release Python objects without the GIL held.
Removed
IntoPyObject
was replaced withIntoPy<PyObject>
#[pyclass(subclass)]
is hidden aunsound-subclass
feature because it's causing segmentation faults.
Fixed
- More readable error message for generics in pyclass #503
0.7.0 - 2019-05-26
Added
- PyPy support by omerbenamram in #393
- Have
PyModule
generate an index of its members (__all__
list). - Allow
slf: PyRef<T>
for pyclass(#419) - Allow to use lifetime specifiers in
pymethods
- Add
marshal
module. #460
Changed
Python::run
returnsPyResult<()>
instead ofPyResult<&PyAny>
.- Methods decorated with
#[getter]
and#[setter]
can now omit wrapping the result type inPyResult
if they don't raise exceptions.
Fixed
type_object::PyTypeObject
has been marked unsafe because breaking the contracttype_object::PyTypeObject::init_type
can lead to UB.- Fixed automatic derive of
PySequenceProtocol
implementation in #423. - Capitalization & better wording to README.md.
- Docstrings of properties is now properly set using the doc of the
#[getter]
method. - Fixed issues with
pymethods
crashing on doc comments containing double quotes. PySet::new
andPyFrozenSet::new
now returnPyResult<&Py[Frozen]Set>
; exceptions are raised if the items are not hashable.- Fixed building using
venv
on Windows. PyTuple::new
now returns&PyTuple
instead ofPy<PyTuple>
.- Fixed several issues with argument parsing; notable, the
*args
and**kwargs
tuple/dict now doesn't contain arguments that are otherwise assigned to parameters.
0.6.0 - 2019-03-28
Regressions
- Currently, #341 causes
cargo test
to fail with weird linking errors when theextension-module
feature is activated. For now you can work around this by making theextension-module
feature optional and running the tests withcargo test --no-default-features
:
[dependencies.pyo3]
version = "0.6.0"
[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]
Added
- Added a
wrap_pymodule!
macro similar to the existingwrap_pyfunction!
macro. Only available on python 3 - Added support for cross compiling (e.g. to arm v7) by mtp401 in #327. See the "Cross Compiling" section in the "Building and Distribution" chapter of the guide for more details.
- The
PyRef
andPyRefMut
types, which allow to differentiate between an instance of a rust struct on the rust heap and an instance that is embedded inside a python object. By kngwyu in #335 - Added
FromPy<T>
andIntoPy<T>
which are equivalent toFrom<T>
andInto<T>
except that they require a gil token. - Added
ManagedPyRef
, which should eventually replaceToBorrowedObject
.
Changed
- Renamed
PyObjectRef
toPyAny
in #388 - Renamed
add_function
toadd_wrapped
as it now also supports modules. - Renamed
#[pymodinit]
to#[pymodule]
py.init(|| value)
becomesPy::new(value)
py.init_ref(|| value)
becomesPyRef::new(value)
py.init_mut(|| value)
becomesPyRefMut::new(value)
.PyRawObject::init
is now infallible, e.g. it returns()
instead ofPyResult<()>
.- Renamed
py_exception!
tocreate_exception!
and refactored the error macros. - Renamed
wrap_function!
towrap_pyfunction!
- Renamed
#[prop(get, set)]
to#[pyo3(get, set)]
#[pyfunction]
now supports the same arguments as#[pyfn()]
- Some macros now emit proper spanned errors instead of panics.
- Migrated to the 2018 edition
crate::types::exceptions
moved tocrate::exceptions
- Replace
IntoPyTuple
withIntoPy<Py<PyTuple>>
. IntoPyPointer
andToPyPointer
moved into the crate root.class::CompareOp
moved intoclass::basic::CompareOp
- PyTypeObject is now a direct subtrait PyTypeCreate, removing the old cyclical implementation in #350
- Add
PyList::{sort, reverse}
by chr1sj0nes in #357 and #358 - Renamed the
typeob
module totype_object
Removed
PyToken
was removed due to unsoundness (See #94).- Removed the unnecessary type parameter from
PyObjectAlloc
NoArgs
. Just use an empty tuplePyObjectWithGIL
.PyNativeType
is sufficient now that PyToken is removed.
Fixed
- A soudness hole where every instances of a
#[pyclass]
struct was considered to be part of a python object, even though you can create instances that are not part of the python heap. This was fixed throughPyRef
andPyRefMut
. - Fix kwargs support in #328.
- Add full support for
__dict__
in #403.
0.5.3 - 2019-01-04
Fixed
- Fix memory leak in ArrayList by kngwyu #316
0.5.2 - 2018-11-25
Fixed
- Fix undeterministic segfaults when creating many objects by kngwyu in #281
0.5.1 - 2018-11-24
Yanked
0.5.0 - 2018-11-11
Added
#[pyclass]
objects can now be returned from rust functionsPyComplex
by kngwyu in #226PyDict::from_sequence
, equivalent todict([(key, val), ...])
- Bindings for the
datetime
standard library types:PyDate
,PyTime
,PyDateTime
,PyTzInfo
,PyDelta
with associatedffi
types, by pganssle #200. PyString
,PyUnicode
, andPyBytes
now have anas_bytes
method that returns&[u8]
.PyObjectProtocol::get_type_ptr
by ijl in #242
Changed
- Removes the types from the root module and the prelude. They now live in
pyo3::types
instead. - All exceptions are constructed with
py_err
instead ofnew
, as they returnPyErr
and notSelf
. as_mut
and friends take and&mut self
instead of&self
ObjectProtocol::call
now takes anOption<&PyDict>
for the kwargs instead of anIntoPyDictPointer
.IntoPyDictPointer
was replace byIntoPyDict
which doesn't convertPyDict
itself anymore and returns aPyDict
instead of*mut PyObject
.PyTuple::new
now takes anIntoIterator
instead of a slice- Updated to syn 0.15
- Splitted
PyTypeObject
intoPyTypeObject
without the create method andPyTypeCreate
with requiresPyObjectAlloc<Self> + PyTypeInfo + Sized
. - Ran
cargo edition --fix
which prefixed path withcrate::
for rust 2018 - Renamed
async
topyasync
as async will be a keyword in the 2018 edition. - Starting to use
NonNull<*mut PyObject>
for Py and PyObject by ijl #260
Removed
- Removed most entries from the prelude. The new prelude is small and clear.
- Slowly removing specialization uses
PyString
,PyUnicode
, andPyBytes
no longer have adata
method (replaced byas_bytes
) andPyStringData
has been removed.- The pyobject_extract macro
Fixed
- Added an explanation that the GIL can temporarily be released even while holding a GILGuard.
- Lots of clippy errors
- Fix segfault on calling an unknown method on a PyObject
- Work around a bug in the rust compiler by kngwyu #252
- Fixed a segfault with subclassing pyo3 create classes and using
__class__
by kngwyu #263
0.4.1 - 2018-08-20
Changed
- PyTryFrom's error is always to
PyDowncastError
Fixed
- Fixed compilation on nightly since
use_extern_macros
was stabilized
Removed
- The pyobject_downcast macro
0.4.0 - 2018-07-30
Changed
- Merged both examples into one
- Rustfmt all the things :heavy_check_mark:
- Switched to Keep a Changelog
Removed
- Conversions from tuples to PyDict due to rust-lang/rust#52050
0.3.2 - 2018-07-22
Changed
- Replaced
concat_idents
with mashup
0.3.1 - 2018-07-18
Fixed
- Fixed scoping bug in pyobject_native_type that would break rust-numpy
0.3.0 - 2018-07-18
Added
- A few internal macros became part of the public api (#155, #186)
- Always clone in getters. This allows using the get-annotation on all Clone-Types
Changed
- Upgraded to syn 0.14 which means much better error messages :tada:
- 128 bit integer support by kngwyu (#137)
proc_macro
has been stabilized on nightly (rust-lang/rust#52081). This means that we can remove theproc_macro
feature, but now we need theuse_extern_macros
from the 2018 edition instead.- All proc macro are now prefixed with
py
and live in the prelude. This means you can use#[pyclass]
,#[pymethods]
,#[pyproto]
,#[pyfunction]
and#[pymodinit]
directly, at least after ause pyo3::prelude::*
. They were also moved into a module calledproc_macro
. You shouldn't use#[pyo3::proc_macro::pyclass]
or other longer paths in attributes becauseproc_macro_path_invoc
isn't going to be stabilized soon. - Renamed the
base
option in thepyclass
macro toextends
. #[pymodinit]
uses the function name as module name, unless the name is overrriden with#[pymodinit(name)]
- The guide is now properly versioned.
0.2.7 - 2018-05-18
Fixed
- Fix nightly breakage with proc_macro_path
0.2.6 - 2018-04-03
Fixed
- Fix compatibility with TryFrom trait #137
0.2.5 - 2018-02-21
Added
- CPython 3.7 support
Fixed
- Embedded CPython 3.7b1 crashes on initialization #110
- Generated extension functions are weakly typed #108
- call_method* crashes when the method does not exist #113
- Allow importing exceptions from nested modules #116
0.2.4 - 2018-01-19
Added
- Allow to get mutable ref from PyObject #106
- Drop
RefFromPyObject
trait - Add Python::register_any method
Fixed
- Fix impl
FromPyObject
forPy<T>
- Mark method that work with raw pointers as unsafe #95
0.2.3 - 11-27-2017
Changed
- Rustup to 1.23.0-nightly 2017-11-07
Fixed
- Proper
c_char
usage #93
Removed
- Remove use of now unneeded 'AsciiExt' trait
0.2.2 - 09-26-2017
Changed
- Rustup to 1.22.0-nightly 2017-09-30
0.2.1 - 09-26-2017
Fixed
- Fix rustc const_fn nightly breakage
0.2.0 - 08-12-2017
Added
- Added inheritance support #15
- Added weakref support #56
- Added subclass support #64
- Added
self.__dict__
supoort #68 - Added
pyo3::prelude
module #70 - Better
Iterator
support for PyTuple, PyList, PyDict #75 - Introduce IntoPyDictPointer similar to IntoPyTuple #69
Changed
- Allow to add gc support without implementing PyGCProtocol #57
- Refactor
PyErr
implementation. Droppy
parameter from constructor.
0.1.0 - 07-23-2017
Added
- Initial release
Contributing
Thank you for your interest in contributing to PyO3! All are welcome - please consider reading our Code of Conduct to keep our community positive and inclusive.
If you are searching for ideas how to contribute, proceed to the "Getting started contributing" section. If you have found a specific issue to contribute to and need information about the development process, you may find the section "Writing pull requests" helpful.
If you want to become familiar with the codebase, see Architecture.md.
Getting started contributing
Please join in with any part of PyO3 which interests you. We use GitHub issues to record all bugs and ideas. Feel free to request an issue to be assigned to you if you want to work on it.
You can browse the API of the non-public parts of PyO3 here.
The following sections also contain specific ideas on where to start contributing to PyO3.
Setting up a development environment
To work and develop PyO3, you need Python & Rust installed on your system.
- We encourage the use of rustup to be able to select and choose specific toolchains based on the project.
- Pyenv is also highly recommended for being able to choose a specific Python version.
- virtualenv can also be used with or without Pyenv to use specific installed Python versions.
nox
is used to automate many of our CI tasks.
Caveats
- When using pyenv on macOS, installing a Python version using
--enable-shared
is required to make it work. i.eenv PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.7.12
Help users identify bugs
The PyO3 Discord server is very active with users who are new to PyO3, and often completely new to Rust. Helping them debug is a great way to get experience with the PyO3 codebase.
Helping others often reveals bugs, documentation weaknesses, and missing APIs. It's a good idea to open GitHub issues for these immediately so the resolution can be designed and implemented!
Implement issues ready for development
Issues where the solution is clear and work is not in progress use the needs-implementer label.
Don't be afraid if the solution is not clear to you! The core PyO3 contributors will be happy to mentor you through any questions you have to help you write the solution.
Help write great docs
PyO3 has a user guide (using mdbook) as well as the usual Rust API docs. The aim is for both of these to be detailed, easy to understand, and up-to-date. Pull requests are always welcome to fix typos, change wording, add examples, etc.
There are some specific areas of focus where help is currently needed for the documentation:
- Issues requesting documentation improvements are tracked with the documentation label.
- Not all APIs had docs or examples when they were made. The goal is to have documentation on all PyO3 APIs (#306). If you see an API lacking a doc, please write one and open a PR!
To build the docs (including all features), install nox
and then run
nox -s docs -- open
Doctests
We use lots of code blocks in our docs. Run cargo test --doc
when making changes to check that
the doctests still work, or cargo test
to run all the tests including doctests. See
https://doc.rust-lang.org/rustdoc/documentation-tests.html for a guide on doctests.
Building the guide
You can preview the user guide by building it locally with mdbook
.
First, install mdbook
and nox
. Then, run
nox -s build-guide -- --open
To check all links in the guide are valid, also install lychee
and use the check-guide
session instead:
nox -s check-guide
Help design the next PyO3
Issues which don't yet have a clear solution use the needs-design label.
If any of these issues interest you, please join in with the conversation on the issue! All opinions are valued, and if you're interested in going further with e.g. draft PRs to experiment with API designs, even better!
Review pull requests
Everybody is welcome to submit comments on open PRs. Please help ensure new PyO3 APIs are safe, performant, tidy, and easy to use!
Writing pull requests
Here are a few things to note when you are writing PRs.
Continuous Integration
The PyO3 repo uses GitHub Actions. PRs are blocked from merging if CI is not successful.
Formatting, linting and tests are checked for all Rust and Python code. In addition, all warnings in Rust code are disallowed (using RUSTFLAGS="-D warnings"
).
Tests run with all supported Python versions with the latest stable Rust compiler, as well as for Python 3.9 with the minimum supported Rust version.
If you are adding a new feature, you should add it to the full
feature in our Cargo.toml* so that it is tested in CI.
You can run these tests yourself with
nox
. Use nox -l
to list the full set of subcommands you can run.
UI Tests
PyO3 uses [trybuild
][trybuild] to develop UI tests to capture error messages from the Rust compiler for some of the macro functionality.
Because there are several feature combinations for these UI tests, when updating them all (e.g. for a new Rust compiler version) it may be helpful to use the update-ui-tests
nox session:
nox -s update-ui-tests
Documenting changes
We use towncrier to generate a CHANGELOG for each release.
To include your changes in the release notes, you should create one (or more) news items in the newsfragments
directory. Valid news items should be saved as <PR>.<CATEGORY>.md
where <PR>
is the pull request number and <CATEGORY>
is one of the following:
packaging
- for dependency changes and Python / Rust version compatibility changesadded
- for new featureschanged
- for features which already existed but have been altered or deprecatedremoved
- for features which have been removedfixed
- for "changed" features which were classed as a bugfix
Docs-only PRs do not need news items; start your PR title with docs:
to skip the check.
Style guide
Generic code
PyO3 has a lot of generic APIs to increase usability. These can come at the cost of generic code bloat. Where reasonable, try to implement a concrete sub-portion of generic functions. There are two forms of this:
- If the concrete sub-portion doesn't benefit from re-use by other functions, name it
inner
and keep it as a local to the function. - If the concrete sub-portion is re-used by other functions, preferably name it
_foo
and place it directly belowfoo
in the source code (wherefoo
is the original generic function).
FFI calls
PyO3 makes a lot of FFI calls to Python's C API using raw pointers. Where possible try to avoid using pointers-to-temporaries in expressions:
// dangerous
pyo3::ffi::Something(name.to_object(py).as_ptr());
// because the following refactoring is a use-after-free error:
let name = name.to_object(py).as_ptr();
pyo3::ffi::Something(name)
Instead, prefer to bind the safe owned PyObject
wrapper before passing to ffi functions:
let name: PyObject = name.to_object(py);
pyo3::ffi::Something(name.as_ptr())
// name will automatically be freed when it falls out of scope
Python and Rust version support policy
PyO3 aims to keep sufficient compatibility to make packaging Python extensions built with PyO3 feasible on most common package managers.
To keep package maintainers' lives simpler, PyO3 will commit, wherever possible, to only adjust minimum supported Rust and Python versions at the same time. This bump will only come in an 0.x
release, roughly once per year, after the oldest supported Python version reaches its end-of-life. (Check https://endoflife.date/python for a clear timetable on these.)
Below are guidelines on what compatibility all PRs are expected to deliver for each language.
Python
PyO3 supports all officially supported Python versions, as well as the latest PyPy3 release. All of these versions are tested in CI.
Rust
PyO3 aims to make use of up-to-date Rust language features to keep the implementation as efficient as possible.
The minimum Rust version supported will be decided when the release which bumps Python and Rust versions is made. At the time, the minimum Rust version will be set no higher than the lowest Rust version shipped in the current Debian, RHEL and Alpine Linux distributions.
CI tests both the most recent stable Rust version and the minimum supported Rust version. Because of Rust's stability guarantees this is sufficient to confirm support for all Rust versions in between.
Benchmarking
PyO3 has two sets of benchmarks for evaluating some aspects of its performance. The benchmark suite is currently very small - please open PRs with new benchmarks if you're interested in helping to expand it!
First, there are Rust-based benchmarks located in the pyo3-benches
subdirectory. You can run these benchmarks with:
nox -s bench
Second, there is a Python-based benchmark contained in the pytests
subdirectory. You can read more about it here.
Code coverage
You can view what code is and isn't covered by PyO3's tests. We aim to have 100% coverage - please check coverage and add tests if you notice a lack of coverage!
- First, ensure the llmv-cov cargo plugin is installed. You may need to run the plugin through cargo once before using it with
nox
.
cargo install cargo-llvm-cov
cargo llvm-cov
- Then, generate an
lcov.info
file with
nox -s coverage -- lcov
You can install an IDE plugin to view the coverage. For example, if you use VSCode:
- Add the coverage-gutters plugin.
- Add these settings to VSCode's
settings.json
:
{
"coverage-gutters.coverageFileNames": [
"lcov.info",
"cov.xml",
"coverage.xml",
],
"coverage-gutters.showLineCoverage": true
}
- You should now be able to see green highlights for code that is tested, and red highlights for code that is not tested.
Sponsor this project
At the moment there is no official organisation that accepts sponsorship on PyO3's behalf. If you're seeking to provide significant funding to the PyO3 ecosystem, please reach out to us on GitHub or Discord and we can discuss.
In the meanwhile, some of our maintainers have personal GitHub sponsorship pages and would be grateful for your support: