Overview
PyO3 is a Rust bindings for the Python interpreter.
Supported Python versions:
- Python2.7, Python 3.5 and up
Supported Rust version:
- Rust 1.20.0-nightly or later
- On Windows, we require rustc 1.20.0-nightly
Usage
To use pyo3
, add this to your Cargo.toml
:
[dependencies]
pyo3 = "0.2"
Example program displaying the value of sys.version
:
extern crate pyo3; use pyo3::{Python, PyDict, PyResult}; fn main() { let gil = Python::acquire_gil(); hello(gil.python()).unwrap(); } fn hello(py: Python) -> PyResult<()> { let sys = py.import("sys")?; let version: String = sys.get("version")?.extract()?; let locals = PyDict::new(py); locals.set_item("os", py.import("os")?)?; let user: String = py.eval("os.getenv('USER') or os.getenv('USERNAME')", None, Some(&locals))?.extract()?; println!("Hello {}, I'm Python {}", user, version); Ok(()) }
Example library with python bindings:
The following two files will build with cargo build
, and will generate a python-compatible library.
For MacOS, "-C link-arg=-undefined -C link-arg=dynamic_lookup" is required to build the library.
setuptools-rust
includes this by default.
See examples/word-count.
Also on macOS, you will need to rename the output from *.dylib to *.so.
On Windows, you will need to rename the output from *.dll to *.pyd.
Cargo.toml
:
[lib]
name = "rust2py"
crate-type = ["cdylib"]
[dependencies.pyo3]
version = "0.2"
features = ["extension-module"]
src/lib.rs
#![feature(proc_macro, specialization)] extern crate pyo3; use pyo3::{py, PyResult, Python, PyModule}; use pyo3::py::modinit as pymodinit; // add bindings to the generated python module // N.B: names: "librust2py" must be the name of the `.so` or `.pyd` file /// This module is implemented in Rust. #[pymodinit(rust2py)] fn init_mod(py: Python, m: &PyModule) -> PyResult<()> { #[pyfn(m, "sum_as_string")] // pyo3 aware function. All of our python interface could be declared in a separate module. // Note that the `#[pyfn()]` annotation automatically converts the arguments from // Python objects to Rust values; and the Rust return value back into a Python object. fn sum_as_string_py(_: Python, a:i64, b:i64) -> PyResult<String> { let out = sum_as_string(a, b); Ok(out) } Ok(()) } // logic implemented as a normal Rust function fn sum_as_string(a:i64, b:i64) -> String { format!("{}", a + b).to_string() } # fn main() {}
For setup.py
integration, see setuptools-rust
Ownership and Lifetimes
In Python, all objects are implicitly reference counted.
In Rust, we will use the PyObject
type
to represent a reference to a Python object.
Because all Python objects potentially have multiple owners, the concept of Rust mutability does not apply to Python objects. As a result, this API will allow mutating Python objects even if they are not stored in a mutable Rust variable.
The Python interpreter uses a global interpreter lock (GIL) to ensure thread-safety.
This API uses a zero-sized struct Python<'p>
as a token to indicate
that a function can assume that the GIL is held.
You obtain a Python
instance
by acquiring the GIL, and have to pass it into some operations that call into the Python runtime.
PyO3 library provides wrappers for python native objects. Ownership of python objects are disallowed because any access to python runtime has to be protected by GIL. All APIs are available through references. Lifetimes of python object's references are bound to GIL lifetime.
There are two types of pointers that could be stored on Rust structs.
Both implements Send
and Sync
traits and maintain python object's reference count.
-
PyObject
is general purpose type. It does not maintain type of the referenced object. It provides helper methods for extracting Rust values and casting to specific python object type. -
Py<T>
represents a reference to a concrete python objectT
.
To upgrade to a reference AsPyRef
trait can be used.
Getting Started
In this tutorial, we will walk through the steps of building a simple Python extension called TODO
.
Install Rust
Before we begin, we need to install Rust using the rustup installer:
curl https://sh.rustup.rs -sSf | sh
If you already have rustup installed, run this command to ensure you have the latest version of Rust:
rustup update
Type Conversions
PyO3
provides some handy traits to convert between Python types and Rust types.
ToPyObject
and IntoPyObject
trait
ToPyObject
trait is a conversion trait that allows various objects to be converted into PyObject
. IntoPyObject
serves the same purpose except it consumes self
.
IntoPyTuple
trait
IntoPyTuple
trait is a conversion trait that allows various objects to be converted into PyTuple
object.
For example, IntoPyTuple
trait is implemented for ()
so that you can convert it into a empty PyTuple
extern crate pyo3; use pyo3::{Python, IntoPyTuple}; fn main() { let gil = Python::acquire_gil(); let py = gil.python(); let py_tuple = ().into_tuple(py); }
FromPyObject
and RefFromPyObject
trait
*args
and **kwargs
for python object call
There are several way how to pass positional and keyword arguments to python object call.
ObjectProtocol
trait
provides two methods:
call
- call callable python object.call_method
- call specific method on the object.
Both methods accept args
and kwargs
arguments. args
argument is generate over
IntoPyTuple
trait. So args could be PyTuple
instance or
rust tuple with up to 10 elements. Or NoArgs
object which represents empty tuple object.
extern crate pyo3;
use pyo3::prelude::*;
fn main() {
let gil = Python::acquire_gil();
let py = gil.python();
let obj = SomeObject::new();
// call object without empty arguments
obj.call(NoArgs, NoArg);
// call object with PyTuple
let args = PyTuple::new(py, &[arg1, arg2, arg3]);
obj.call(args, NoArg);
// pass arguments as rust tuple
let args = (arg1, arg2, arg3);
obj.call(args, NoArg);
}
kwargs
argument is generate over
IntoPyDictPointer
trait. HashMap
or BTreeMap
could be used as
keyword arguments. rust tuple with up to 10 elements where each element is tuple with size 2
could be used as kwargs as well. Or NoArgs
object can be used to indicate that
no keywords arguments are provided.
extern crate pyo3;
use pyo3::prelude::*;
fn main() {
let gil = Python::acquire_gil();
let py = gil.python();
let obj = SomeObject::new();
// call object with PyDict
let kwargs = PyDict::new(py);
kwargs.set_item(key, value);
obj.call(NoArg, kwargs);
// pass arguments as rust tuple
let kwargs = ((key1, val1), (key2, val2), (key3, val3));
obj.call(args, kwargs);
// pass arguments as HashMap
let mut kwargs = HashMap::<i32, i32>::new();
kwargs.insert(1, 1);
obj.call(args, kwargs);
}
TODO
Python Exception
Define a new exception
You can use the py_exception!
macro to define a new exception type:
# #![allow(unused_variables)] #fn main() { py_exception!(module, MyError); #}
module
is the name of the containing module.MyError
is the name of the new exception type.
For example:
#[macro_use] extern crate pyo3; use pyo3::{Python, PyDict}; py_exception!(mymodule, CustomError); fn main() { let gil = Python::acquire_gil(); let py = gil.python(); let ctx = PyDict::new(py); ctx.set_item(py, "CustomError", py.get_type::<CustomError>()).unwrap(); py.run("assert str(CustomError) == \"<class 'mymodule.CustomError'>\"", None, Some(&ctx)).unwrap(); py.run("assert CustomError('oops').args == ('oops',)", None, Some(&ctx)).unwrap(); }
Raise an exception
To raise an exception, first you need to obtain an exception type and construct a new PyErr
, then call PyErr::restore()
method to write the exception back to the Python interpreter's global state.
extern crate pyo3; use pyo3::{Python, PyErr, exc}; fn main() { let gil = Python::acquire_gil(); let py = gil.python(); PyErr::new::<exc::TypeError, _>("Error").restore(py); assert!(PyErr::occurred(py)); drop(PyErr::fetch(py)); }
If you already have a Python exception instance, you can simply call PyErr::from_instance()
.
# #![allow(unused_variables)] #fn main() { PyErr::from_instance(py, err).restore(py); #}
If rust type exists for exception, then it is possible to use new
method.
For example each standard exception defined in exc
module
has corresponding rust type, exceptions defined by py_exception!
and import_exception!
macro
have rust type as well.
# #![allow(unused_variables)] #fn main() { fn my_func(arg: PyObject) -> PyResult<()> { if check_for_error() { Err(exc::ValueError::new("argument is wrong")) } else { Ok(()) } } #}
Check exception type
Python has an isinstance
method to check object type,
in PyO3
there is a Python::is_instance()
method which does the same thing.
extern crate pyo3; use pyo3::{Python, PyBool, PyList}; fn main() { let gil = Python::acquire_gil(); let py = gil.python(); assert!(py.is_instance::<PyBool>(py.True().as_ref()).unwrap()); let list = PyList::new(py, &[1, 2, 3, 4]); assert!(!py.is_instance::<PyBool>(list.as_ref()).unwrap()); assert!(py.is_instance::<PyList>(list.as_ref()).unwrap()); }
Python::is_instance()
calls the underlying PyType::is_instance
method to do the actual work.
To check the type of an exception, you can simply do:
# #![allow(unused_variables)] #fn main() { let ret = py.is_instance::<exc::TypeError>(&err.instance(py)).expect("Error calling is_instance"); #}
Handle Rust Error
The vast majority of operations in this library will return PyResult<T>
.
This is an alias for the type Result<T, PyErr>
.
A PyErr
represents a Python exception.
Errors within the PyO3
library are also exposed as Python exceptions.
PyO3 library handles python exception in two stages. During first stage PyErr
instance get
created. At this stage python GIL is not required. During second stage, actual python
exception instance get crated and set to python interpreter.
In simple case, for custom errors support implementation of std::convert::From<T>
trait
for this custom error is enough. PyErr::new
accepts arguments in form
of ToPyObject + 'static
. In case if 'static
constraint can not be satisfied or
more complex arguments are required PyErrArgument
trait can be implemented. In that case actual exception arguments creation get delayed
until Python
object is available.
# #![allow(unused_variables)] #fn main() { use std; use std::net::TcpListener; use pyo3::{PyErr, PyResult, ToPyErr, exc}; impl std::convert::From<std::io::Error> for PyErr { fn from(err: std::io::Error) -> PyErr { exc::OSError.into() } } fn connect(s: String) -> PyResult<bool> { TcpListener::bind("127.0.0.1:80")?; Ok(true) } #}
The code snippet above will raise OSError
in Python if TcpListener::bind()
return an error.
std::convert::From<T>
trait is implemented for most of the standard library's error
types so try!
macro or ?
operator can be used.
# #![allow(unused_variables)] #fn main() { use pyo3::*; fn parse_int(s: String) -> PyResult<usize> { Ok(s.parse::<usize>()?) } #}
The code snippet above will raise ValueError
in Python if String::parse()
return an error.
Using exceptions defined in python code
It is possible to use exception defined in python code as native rust types.
import_exception!
macro allows to import specific exception class and defined zst type
for that exception.
# #![allow(unused_variables)] #fn main() { use pyo3::{PyErr, PyResult, exc}; import_exception!(asyncio, CancelledError) fn cancel(fut: PyFuture) -> PyResult<()> { if fut.cancelled() { Err(CancelledError.into()) } Ok(()) } #}
exc
defines exceptions for
several standard library modules.
Python Module
Python module generation is powered by unstable Procedural Macros feature, so you need to turn on proc_macro
feature:
#![feature(proc_macro)] extern crate pyo3; # fn main() {}
You need to change your crate-type
to cdylib
to produce a Python compatible library:
[lib]
name = "rust2py"
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.2", features = ["extension-module"] }
Now you can write your module, for example
#![feature(proc_macro)] extern crate pyo3; use pyo3::{py, PyResult, Python, PyModule}; use pyo3::py::modinit as pymodinit; // add bindings to the generated python module // N.B: names: "librust2py" must be the name of the `.so` or `.pyd` file /// This module is implemented in Rust. #[pymodinit(rust2py)] fn init_mod(py: Python, m: &PyModule) -> PyResult<()> { // pyo3 aware function. All of our python interface could be declared in a separate module. // Note that the `#[pyfn()]` annotation automatically converts the arguments from // Python objects to Rust values; and the Rust return value back into a Python object. #[pyfn(m, "sum_as_string")] fn sum_as_string_py(_: Python, a:i64, b:i64) -> PyResult<String> { let out = sum_as_string(a, b); Ok(out) } Ok(()) } // logic implemented as a normal rust function fn sum_as_string(a:i64, b:i64) -> String { format!("{}", a + b).to_string() } # fn main() {}
The modinit
procedural macro attribute takes care of exporting the initialization function of your module to Python. It takes one argument as the name of your module, it must be the name of the .so
or .pyd
file.
The Rust doc comments of the module initialization function will be applied automatically as the Python doc string of your module.
import rust2py
print(rust2py.__doc__)
Which means that the above Python code will print This module is implemented in Rust.
.
On macOS, you will need to rename the output from
*.dylib
to*.so
.On Windows, you will need to rename the output from
*.dll
to*.pyd
.
For setup.py
integration, You can use setuptools-rust,
learn more about it in Distribution.
Python Function
TODO
Python Class
Python class generation is powered by unstable Procedural Macros and
Specialization and Const fn
features, so you need to turn on proc_macro
and specialization
features:
# #![allow(unused_variables)] #![feature(proc_macro, specialization, const_fn)] #fn main() { extern crate pyo3; #}
Define new class
To define python custom class, rust struct needs to be annotated with #[py::class]
attribute.
# #![allow(unused_variables)] #fn main() { use pyo3::prelude::*; #[py::class] struct MyClass { num: i32, debug: bool, token: PyToken, } #}
The above example generates the following implementations for MyClass
struct
impl PyTypeInfo for MyClass { ... }
impl PyTypeObject for MyClass { ... }
impl PyObjectWithToken for MyClass { ... }
impl ToPyObject for MyClass { ... }
impl IntoPyObject for MyClass { ... }
impl ToPyPointer for MyClass { ... }
Following implementations PyObjectWithToken
, ToPyObject
, IntoPyObject
, ToPyPointer
are generated only if struct contains PyToken
attribute.
PyToken
instance available only in py.init
method.
TODO - continue
py::class macro
Python class generation is powered by Procedural Macros.
To define python custom class, rust struct needs to be annotated with #[py::class]
attribute.
py::class
macro accepts following parameters:
name=XXX
- customize class name visible to python code. By default struct name is used as a class name.freelist=XXX
-freelist
parameter add support of free allocation list to custom class. The performance improvement applies to types that are often created and deleted in a row, so that they can benefit from a freelist.XXX
is a number of items for free list.gc
- adds support for python garbage collector. classes that build withgc
parameter participate in python garbage collector. If custom class contains references to other python object that can be collectorPyGCProtocol
trait has to be implemented.weakref
- adds support for python weak referencesbase=BaseType
- use custom base class. BaseType is type which is implementsPyTypeInfo
trait.subclass
- adds subclass support so that Python classes can inherit from this classdict
- adds__dict__
support, the instances of this type have a dictionary containing instance variables
Constructor
By default it is not possible to create instance of custom class from python code.
To declare constructor, you need to define class method and annotate it with #[new]
attribute. Only python __new__
method can be specified, __init__
is not available.
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { #[new] fn __new__(obj: &PyRawObject, ...) -> PyResult<()> { obj.init(|token| { MyClass { num: 10, debug: false, token: token } }) } } #}
Some rules of new
method
- If no method marked with
#[new]
is declared, object instances can only be created from Rust, but not from Python. - The first parameter is the raw object, custom
new
method must initialize object with value of struct usinginit
method. Type of the object may be the type object of a derived class declared in Python. - The first parameter implicitly has type
&PyRawObject
. - For details on
parameter-list
, see the documentation ofMethod arguments
section. - The return type must be
PyResult<T>
for someT
that implementsIntoPyObject
. Usually,T
will beMyType
.
Inheritance
By default PyObject
is used as default base class. To override default base class
base
parameter for py::class
needs to be used. Value is full path to base class.
__new__
method accepts PyRawObject
object. obj
instance must be initialized
with value of custom class struct. Subclass must call parent's __new__
method.
# #![allow(unused_variables)] #fn main() { #[py::class] struct BaseClass { val1: usize } #[py::methods] impl BaseClass { #[new] fn __new__(obj: &PyRawObject) -> PyResult<()> { obj.init(|t| BaseClass{val1: 10}) } pub fn method(&self) -> PyResult<() { Ok(()) } } #[py::class(base=BaseClass)] struct SubClass { val2: usize } #[py::methods] impl SubClass { #[new] fn __new__(obj: &PyRawObject) -> PyResult<()> { obj.init(|t| SubClass{val2: 10}) BaseClass::__new__(obj) } fn method2(&self) -> PyResult<()> { self.get_base().method() } } #}
ObjectProtocol
trait provides get_base()
method. It returns reference to instance of
base class.
Object properties
Descriptor methods can be defined in
#[py::methods]
impl
block only and has to be annotated with #[getter]
or [setter]
attributes. i.e.
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { #[getter] fn num(&self) -> PyResult<i32> { Ok(self.num) } } #}
Getter or setter function's name is used as property name by default. There are several ways how to override name.
If function name starts with get_
or set_
for getter or setter respectively.
Descriptor name becomes function name with prefix removed. This is useful in case os
rust's special keywords like type
.
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { #[getter] fn get_num(&self) -> PyResult<i32> { Ok(self.num) } #[setter] fn set_num(&mut self, value: i32) -> PyResult<()> { self.num = value Ok(()) } } #}
In this case property num
is defined. And it is available from python code as self.num
.
Also both #[getter]
and #[setter]
attributes accepts one parameter.
If parameter is specified, it is used and property name. i.e.
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { #[getter(number)] fn num(&self) -> PyResult<i32> { Ok(self.num) } #[setter(number)] fn set_num(&mut self, value: i32) -> PyResult<()> { self.num = value Ok(()) } } #}
In this case property number
is defined. And it is available from python code as self.number
.
For simple cases you can also define getters and setters in your Rust struct field definition, for example:
# #![allow(unused_variables)] #fn main() { #[py:class] struct MyClass { #[prop(get, set)] num: i32 } #}
Then it is available from Python code as self.num
.
Instance methods
To define python compatible method, impl
block for struct has to be annotated
with #[py::methods]
attribute. pyo3
library generates python compatible
wrappers for all functions in this block with some variations, like descriptors,
class method static methods, etc.
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { fn method1(&self) -> PyResult<i32> { Ok(10) } fn set_method(&mut self, value: i32) -> PyResult<()> { self.num = value Ok(()) } } #}
Calls to this methods protected by GIL
, &self
or &mut self
can be used.
The return type must be PyResult<T>
for some T
that implements IntoPyObject
.
Python
parameter can be specified as part of method signature, in this case py
argument
get injected by method wrapper. i.e
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { fn method2(&self, py: Python) -> PyResult<i32> { Ok(10) } } #}
From python perspective method2
, in above example, does not accept any arguments.
Class methods
To specify class method for custom class, method needs to be annotated
with#[classmethod]
attribute.
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { #[classmethod] fn cls_method(cls: &PyType) -> PyResult<i32> { Ok(10) } } #}
Declares a class method callable from Python.
- The first parameter is the type object of the class on which the method is called. This may be the type object of a derived class.
- The first parameter implicitly has type
&PyType
. - For details on
parameter-list
, see the documentation ofMethod arguments
section. - The return type must be
PyResult<T>
for someT
that implementsIntoPyObject
.
Static methods
To specify class method for custom class, method needs to be annotated
with #[staticmethod]
attribute. The return type must be PyResult<T>
for some T
that implements IntoPyObject
.
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { #[staticmethod] fn static_method(param1: i32, param2: &str) -> PyResult<i32> { Ok(10) } } #}
Callable object
To specify custom __call__
method for custom class, call method needs to be annotated
with #[call]
attribute. Arguments of the method are specified same as for instance method.
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { #[call] #[args(args="*")] fn __call__(&self, args: &PyTuple) -> PyResult<i32> { println!("MyCLS has been called"); Ok(self.num) } } #}
Method arguments
By default pyo3 library uses function signature to determine which arguments are required.
Then it scans incoming args
parameter and then incoming kwargs
parameter. If it can not
find all required parameters, it raises TypeError
exception.
It is possible to override default behavior with #[args(...)]
attribute. args
attribute
accept comma separated list of parameters in form attr_name="default value"
. Each parameter
has to match method parameter by name.
Each parameter could one of following type:
- "*": var arguments separator, each parameter defined after "*" is keyword only parameters.
corresponds to python's
def meth(*, arg1.., arg2=..)
- args="*": "args" is var args, corresponds to python's
def meth(*args)
. Type ofargs
parameter has to be&PyTuple
. - kwargs="**": "kwargs" is keyword arguments, corresponds to python's
def meth(**kwargs)
. Type ofkwargs
parameter has to beOption<&PyDict>
. - arg="Value": arguments with default value. corresponds to python's
def meth(arg=Value)
. ifarg
argument is defined after var arguments it is treated as keyword argument. Note thatValue
has to be valid rust code, pyo3 just inserts it into generated code unmodified.
Example:
# #![allow(unused_variables)] #fn main() { #[py::methods] impl MyClass { #[args(arg1=true, args="*", arg2=10, kwargs="**")] fn method(&self, arg1: bool, args: &PyTuple, arg2: i32, kwargs: Option<&PyDict>) -> PyResult<i32> { Ok(1) } } #}
Class customizations
Python object model defines several protocols for different object behavior,
like sequence, mapping or number protocols. pyo3 library defines separate trait for each
of them. To provide specific python object behavior you need to implement specific trait
for your struct. Important note, each protocol implementation block has to be annotated
with #[py::proto]
attribute.
Basic object customization
PyObjectProtocol
trait provide several basic customizations.
Attribute access
To customize object attribute access define following methods:
fn __getattr__(&self, name: FromPyObject) -> PyResult<impl IntoPyObject>
fn __setattr__(&mut self, name: FromPyObject, value: FromPyObject) -> PyResult<()>
fn __delattr__(&mut self, name: FromPyObject) -> PyResult<()>
Each methods corresponds to python's self.attr
, self.attr = value
and del self.attr
code.
String Conversions
-
fn __repr__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>
-
fn __str__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>
Possible return types for
__str__
and__repr__
arePyResult<String>
orPyResult<PyString>
. In Python 2.7, Unicode strings returned by__str__
and__repr__
will be converted to byte strings by the Python runtime, which results in an exception if the string contains non-ASCII characters. -
fn __bytes__(&self) -> PyResult<PyBytes>
On Python 3.x, provides the conversion to
bytes
. On Python 2.7,__bytes__
is allowed but has no effect. -
fn __unicode__(&self) -> PyResult<PyUnicode>
On Python 2.7, provides the conversion to
unicode
. On Python 3.x,__unicode__
is allowed but has no effect. -
fn __format__(&self, format_spec: &str) -> PyResult<impl ToPyObject<ObjectType=PyString>>
Special method that is used by the
format()
builtin and thestr.format()
method. Possible return types arePyResult<String>
orPyResult<PyString>
.
Comparison operators
-
fn __richcmp__(&self, other: impl FromPyObject, op: CompareOp) -> PyResult<impl ToPyObject>
Overloads Python comparison operations (
==
,!=
,<
,<=
,>
, and>=
). Theop
argument indicates the comparison operation being performed. The return type will normally bePyResult<bool>
, but any Python object can be returned. Ifother
is not of the type specified in the signature, the generated code will automaticallyreturn NotImplemented
. -
fn __hash__(&self) -> PyResult<impl PrimInt>
Objects that compare equal must have the same hash value. The return type must be
PyResult<T>
whereT
is one of Rust's primitive integer types.
Other methods
-
fn __bool__(&self) -> PyResult<bool>
Determines the "truthyness" of the object. This method works for both python 3 and python 2, even on Python 2.7 where the Python spelling was
__nonzero__
.
Garbage Collector Integration
If your type owns references to other python objects, you will need to
integrate with Python's garbage collector so that the GC is aware of
those references.
To do this, implement PyGCProtocol
trait for your struct.
It includes two methods __traverse__
and __clear__
.
These correspond to the slots tp_traverse
and tp_clear
in the Python C API.
__traverse__
must call visit.call()
for each reference to another python object.
__clear__
must clear out any mutable references to other python objects
(thus breaking reference cycles). Immutable references do not have to be cleared,
as every cycle must contain at least one mutable reference.
Example:
# #![allow(unused_variables)] #![feature(proc_macro, specialization)] #fn main() { extern crate pyo3; use pyo3::{py, PyObject, PyGCProtocol, PyVisit, PyTraverseError}; #[py::class] struct ClassWithGCSupport { obj: Option<PyObject>, } #[py::proto] impl PyGCProtocol for ClassWithGCSupport { fn __traverse__(&self, visit: PyVisit) -> Result<(), PyTraverseError> { if let Some(ref obj) = self.obj { visit.call(obj)? } Ok(()) } fn __clear__(&mut self) { if let Some(obj) = self.obj.take() { // Release reference, this decrements ref counter. self.py().release(obj); } } } #}
Special protocol trait implementation has to be annotated with #[py::proto]
attribute.
It is also possible to enable gc for custom class using gc
parameter for py::class
annotation.
i.e. #[py::class(gc)]
. In that case instances of custom class participate in python garbage
collector, and it is possible to track them with gc
module methods.
Iterator Types
Iterators can be defined using the
PyIterProtocol
trait.
It includes two methods __iter__
and __next__
:
fn __iter__(&mut self) -> PyResult<impl IntoPyObject>
fn __next__(&mut self) -> PyResult<Option<impl IntoPyObject>>
Returning Ok(None)
from __next__
indicates that that there are no further items.
Example:
#![feature(proc_macro, specialization)] extern crate pyo3; use pyo3::{py, PyObject, PyResult, PyIterProtocol}; #[py::class] struct MyIterator { iter: Box<Iterator<Item=PyObject> + Send> } #[py::proto] impl PyIterProtocol { fn __iter__(&mut self) -> PyResult<PyObject> { Ok(self.into()) } fn __next__(&mut self) -> PyResult<Option<PyObject>> { Ok(self.iter.next()) } } # fn main() {}
Parallelism
CPython has an infamous GIL(Global Interpreter Lock) prevents developers
getting true parallelism. With pyo3
you can release GIL when executing
Rust code to achieve true parallelism.
The Python::allow_threads
method temporarily releases the GIL, thus allowing other Python threads to run.
impl Python {
pub fn allow_threads<T, F>(self, f: F) -> T where F: Send + FnOnce() -> T {}
}
Let's take a look at our word-count example,
we have a wc_parallel
function utilize the rayon crate to count words in parallel.
fn wc_parallel(lines: &str, search: &str) -> i32 {
lines.par_lines()
.map(|line| wc_line(line, search))
.sum()
}
Then in the Python bridge, we have a function search
exposed to Python runtime which calls wc_parallel
inside
Python::allow_threads
method to enable true parallelism:
#[pymodinit(_word_count)]
fn init_mod(py: Python, m: &PyModule) -> PyResult<()> {
#[pyfn(m, "search")]
fn search(py: Python, path: String, search: String) -> PyResult<i32> {
let mut file = File::open(path)?;
let mut contents = String::new();
file.read_to_string(&mut contents)?;
let count = py.allow_threads(move || wc_parallel(&contents, &search));
Ok(count)
}
Ok(())
}
Benchmark
Let's benchmark the word-count
example to verify that we did unlock true parallelism with pyo3
.
We are using pytest-benchmark
to benchmark three word count functions:
Benchmark script can be found here,
then we can run pytest tests
to benchmark them.
On MacBook Pro (Retina, 15-inch, Mid 2015) the benchmark gives:
Distribution
setuptools-rust
integration
setuptools-rust
is a setuptools helpers for Rust Python extensions. It supports PyO3
by default.
For detailed usage, please refer to its README
Source distribution
To build a source code distribution, you need to add the following lines to your MANIFEST.in
file to ensure it correctly packages Rust extension source code.
include Cargo.toml
recursive-include src *
Then you can build a source code distribution by (assuming you have already written a setup.py
):
python setup.py sdist
Binary wheel distribution
To build a binary wheel, manylinux would be a natural choice for Linux.
Take the example project in setuptools-rust
repository for example,
we have a build-wheels.sh
to be used with Docker to build manylinux1 wheels.
First you need to pull the manylinux1
Docker image:
$ docker pull quay.io/pypa/manylinux1_x86_64
Then use the following command to build wheels for supported Python versions:
$ docker run --rm -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/build-wheels.sh
You will find all the wheels in dist
directory:
$ ls dist
hello_rust-1.0-cp27-cp27m-linux_x86_64.whl hello_rust-1.0-cp35-cp35m-linux_x86_64.whl
hello_rust-1.0-cp27-cp27m-manylinux1_x86_64.whl hello_rust-1.0-cp35-cp35m-manylinux1_x86_64.whl
hello_rust-1.0-cp27-cp27mu-linux_x86_64.whl hello_rust-1.0-cp36-cp36m-linux_x86_64.whl
hello_rust-1.0-cp27-cp27mu-manylinux1_x86_64.whl hello_rust-1.0-cp36-cp36m-manylinux1_x86_64.whl
The *-manylinux1_x86_64.whl
files are the manylinux1
wheels that you can upload to PyPI.