Rust bindings for Python. This includes running and interacting with python code from a rust binaries as well as writing native python modules.

Pyo3 supports python 2.7 as well as python 3.5 and up. The minimum required rust version is 1.29.0-nightly 2018-07-16.

You can either write a native python module in rust or use python from a rust binary.

Pyo3 can be used to generate a native python module.

Cargo.toml:

[package]
name = "rust-py"
version = "0.1.0"

[lib]
name = "rust_py"
crate-type = ["cdylib"]

[dependencies.pyo3]
version = "0.3"
features = ["extension-module"]

src/lib.rs

#![allow(unused_variables)]
#![feature(specialization)]

fn main() {
#[macro_use]
extern crate pyo3;

use pyo3::prelude::*;

#[pyfunction]
/// Formats the sum of two numbers as string
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Ok((a + b).to_string())
}

/// This module is a python moudle implemented in Rust.
#[pymodinit]
fn rust_py(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_function!(sum_as_string))?;

    Ok(())
}
}

On windows and linux, you can build normally with cargo build --release. On Mac Os, you need to set additional linker arguments. One option is to compile with cargo rustc --release -- -C link-arg=-undefined -C link-arg=dynamic_lookup, the other is to create a .cargo/config with the following content:

[target.x86_64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

Also on macOS, you will need to rename the output from *.dylib to *.so. On Windows, you will need to rename the output from *.dll to *.pyd.

setuptools-rust can be used to generate a python package and includes the commands above by default. See examples/word-count and the associated setup.py.

Add pyo3 this to your Cargo.toml:

[dependencies]
pyo3 = "0.3"

Example program displaying the value of sys.version:

#![feature(specialization)]

extern crate pyo3;

use pyo3::prelude::*;

fn main() -> PyResult<()> {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let sys = py.import("sys")?;
    let version: String = sys.get("version")?.extract()?;

    let locals = PyDict::new(py);
    locals.set_item("os", py.import("os")?)?;
    let user: String = py.eval("os.getenv('USER') or os.getenv('USERNAME')", None, Some(&locals))?.extract()?;

    println!("Hello {}, I'm Python {}", user, version);
    Ok(())
}

• examples/word-count Counting the occurences of a word in a text file
• hyperjson A hyper-fast Python module for reading/writing JSON data using Rust's serde-json
• rust-numpy Rust binding of NumPy C-API
• pyo3-built Simple macro to expose metadata obtained with the built crate as a PyDict
• point-process High level API for pointprocesses as a Python library

PyO3 provides some handy traits to convert between Python types and Rust types.

The easiest way to convert a python object to a rust value is using .extract().

ToPyObject trait is a conversion trait that allows various objects to be converted into PyObject. IntoPyObject serves the same purpose except it consumes self.

IntoPyTuple trait is a conversion trait that allows various objects to be converted into PyTuple object.

For example, IntoPyTuple trait is implemented for () so that you can convert it into a empty PyTuple

extern crate pyo3;
use pyo3::{Python, IntoPyTuple};

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let py_tuple = ().into_tuple(py);
}

There are several way how to pass positional and keyword arguments to python object call. ObjectProtocol trait provides two methods:

• call - call callable python object.
• call_method - call specific method on the object.

Both methods accept args and kwargs arguments. args argument is generate over IntoPyTuple trait. So args could be PyTuple instance or rust tuple with up to 10 elements. Or NoArgs object which represents empty tuple object.

extern crate pyo3;
use pyo3::prelude::*;

struct SomeObject;
impl SomeObject {
    fn new(py: Python) -> PyObject {
          pyo3::PyDict::new(py).to_object(py)
    }
}

fn main() {
    let arg1 = "arg1";
    let arg2 = "arg2";
    let arg3 = "arg3";

    let gil = Python::acquire_gil();
    let py = gil.python();

    let obj = SomeObject::new(py);

    // call object without empty arguments
    obj.call0(py);

    // call object with PyTuple
    let args = PyTuple::new(py, &[arg1, arg2, arg3]);
    obj.call1(py, args);

    // pass arguments as rust tuple
    let args = (arg1, arg2, arg3);
    obj.call1(py, args);
}

kwargs argument is generate over IntoPyDictPointer trait. HashMap or BTreeMap could be used as keyword arguments. rust tuple with up to 10 elements where each element is tuple with size 2 could be used as kwargs as well. Or NoArgs object can be used to indicate that no keywords arguments are provided.

extern crate pyo3;

use std::collections::HashMap;
use pyo3::prelude::*;

struct SomeObject;
impl SomeObject {
    fn new(py: Python) -> PyObject {
          pyo3::PyDict::new(py).to_object(py)
    }
}
fn main() {
    let key1 = "key1";
    let val1 = 1;
    let key2 = "key2";
    let val2 = 2;

    let gil = Python::acquire_gil();
    let py = gil.python();

    let obj = SomeObject::new(py);

    // call object with PyDict
    let kwargs = PyDict::new(py);
    kwargs.set_item(key1, val1);
    obj.call(py, NoArgs, kwargs);

    // pass arguments as rust tuple
    let kwargs = ((key1, val1), (key2, val2));
    obj.call(py, NoArgs, kwargs);

    // pass arguments as HashMap
    let mut kwargs = HashMap::<&str, i32>::new();
    kwargs.insert(key1, 1);
    obj.call(py, NoArgs, kwargs);
}

TODO

You can use the py_exception! macro to define a new exception type:

#![allow(unused_variables)]
fn main() {
#[macro_use] extern crate pyo3;

py_exception!(module, MyError);
}

• module is the name of the containing module.
• MyError is the name of the new exception type.

For example:

#[macro_use] extern crate pyo3;

use pyo3::{Python, PyDict};

py_exception!(mymodule, CustomError);

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let ctx = PyDict::new(py);

    ctx.set_item("CustomError", py.get_type::<CustomError>()).unwrap();

    py.run("assert str(CustomError) == \"<class 'mymodule.CustomError'>\"", None, Some(&ctx)).unwrap();
    py.run("assert CustomError('oops').args == ('oops',)", None, Some(&ctx)).unwrap();
}

To raise an exception, first you need to obtain an exception type and construct a new PyErr, then call PyErr::restore() method to write the exception back to the Python interpreter's global state.

extern crate pyo3;

use pyo3::{Python, PyErr, exc};

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    PyErr::new::<exc::TypeError, _>("Error").restore(py);
    assert!(PyErr::occurred(py));
    drop(PyErr::fetch(py));
}

If you already have a Python exception instance, you can simply call PyErr::from_instance().

PyErr::from_instance(py, err).restore(py);

If rust type exists for exception, then it is possible to use new method. For example each standard exception defined in exc module has corresponding rust type, exceptions defined by py_exception! and import_exception! macro have rust type as well.

#![allow(unused_variables)]
fn main() {
extern crate pyo3;
use pyo3::prelude::*;
fn check_for_error() -> bool {false}
fn my_func(arg: PyObject) -> PyResult<()> {
    if check_for_error() {
        Err(exc::ValueError::new("argument is wrong"))
    } else {
        Ok(())
    }
}
}

Python has an isinstance method to check object type, in PyO3 there is a Python::is_instance() method which does the same thing.

extern crate pyo3;

use pyo3::{Python, PyBool, PyList};

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    assert!(py.is_instance::<PyBool, _>(PyBool::new(py, true)).unwrap());
    let list = PyList::new(py, &[1, 2, 3, 4]);
    assert!(!py.is_instance::<PyBool, _>(list.as_ref()).unwrap());
    assert!(py.is_instance::<PyList, _>(list.as_ref()).unwrap());
}

Python::is_instance() calls the underlying PyType::is_instance method to do the actual work.

To check the type of an exception, you can simply do:

extern crate pyo3;
use pyo3::prelude::*;
fn main() {
let gil = Python::acquire_gil();
let py = gil.python();
let err = exc::TypeError::new(NoArgs);
err.is_instance::<exc::TypeError>(py);
}

The vast majority of operations in this library will return PyResult<T>. This is an alias for the type Result<T, PyErr>.

A PyErr represents a Python exception. Errors within the PyO3 library are also exposed as Python exceptions.

PyO3 library handles python exception in two stages. During first stage PyErr instance get created. At this stage python GIL is not required. During second stage, actual python exception instance get crated and set to python interpreter.

In simple case, for custom errors support implementation of std::convert::From<T> trait for this custom error is enough. PyErr::new accepts arguments in form of ToPyObject + 'static. In case if 'static constraint can not be satisfied or more complex arguments are required PyErrArgument trait can be implemented. In that case actual exception arguments creation get delayed until Python object is available.

#![feature(specialization)]
extern crate pyo3;

use std::net::TcpListener;
use pyo3::{PyErr, PyResult, exc};

impl std::convert::From<std::io::Error> for PyErr {
    fn from(err: std::io::Error) -> PyErr {
        exc::OSError.into()
    }
}

fn connect(s: String) -> PyResult<bool> {
    TcpListener::bind("127.0.0.1:80")?;
    Ok(true)
}

The code snippet above will raise OSError in Python if TcpListener::bind() return an error.

std::convert::From<T> trait is implemented for most of the standard library's error types so try! macro or ? operator can be used.

#![allow(unused_variables)]
fn main() {
extern crate pyo3;
use pyo3::prelude::*;

fn parse_int(s: String) -> PyResult<usize> {
    Ok(s.parse::<usize>()?)
}
}

The code snippet above will raise ValueError in Python if String::parse() return an error.

It is possible to use exception defined in python code as native rust types. import_exception! macro allows to import specific exception class and defined zst type for that exception.

#![allow(unused_variables)]
fn main() {
#[macro_use] extern crate pyo3;
use pyo3::prelude::*;

import_exception!(io, UnsupportedOperation);

fn tell(file: PyObject) -> PyResult<u64> {
    use pyo3::exc::*;

    let gil = Python::acquire_gil();
    let py = gil.python();

    match file.call_method0(py, "tell") {
        Err(_) => Err(UnsupportedOperation::new("not supported: tell")),
        Ok(x) => x.extract::<u64>(py),
    }    
}

}

exc defines exceptions for several standard library modules.

As shown in the Getting Started chapter, you can create a module as follows:

#![feature(proc_macro)]

extern crate pyo3;
use pyo3::{PyResult, Python, PyModule};



// add bindings to the generated python module
// N.B: names: "librust2py" must be the name of the `.so` or `.pyd` file
/// This module is implemented in Rust.
#[pymodinit]
fn rust2py(py: Python, m: &PyModule) -> PyResult<()> {

    // pyo3 aware function. All of our python interface could be declared in a separate module.
    // Note that the `#[pyfn()]` annotation automatically converts the arguments from
    // Python objects to Rust values; and the Rust return value back into a Python object.
    #[pyfn(m, "sum_as_string")]
    fn sum_as_string_py(_py: Python, a:i64, b:i64) -> PyResult<String> {
       let out = sum_as_string(a, b);
       Ok(out)
    }

    Ok(())
}

// logic implemented as a normal rust function
fn sum_as_string(a:i64, b:i64) -> String {
    format!("{}", a + b).to_string()
}

fn main() {}

The #[pymodinit} procedural macro attribute takes care of exporting the initialization function of your module to Python. It takes one argument as the name of your module, it must be the name of the .so or .pyd file.

The Rust doc comments of the module initialization function will be applied automatically as the Python doc string of your module.

import rust2py

print(rust2py.__doc__)

Which means that the above Python code will print This module is implemented in Rust..

On macOS, you will need to rename the output from *.dylib to *.so.

On Windows, you will need to rename the output from *.dll to *.pyd.

For setup.py integration, You can use setuptools-rust, learn more about it in Distribution.

Pyo3 supports two ways to define a function in python. Both require registering the function to a module

One way is defining the function in the module definition.

#![feature(proc_macro)]

extern crate pyo3;
use pyo3::prelude::*;


#[pymodinit]
fn rust2py(py: Python, m: &PyModule) -> PyResult<()> {

    // Note that the `#[pyfn()]` annotation automatically converts the arguments from
    // Python objects to Rust values; and the Rust return value back into a Python object.
    #[pyfn(m, "sum_as_string")]
    fn sum_as_string_py(_py: Python, a:i64, b:i64) -> PyResult<String> {
       Ok(format!("{}", a + b).to_string())
    }

    Ok(())
}

fn main() {}

The other is annotating a function with #[py::function] and then adding it to the module using the add_function_to_module! macro, which takes the module as first parameter, the function name as second and an instance of Python as third.

#![feature(specialization)]

#[macro_use]
extern crate pyo3;
use pyo3::prelude::*;

#[pyfunction]
fn double(x: usize) -> usize {
    x * 2
}

#[pymodinit]
fn module_with_functions(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_function!(double)).unwrap();

    Ok(())
}

fn main() {}

Currently, there are no conversions between Fns in rust and callables in python. This would definitely be possible and very useful, so contributions are welcome. In the meantime, you can do the following:

You can use ObjectProtocol::is_callable to check if you got a callable, which is true for functions (including lambdas), methods and objects with a __call__ method. You can call the object with ObjectProtocol::call with the args as first parameter and the kwargs (or NoArgs) as second paramter. There are also ObjectProtocol::call0 with no args and ObjectProtocol::call1 with only the args.

If you have a static function, you can expose it with #[pyfunction] and use wrap_function! to get the corresponding PyObject. For dynamic functions, e.g. lambda and functions that were passed as arguments, you must put them in some kind of owned container, e.g. a box. (Long-Term a special container similar to wasm-bindgen's Closure should take care of that). You can than use a #[pyclass] struct with that container as field as a way to pass the function over the ffi-barrier. You can even make that class callable with __call__ so it looks like a function in python code.

To define python custom class, rust struct needs to be annotated with #[pyclass] attribute.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;


#[pyclass]
struct MyClass {
   num: i32,
   debug: bool,
}
}

The above example generates implementations for PyTypeInfo and PyTypeObject for MyClass.

If the class has a PyToken attribute, implementations for PyObjectWithToken, ToPyObject, IntoPyObject and ToPyPointer are also generated. You can only get a PyToken instance through the __new__ method.

The #[pyclass] macro accepts following parameters:

• name=XXX - Set the class name shown in python code. By default struct name is used as a class name.
• freelist=XXX - freelist parameter add support of free allocation list to custom class. The performance improvement applies to types that are often created and deleted in a row, so that they can benefit from a freelist. XXX is a number of items for free list.
• gc - Classes with the gc parameter participate in python garbage collector. If a custom class contains references to other python object that can be collected, the PyGCProtocol trait has to be implemented.
• weakref - adds support for python weak references
• extends=BaseType - use a custom base class. The base BaseType must implement PyTypeInfo.
• subclass - Allows Python classes to inherit from this class
• dict - adds __dict__ support, the instances of this type have a dictionary containing instance variables. (Incomplete, see #123)

By default it is not possible to create an instance of a custom class from python code. To declare a constructor, you need to define a class method and annotate it with #[new] attribute. Only the python __new__ method can be specified, __init__ is not available.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]

extern crate pyo3;
use pyo3::prelude::*;


#[pyclass]
struct MyClass {
   num: i32,
   token: PyToken,
}

#[pymethods]
impl MyClass {

     #[new]
     fn __new__(obj: &PyRawObject, num: i32) -> PyResult<()> {
         obj.init(|token| {
             MyClass {
                 num,
                 token
             }
         })
     }
}
}

Rules for the new method:

• If no method marked with #[new] is declared, object instances can only be created from Rust, but not from Python.
• The first parameter is the raw object and the custom new method must initialize the object with an instance of the struct using init method. The type of the object may be the type object of a derived class declared in Python.
• The first parameter implicitly has type &PyRawObject.
• For details on parameter-list, see the documentation of Method arguments section.
• The return type must be PyResult<T> for some T that implements IntoPyObject. Usually, T will be MyType.

By default PyObject is used as default base class. To override default base class base parameter for class needs to be used. Value is full path to base class. __new__ method accepts PyRawObject object. obj instance must be initialized with value of custom class struct. Subclass must call parent's __new__ method.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;

#[pyclass]
struct BaseClass {
   val1: usize,
   token: PyToken,
}

#[pymethods]
impl BaseClass {
   #[new]
   fn __new__(obj: &PyRawObject) -> PyResult<()> {
       obj.init(|token| BaseClass{val1: 10, token})
   }

   pub fn method(&self) -> PyResult<()> {
      Ok(())
   }
}

#[pyclass(extends=BaseClass)]
struct SubClass {
   val2: usize,
   token: PyToken,
}

#[pymethods]
impl SubClass {
   #[new]
   fn __new__(obj: &PyRawObject) -> PyResult<()> {
       obj.init(|token| SubClass{val2: 10, token});
       BaseClass::__new__(obj)
   }

   fn method2(&self) -> PyResult<()> {
       self.get_base().method()
   }
}
}

ObjectProtocol trait provides get_base() method. It returns reference to instance of base class.

Descriptor methods can be defined in #[pymethods] impl block only and has to be annotated with #[getter] or [setter] attributes. i.e.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   token: PyToken,
}

#[pymethods]
impl MyClass {

     #[getter]
     fn num(&self) -> PyResult<i32> {
        Ok(self.num)
     }
}
}

Getter or setter function's name is used as property name by default. There are several ways how to override name.

If function name starts with get_ or set_ for getter or setter respectively. Descriptor name becomes function name with prefix removed. This is useful in case os rust's special keywords like type.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   token: PyToken,
}

#[pymethods]
impl MyClass {

     #[getter]
     fn get_num(&self) -> PyResult<i32> {
        Ok(self.num)
     }

     #[setter]
     fn set_num(&mut self, value: i32) -> PyResult<()> {
        self.num = value;
        Ok(())
     }
}
}

In this case property num is defined. And it is available from python code as self.num.

Also both #[getter] and #[setter] attributes accepts one parameter. If parameter is specified, it is used and property name. i.e.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   token: PyToken,
}

#[pymethods]
impl MyClass {

     #[getter(number)]
     fn num(&self) -> PyResult<i32> {
        Ok(self.num)
     }

     #[setter(number)]
     fn set_num(&mut self, value: i32) -> PyResult<()> {
        self.num = value;
        Ok(())
     }
}
}

In this case property number is defined. And it is available from python code as self.number.

For simple cases you can also define getters and setters in your Rust struct field definition, for example:

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
  #[prop(get, set)]
  num: i32
}
}

Then it is available from Python code as self.num.

To define python compatible method, impl block for struct has to be annotated with #[pymethods] attribute. pyo3 library generates python compatible wrappers for all functions in this block with some variations, like descriptors, class method static methods, etc.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   token: PyToken,
}

#[pymethods]
impl MyClass {

     fn method1(&self) -> PyResult<i32> {
        Ok(10)
     }

     fn set_method(&mut self, value: i32) -> PyResult<()> {
        self.num = value;
        Ok(())
     }
}
}

Calls to this methods protected by GIL, &self or &mut self can be used. The return type must be PyResult<T> for some T that implements IntoPyObject.

Python parameter can be specified as part of method signature, in this case py argument get injected by method wrapper. i.e

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   debug: bool,
   token: PyToken,
}

#[pymethods]
impl MyClass {
     fn method2(&self, py: Python) -> PyResult<i32> {
        Ok(10)
     }
}
}

From python perspective method2, in above example, does not accept any arguments.

To specify class method for custom class, method needs to be annotated with#[classmethod] attribute.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   debug: bool,
   token: PyToken,
}

#[pymethods]
impl MyClass {
     #[classmethod]
     fn cls_method(cls: &PyType) -> PyResult<i32> {
        Ok(10)
     }
}
}

Declares a class method callable from Python.

• The first parameter is the type object of the class on which the method is called. This may be the type object of a derived class.
• The first parameter implicitly has type &PyType.
• For details on parameter-list, see the documentation of Method arguments section.
• The return type must be PyResult<T> for some T that implements IntoPyObject.

To specify class method for custom class, method needs to be annotated with #[staticmethod] attribute. The return type must be PyResult<T> for some T that implements IntoPyObject.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   debug: bool,
   token: PyToken,
}

#[pymethods]
impl MyClass {
     #[staticmethod]
     fn static_method(param1: i32, param2: &str) -> PyResult<i32> {
        Ok(10)
     }
}
}

To specify custom __call__ method for custom class, call method needs to be annotated with #[call] attribute. Arguments of the method are specified same as for instance method.

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
#[pyclass]
struct MyClass {
   num: i32,
   debug: bool,
   token: PyToken,
}

#[pymethods]
impl MyClass {
     #[call]
     #[args(args="*")]
     fn __call__(&self, args: &PyTuple) -> PyResult<i32> {
        println!("MyCLS has been called");
        Ok(self.num)
     }
}
}

By default pyo3 library uses function signature to determine which arguments are required. Then it scans incoming args parameter and then incoming kwargs parameter. If it can not find all required parameters, it raises TypeError exception. It is possible to override default behavior with #[args(...)] attribute. args attribute accept comma separated list of parameters in form attr_name="default value". Each parameter has to match method parameter by name.

Each parameter could one of following type:

• "*": var arguments separator, each parameter defined after "*" is keyword only parameters. corresponds to python's def meth(*, arg1.., arg2=..)
• args="*": "args" is var args, corresponds to python's def meth(*args). Type of args parameter has to be &PyTuple.
• kwargs="**": "kwargs" is keyword arguments, corresponds to python's def meth(**kwargs). Type of kwargs parameter has to be Option<&PyDict>.
• arg="Value": arguments with default value. corresponds to python's def meth(arg=Value). if arg argument is defined after var arguments it is treated as keyword argument. Note that Value has to be valid rust code, pyo3 just inserts it into generated code unmodified.

Example:

#![allow(unused_variables)]
fn main() {
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;

#[pyclass]
struct MyClass {
   num: i32,
   debug: bool,
   token: PyToken,
}

#[pymethods]
impl MyClass {
    #[args(arg1=true, args="*", arg2=10, kwargs="**")]
    fn method(&self, arg1: bool, args: &PyTuple, arg2: i32, kwargs: Option<&PyDict>) -> PyResult<i32> {
        Ok(1)
    }
}
}

Python object model defines several protocols for different object behavior, like sequence, mapping or number protocols. pyo3 library defines separate trait for each of them. To provide specific python object behavior you need to implement specific trait for your struct. Important note, each protocol implementation block has to be annotated with #[pyproto] attribute.

PyObjectProtocol trait provide several basic customizations.

To customize object attribute access define following methods:

• fn __getattr__(&self, name: FromPyObject) -> PyResult<impl IntoPyObject>
• fn __setattr__(&mut self, name: FromPyObject, value: FromPyObject) -> PyResult<()>
• fn __delattr__(&mut self, name: FromPyObject) -> PyResult<()>

Each methods corresponds to python's self.attr, self.attr = value and del self.attr code.

• fn __repr__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>

• fn __str__(&self) -> PyResult<impl ToPyObject<ObjectType=PyString>>

  Possible return types for __str__ and __repr__ are PyResult<String> or PyResult<PyString>. In Python 2.7, Unicode strings returned by __str__ and __repr__ will be converted to byte strings by the Python runtime, which results in an exception if the string contains non-ASCII characters.

• fn __bytes__(&self) -> PyResult<PyBytes>

  On Python 3.x, provides the conversion to bytes. On Python 2.7, __bytes__ is allowed but has no effect.

• fn __unicode__(&self) -> PyResult<PyUnicode>

  On Python 2.7, provides the conversion to unicode. On Python 3.x, __unicode__ is allowed but has no effect.

• fn __format__(&self, format_spec: &str) -> PyResult<impl ToPyObject<ObjectType=PyString>>

  Special method that is used by the format() builtin and the str.format() method. Possible return types are PyResult<String> or PyResult<PyString>.

• fn __richcmp__(&self, other: impl FromPyObject, op: CompareOp) -> PyResult<impl ToPyObject>

  Overloads Python comparison operations (==, !=, <, <=, >, and >=). The op argument indicates the comparison operation being performed. The return type will normally be PyResult<bool>, but any Python object can be returned. If other is not of the type specified in the signature, the generated code will automatically return NotImplemented.

• fn __hash__(&self) -> PyResult<impl PrimInt>

  Objects that compare equal must have the same hash value. The return type must be PyResult<T> where T is one of Rust's primitive integer types.

• fn __bool__(&self) -> PyResult<bool>

  Determines the "truthyness" of the object. This method works for both python 3 and python 2, even on Python 2.7 where the Python spelling was __nonzero__.

If your type owns references to other python objects, you will need to integrate with Python's garbage collector so that the GC is aware of those references. To do this, implement PyGCProtocol trait for your struct. It includes two methods __traverse__ and __clear__. These correspond to the slots tp_traverse and tp_clear in the Python C API. __traverse__ must call visit.call() for each reference to another python object. __clear__ must clear out any mutable references to other python objects (thus breaking reference cycles). Immutable references do not have to be cleared, as every cycle must contain at least one mutable reference. Example:

#![allow(unused_variables)]
#![feature(specialization)]
fn main() {
extern crate pyo3;

use pyo3::prelude::*;


#[pyclass]
struct ClassWithGCSupport {
    obj: Option<PyObject>,
    token: PyToken,
}

#[pyproto]
impl PyGCProtocol for ClassWithGCSupport {
    fn __traverse__(&self, visit: PyVisit) -> Result<(), PyTraverseError> {
        if let Some(ref obj) = self.obj {
            visit.call(obj)?
        }
        Ok(())
    }

    fn __clear__(&mut self) {
        if let Some(obj) = self.obj.take() {
          // Release reference, this decrements ref counter.
          self.py().release(obj);
        }
    }
}
}

Special protocol trait implementation has to be annotated with #[pyproto] attribute.

It is also possible to enable gc for custom class using gc parameter for class annotation. i.e. #[pyclass(gc)]. In that case instances of custom class participate in python garbage collector, and it is possible to track them with gc module methods.

Iterators can be defined using the PyIterProtocol trait. It includes two methods __iter__ and __next__:

• fn __iter__(&mut self) -> PyResult<impl IntoPyObject>
• fn __next__(&mut self) -> PyResult<Option<impl IntoPyObject>>

Returning Ok(None) from __next__ indicates that that there are no further items.

Example:

#![allow(unused_variables)]
#![feature(specialization)]
fn main() {
extern crate pyo3;

use pyo3::prelude::*;


#[pyclass]
struct MyIterator {
    iter: Box<Iterator<Item=PyObject> + Send>,
    token: PyToken,
}

#[pyproto]
impl PyIterProtocol for MyIterator {

    fn __iter__(&mut self) -> PyResult<PyObject> {
        Ok(self.into())
    }
    fn __next__(&mut self) -> PyResult<Option<PyObject>> {
        Ok(self.iter.next())
    }
}
}

CPython has an infamous GIL(Global Interpreter Lock) prevents developers getting true parallelism. With pyo3 you can release GIL when executing Rust code to achieve true parallelism.

The Python::allow_threads method temporarily releases the GIL, thus allowing other Python threads to run.

impl Python {
    pub fn allow_threads<T, F>(self, f: F) -> T where F: Send + FnOnce() -> T {}
}

Let's take a look at our word-count example, we have a wc_parallel function utilize the rayon crate to count words in parallel.

fn wc_parallel(lines: &str, search: &str) -> i32 {
    lines.par_lines()
         .map(|line| wc_line(line, search))
         .sum()
}

Then in the Python bridge, we have a function search exposed to Python runtime which calls wc_parallel inside Python::allow_threads method to enable true parallelism:

#[pymodinit]
fn word_count(py: Python, m: &PyModule) -> PyResult<()> {

    #[pyfn(m, "search")]
    fn search(py: Python, path: String, search: String) -> PyResult<i32> {
        let mut file = File::open(path)?;
        let mut contents = String::new();
        file.read_to_string(&mut contents)?;

        let count = py.allow_threads(move || wc_parallel(&contents, &search));
        Ok(count)
    }

    Ok(())
}

Let's benchmark the word-count example to verify that we did unlock true parallelism with pyo3. We are using pytest-benchmark to benchmark three word count functions:

1. Pure Python version
2. Rust sequential version
3. Rust parallel version

Benchmark script can be found here, then we can run pytest tests to benchmark them.

On MacBook Pro (Retina, 15-inch, Mid 2015) the benchmark gives:

Pyo3's attributes, #[pyclass], #[pymodinit], etc. are procedural macros, which means that rewrite the source of the annotated item. You can view the generated source with the following command, which also expands a few other things:

cargo rustc --profile=check -- -Z unstable-options --pretty=expanded > expanded.rs; rustfmt expanded.rs

(You might need to install rustfmt if you don't already have it.)

You can also debug classic !-macros by adding -Z trace-macros`:

cargo rustc --profile=check -- -Z unstable-options --pretty=expanded -Z trace-macros > expanded.rs; rustfmt expanded.rs

See cargo expand for a more elaborate version of those commands.

setuptools-rust is a setuptools helpers for Rust Python extensions. It supports PyO3 by default.

For detailed usage, please refer to its README

To build a source code distribution, you need to add the following lines to your MANIFEST.in file to ensure it correctly packages Rust extension source code.

include Cargo.toml
recursive-include src *

Then you can build a source code distribution by (assuming you have already written a setup.py):

python setup.py sdist

To build a binary wheel, manylinux would be a natural choice for Linux.

Take the example project in setuptools-rust repository for example, we have a build-wheels.sh to be used with Docker to build manylinux1 wheels. First you need to pull the manylinux1 Docker image:

$ docker pull quay.io/pypa/manylinux1_x86_64

Then use the following command to build wheels for supported Python versions:

$ docker run --rm -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/build-wheels.sh

You will find all the wheels in dist directory:

$ ls dist
hello_rust-1.0-cp27-cp27m-linux_x86_64.whl       hello_rust-1.0-cp35-cp35m-linux_x86_64.whl
hello_rust-1.0-cp27-cp27m-manylinux1_x86_64.whl  hello_rust-1.0-cp35-cp35m-manylinux1_x86_64.whl
hello_rust-1.0-cp27-cp27mu-linux_x86_64.whl      hello_rust-1.0-cp36-cp36m-linux_x86_64.whl
hello_rust-1.0-cp27-cp27mu-manylinux1_x86_64.whl hello_rust-1.0-cp36-cp36m-manylinux1_x86_64.whl

The *-manylinux1_x86_64.whl files are the manylinux1 wheels that you can upload to PyPI.

Pyo3 began as fork of rust-cpython when rust-cpython wasn't maintained. Over the time pyo3 has become fundamentally different from rust-cpython.

This chapter is based on the discussion in PyO3/pyo3#55.

While rust-cpython has a macro based dsl for declaring modules and classes, pyo3 use proc macros and spezialization. Pyo3 also doesn't change your struct and functions so you can still use them as normal rust functions. The disadvantage is that proc macros and spezialization currently only work on nightly.

rust-cpython

py_class!(class MyClass |py| {
    data number: i32;
    def __new__(_cls, arg: i32) -> PyResult<MyClass> {
        MyClass::create_instance(py, arg)
    }
    def half(&self) -> PyResult<i32> {
        Ok(self.number(py) / 2)
    }
});

pyo3

#![allow(unused_variables)]
#![feature(specialization)]

fn main() {
extern crate pyo3;

use pyo3::prelude::*;


#[pyclass]
struct MyClass {
   num: u32,
}

#[pymethods]
impl MyClass {
    #[new]
    fn __new__(obj: &PyRawObject, num: u32) -> PyResult<()> {
        obj.init(|token| {
            MyClass {
                num,
            }
        })
    }

    fn half(&self) -> PyResult<u32> {
        Ok(self.num / 2)
    }
}
}

All objects are owned by pyo3 library and all apis available with references, while in rust-cpython, you own python objects.

Here is example of PyList api:

rust-cpython

impl PyList {

   fn new(py: Python) -> PyList {...}

   fn get_item(&self, py: Python, index: isize) -> PyObject {...}
}

pyo3

impl PyList {

   fn new(py: Python) -> &PyList {...}

   fn get_item(&self, index: isize) -> &PyObject {...}
}

Because pyo3 allows only references to python object, all reference have the Gil lifetime. So the python object is not required, and it is safe to have functions like fn py<'p>(&'p self) -> Python<'p> {}.

rust-cpython requires a Python parameter for PyErr, so error handling ergonomics is pretty bad. It is not possible to use ? with rust errors.

pyo3 on other hand does not require Python for PyErr, it is only required if you want to raise an exception in python with the PyErr::restore() method. Due to the std::convert::From<Err> for PyErr trait ? is supported automatically.