Using std::variant in interfaces with Boost.Python
Wrapping function templates (or overload sets) into Python can be a real challenge. Python doesn’t let you define functions with the same name but different argument types like C++ does.
Consider this pair of functions that calculate the magnitude of a scalar or complex number:
double mag(double v)
{
return std::fabs(v); // from <cmath>
}
double mag(std::complex<double> v)
{
return std::abs(v); // the overload in <complex>
}
In C++ we use the same name, mag, to refer to either of these and the compiler selects the right one. We’d like it to work the same way when bound into Python:
from cppmodule import mag
if __name__ == "__main__":
print(mag(3+4j)) # complex
print(mag(-3.14)) # scalar double
which should print 5.0 and 3.14.
Two Common Approaches
In my experience looking at Python libraries I’ve seen two popular strategies that give this result:
Use value wrapper, bind one function, dispatch in C++
In this approach we use some kind of type-erased wrapper, like std::any, as a container type for whatever was actually supplied. The bound function then does type checking on the contained value and dispatches to the correct concrete function. For example:
double mag(std::any v)
{
// dispatch in C++ from this single function
auto * d = std::any_cast<double>(&v); // nullptr if not double
if (d)
{
return std::fabs(*d);
} else {
auto c = std::any_cast<std::complex<double>>(v);
return std::abs(c);
}
}
Here I’ve also let Boost.Python know that our argument types are convertible to std::any:
BOOST_PYTHON_MODULE(strategy1) {
using namespace boost::python;
implicitly_convertible<double, std::any>();
implicitly_convertible<std::complex<double>, std::any>();
def("mag", mag);
}
Bind N functions, dispatch in Python
In this alternative we register all the function variations, then let a Python wrapper function dispatch according to what it receives:
from cppmodule import magScalar, magComplex
def mag(v):
# dispatch from Python to per-type C++ functions
if type(v) is complex:
return magComplex(v)
else:
return magScalar(v)
I find both approaches unsatisfying. Iterating over types at runtime is inefficient and error-prone. Plus, we already have a way to express “one of these N types” in C++ : std::variant.
Using std::variant with Python
My goal is to directly express the types my functions can accept directly in the interface, and bind each function just once. Then I can flexibly and efficiently dispatch with std::visit. For example, using the overload set from the first alternative:
double magImpl(std::variant<double, std::complex<double>> var)
{
// compare vs. std::any_cast dispatcher, above
return std::visit([](auto const & v) -> double
{
return mag(v); // don't need to list out each type
},
var);
}
The visitor gets instantiated once per member type so I could also consolidate the overloads with if constexpr :
double mag(std::variant<double, std::complex<double>> var)
{
return std::visit([](auto const & v) -> double
{
if constexpr (std::is_same_v<decltype(v), double>)
return std::fabs(v); // <cmath>
else
return std::abs(v); // <complex>
},
var);
}
Implementation
Goals
With a minimum of boilerplate:
std::variant<X, Y, Z>parameters, when bound into Python, accept any ofX, Y, Zif they are themselves convertible.- Supplying an argument not convertible into one of the member types throws an exception
std::monostate, if a member type, allows the Python valueNoneto be suppliedstd::reference_wrappershould be supported without copying the underlying type- Return values should be supported similarly
Details
Using Boost.Python to make this happen was not too painful, though I had to explore the code to understand how to do it. We need to register special conversion functions for:
- Constructing a variant from one of its alternative types (for arguments)
- Producing Python objects from a variant (for return values)
The good news is I can assume both functions exist for each variant’s constituent types, so a kind of “recursive” implementation follows.
Variant Arguments
The thing that almost works is to notify Boost.Python that the member types are implicitly convertible to the variant itself. Something like this:
// assuming we have std::variant<T...> here:
mp11::mp_for_each< // a little help from Boost.MP11
mp11::mp_list<T...>>(
[](auto t){
using arg_t = std::decay_t<decltype(t)>;
// take advantage of built-in conversions
python::implicitly_convertible<val_t, var_t>();
});
This breaks down in two unfortunate ways:
- Some alternative types may not be default-constructible.
- The implicit conversion process is more lax about numbers than I would like, and picks the first match in the variant, so you get e.g.
intinstead ofboolwhen you try to storeTrueinto astd::variant<int, bool>.
The solution to the first problem is just to iterate over pointer types instead. The second required implementing my own “converter” for numeric types, but it wasn’t bad.
Reference Wrappers
Supporting std::reference_wrapper<T> just required detecting and using the underlying type instead.
Variant Return Types
This one’s quite straightforward - you just dispatch on the stored type with std::visit and convert it naturally:
static PyObject* convert(std::variant<T...> const & var)
{
// convert variant by recursively visiting stored types T...
return std::visit(
[](auto const & v){
// convert this type
return boost::python::to_python_value<decltype(v)>()(v);
},
var);
}
The Code
A repo demonstrating this approach, along with all the examples in this post, can be found here. The key implementation code is here
A More Interesting Example
Real applications might have multiple parameters, each of varying types, and produce variant results as well. Here’s a relatively simple one:
Generalized Adding
Imagine a wrapped C++ function adder that accepts two arguments, each of either integer or string type, and returns:
- The concatenation of the two inputs, if either input is a string
- The sum of the two inputs, if both are integers
Usage might look like this:
>>> print(adder("the answer is ", 42)) # string and int
the answer is 42
>>> print(adder("a monoid", " in the category of endofunctors")) # two strings
a monoid in the category of endofunctors
>>> print(adder(1, 2)) # two ints
3
>>> print(type(adder(1, 2))) # not a string this time
<class 'int'>
>>> print(adder(2, 1.14)) # should fail with type error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
blog_example.adder(int, float)
did not match C++ signature:
...
And the C++ implementation could be:
using var_t = std::variant<std::string, int>;
var_t adder(var_t const& var_a, var_t const& var_b)
{
return std::visit(
[](auto const & a, auto const & b) -> var_t
{
// return integer sum if both are ints, otherwise a string append
if constexpr (std::is_same_v<decltype(a), decltype(b)>)
{
return a + b; // works for both types
} else {
if constexpr (std::is_same_v<std::decay_t<decltype(a)>,
std::string>)
return a + std::to_string(b);
else
return std::to_string(a) + b;
}
},
var_a,
var_b);
}
Conclusion
std::variant is a natural way to express “one of these types” and is a good fit for wrapped Python function parameters. Clean, efficient dispatching is built right into C++17 with std::visit, and type checking happens automatically.