ParamSpec (PEP612) was released in Python3.10 and
there was a big problem in typing decorators that change a function’s signature.
Let’s start with a basic example. How one can type a decorator function that does not change anything?
from typing import Callable, TypeVar C = TypeVar('C', bound=Callable) def logger(function: C) -> C: def decorator(*args, **kwargs): print('Function called!') return function(*args, **kwargs) return decorator
Notice the most important part here:
C = TypeVar('C', bound=Callable)
What does it mean? It means that we take any callable in and return the exact same callable.
This allows you to decorate any function and preserve its signature:
@logger def example(arg: int, other: str) -> tuple[int, str]: return arg, other reveal_type(example) # (arg: int, other: str) -> tuple[int, str]
But, there’s a problem when a function does want to change something.
Imagine, that some decorator might also add
None as a return value in some cases:
def catch_exception(function): def decorator(*args, **kwargs): try: return function(*args, **kwargs) except Exception: return None return decorator
This is a perfectly valid Python code.
But how can we type it? Note that we cannot use
TypeVar('C', bound=Callable) anymore, since we are changing the return type now.
Initially, I’ve tried something like:
def catch_exception(function: Callable[..., T]) -> Callable[..., Optional[T]]: ...
But, this means a different thing: it turns all function’s arguments into
*args: Any, **kwargs: Any, but, the return type will be correct. Generally, this is not what we need when it comes to type-safety.
The second way to do that in a type-safe way is adding a custom Mypy plugin.
Here’s our example from
dry-python/returns to support decorators that were changing return types. But, plugins are quite hard to write (you need to learn a bit of Mypy’s API), they are not universal (for example, Pyright does not understand Mypy plugins), and they require to be explicitly installed by the end user.
ParamSpec was added. Here’s how it can be used in this case:
from typing import Callable, TypeVar, Optional from typing_extensions import ParamSpec # or `typing` for `python>=3.10` T = TypeVar('T') P = ParamSpec('P') def catch_exception(function: Callable[P, T]) -> Callable[P, Optional[T]]: def decorator(*args: P.args, **kwargs: P.kwargs) -> Optional[T]: try: return function(*args, **kwargs) except Exception: return None return decorator
Now, all decorated functions will preserve their argument types and change their return type to include
@catch_exception def div(arg: int) -> float: return arg / arg reveal_type(div) # (arg: int) -> Optional[float] @catch_exception def plus(arg: int, other: int) -> int: return arg + other reveal_type(plus) # (arg: int, other: int) -> Optional[int]:
The recent release of Mypy 0.930 with
ParamSpec support allowed us to remove our custom Mypy plugin and use a well-defined primitive. Here’s a commit to show how easy our transition was. It was even released today in
email@example.com, check it out!
But, that’s not all! Because some decorators modify argument types, PEP612 also adds the
Concatenate type that allows prepending, appending, transforming, or removing function arguments.
Unfortunately, Mypy does not support
Concatenate just yet, but I can show you some examples from PEP itself. Here’s how it is going to work.
Let’s start with some basic definitions:
from typing_extensions import ParamSpec, Concatenate # or `typing` for `python>=3.10` P = ParamSpec('P') def bar(x: int, *args: bool) -> int: ...
We are going to change the type of
bar function with the help of
P parameter specification. First, let’s prepend an
str argument to this function:
def add(x: Callable[P, int]) -> Callable[Concatenate[str, P], int]: ... add(bar) # (str, /, x: int, *args: bool) -> int
Notice that a positional-only
str argument is added to the return type of
Now, let’s try removing an argument:
def remove(x: Callable[Concatenate[int, P], int]) -> Callable[P, int]: ... remove(bar) # (*args: bool) -> int
Because we use
Concatenate in the argument type, the return type will not have an
int argument anymore.
And finally, let’s change an argument type from
str and return type from
def transform( x: Callable[Concatenate[int, P], int] ) -> Callable[Concatenate[str, P], bool]: ... transform(bar) # (str, /, *args: bool) -> bool
Looking forward to new Mypy release with
Concatenate support. I totally know some places where it will be useful.
PEP612 adds two very powerful abstractions that allow us to better type our functions and decorators, which play a very important role in Python’s world.
Complex projects (like Django) or simple type-safe scripts can highly benefit from this new typing feature. And I hope you will!
Happy New Year!