Python 3.5 introduced the new typing module that provides standard library support for leveraging function annotations for optional type hints. That opens the door to new and interesting tools for static type checking like mypy and in the future possibly automated type-based optimization. Type hints are specified in PEP-483 and PEP-484.
In this tutorial I explore the possibilities that type hints present and show you how to use mypy to statically analyze your Python programs and significantly improve the quality of your code.
Type Hints
Type hints are built on top of function annotations. Briefly, function annotations let you annotate the arguments and return value of a function or method with arbitrary metadata. Type hints are a special case of function annotations that specifically annotate function arguments and the return value with standard type information. Function annotations in general and type hints in particular are totally optional. Let’s take a look at a quick example:
```python def reverse_slice(text: str, start: int, end: int) -> str: return text[start:end][::-1]
reverse_slice(‘abcdef’, 3, 5) ‘ed’ ```
The arguments were annotated with their type as well as the return value. But it’s critical to realize that Python ignores this completely. It makes the type information available through the annotations attribute of the function object, but that’s about it.
python
reverse_slice.__annotations
{'end': int, 'return': str, 'start': int, 'text': str}
To verify that Python really ignores the type hints, let’s totally mess up the type hints:
```python def reverse_slice(text: float, start: str, end: bool) -> dict: return text[start:end][::-1]
reverse_slice(‘abcdef’, 3, 5) ‘ed’ ```
As you can see, the code behaves the same, regardless of the type hints.
Motivation for Type Hints
OK. Type hints are optional. Type hints are totally ignored by Python. What’s the point of them, then? Well, there are several good reasons:
- static analysis
- IDE support
- standard documentation
I’ll dive into static analysis with Mypy later. IDE support already started with PyCharm 5’s support for type hints. Standard documentation is great for developers that can easily figure out the type of arguments and return value just by looking at a function signature as well as automated documentation generators that can extract the type information from the hints.
The
typing Module
The typing module contains types designed to support type hints. Why not just use existing Python types like int, str, list and dict? You can definitely use these types, but due to Python’s dynamic typing, beyond basic types you don’t get a lot of information. For example, if you want to specify that an argument may be a mapping between a string and an integer, there is no way to do it with standard Python types. With the typing module, it’s as easy as:
python
Mapping[str, int]
Let’s look at a more complete example: a function that takes two arguments. One of them is a list of dictionaries where each dictionary contains keys that are strings and values that are integers. The other argument is either a string or an integer. The typing module allows precise specifications of such complicated arguments.
```python from typing import List, Dict, Union
def foo(a: List[Dict[str, int]], b: Union[str, int]) -> int: “"”Print a list of dictionaries and return the number of dictionaries “”” if isinstance(b, str): b = int(b) for i in range(b): print(a)
x = [dict(a=1, b=2), dict(c=3, d=4)] foo(x, ‘3’)
[{‘b’: 2, ‘a’: 1}, {‘d’: 4, ‘c’: 3}] [{‘b’: 2, ‘a’: 1}, {‘d’: 4, ‘c’: 3}] [{‘b’: 2, ‘a’: 1}, {‘d’: 4, ‘c’: 3}] ```
Useful Types
Let’s see some of the more interesting types from the typing module.
The Callable type allows you to specify the function which may be passed as arguments or returned as a result, since Python treats functions as first-class citizens. The syntax for callables is to provide an array of argument types (again from the typing module) followed by a return value. If that’s confusing, here is an example:
```python def do_something_fancy(data: Set[float], on_error: Callable[[Exception, int], None]): …
```
The on_error callback function is specified as a function that takes an Exception and an integer as arguments and returns nothing.
The Any type means that a static type checker should allow any operation as well as assignment to any other type. Every type is a subtype of Any.
The Union type you saw earlier is useful when an argument can have multiple types, which is very common in Python. In the following example, the verify_config() function accepts a config argument, which can be either a Config object or a file name. If it’s a file name, it calls another function to parse the file into a Config object and return it.
```python def verify_config(config: Union[str, Config]): if isinstance(config, str): config = parse_config_file(config) …
def parse_config_file(filename: str) -> Config: …
```
The Optional type means the argument may be None too.
Optional[T] is equivalent to
Union[T, None]
There are many more types that denote various capabilities such as Iterable, Iterator, Reversible, SupportsInt, SupportsFloat, Sequence, MutableSequence and IO. Check out the typing module documentation for the full list.
The main thing is that you can specify the type of arguments in a very fine-grained way that supports the Python type system at a high fidelity and allows generics and abstract base classes too.
Forward References
Sometimes you want to refer to a class in a type hint within one of its methods. For example, let’s assume that class A can perform some merge operation that takes another instance of A, merges with itself and returns the result. Here is a naive attempt to use type hints to specify it:
```python class A: def merge(other: A) -> A: …
1 class A: ----> 2 def merge(other: A = None) -> A: 3 ... 4
NameError: name ‘A’ is not defined ```
What happened? The class A is not defined yet when the type hint for its merge() method is checked by Python, so the class A can’t be used at this point (directly). The solution is quite simple, and I’ve seen it used before by SQLAlchemy. You just specify the type hint as a string. Python will understand that it is a forward reference and will do the right thing:
python
class A:
def merge(other: 'A' = None) -> 'A':
...
Type Aliases
One downside of using type hints for long type specifications is that it can clutter the code and make it less readable, even if it provides a lot of type information. You can alias types just like any other object. It’s as simple as:
```python Data = Dict[int, Sequence[Dict[str, Optional[List[float]]]]
def foo(data: Data) -> bool: … ```
The
get_type_hints() Helper Function
The typing module provides the get_type_hints() function, which provides information about the argument types and the return value. While the annotations attribute returns type hints because they are just annotations, I still recommend that you use the get_type_hints() function because it resolves forward references. Also, if you specify a default of None to one of the arguments, the get_type_hints() function will automatically return its type as Union[T, NoneType] if you just specified T. Let’s see the difference using the A.merge() method defined earlier:
```python print(A.merge.annotations)
{‘other’: ‘A’, ‘return’: ‘A’} ```
The annotations attribute simply returns the annotation value as is. In this case it’s just the string ‘A’ and not the A class object, to which ‘A’ is just a forward reference.
```python print(get_type_hints(A.merge))
{‘return’: <class ‘main.A’>, ‘other’: typing.Union[main.A, NoneType]} ```
The get_type_hints() function converted the type of the other argument to a Union of A (the class) and NoneType because of the None default argument. The return type was also converted to the class A.
The Decorators
Type hints are a specialization of function annotations, and they can also work side by side with other function annotation.
In order to do that, the typing module provides two decorators: @no_type_check and @no_type_check_decorator. The @no_type_check decorator can be applied to either a class or a function. It adds the no_type_check attribute to the function (or each method of the class). This way, type checkers will know to ignore annotations, which are not type hints.
It is a little cumbersome because if you write a library that will be used broadly, you must assume that a type checker will be used, and if you want to annotate your functions with non-type hints, you must also decorate them with @no_type_check.
A common scenario when using regular function annotations is also to have a decorator that operates over them. You also want to turn off type checking in this case. One option is to use the @no_type_check decorator in addition to your decorator, but that gets old. Instead, the @no_Type_check_decorator can be used to decorate your decorator so that it also behaves like @no_type_check (adds the no_type_check attribute).
Let me illustrate all these concepts. If you try to get_type_hint() (as any type checker will do) on a function that is annotated with a regular string annotation, the get_type_hints() will interpret it as a forward reference:
```python def f(a: ‘some annotation’): pass
print(get_type_hints(f))
SyntaxError: ForwardRef must be an expression – got ‘some annotation’ ```
To avoid it, add the @no_type_check decorator, and get_type_hints simply returns an empty dict, while the __annotations__ attribute returns the annotations:
```python @no_type_check def f(a: ‘some annotation’): pass
print(get_type_hints(f)) {}
print(f.annotations) {‘a’: ‘some annotation’} ```
Now, suppose we have a decorator that prints the annotations dict. You can decorate it with the @no_Type_check_decorator and then decorate the function and not worry about some type checker calling get_type_hints() and getting confused. This is probably a best practice for every decorator that operates on annotations. Don’t forget the @functools.wraps, otherwise the annotations will not be copied to the decorated function and everything will fall apart. This is covered in detail in Python 3 Function Annotations.
python
@no_type_check_decorator
def print_annotations(f):
@functools.wraps(f)
def decorated(*args, **kwargs):
print(f.__annotations__)
return f(*args, **kwargs)
return decorated
Now, you can decorate the function just with @print_annotations, and whenever it is called it will print its annotations.
```python @print_annotations def f(a: ‘some annotation’): pass
f(4) {‘a’: ‘some annotation’} ```
Calling get_type_hints() is also safe and returns an empty dict.
python
print(get_type_hints(f))
{}
Static Analysis With Mypy
Mypy is a static type checker that was the inspiration for type hints and the typing module. Guido van Rossum himself is the author of PEP-483 and a co-author of PEP-484.
Installing Mypy
Mypy is in very active development, and as of this writing the package on PyPI is out of date and doesn’t work with Python 3.5. To use Mypy with Python 3.5, get the latest from Mypy’s repository on GitHub. It’s as simple as:
bash
pip3 install git+git://github.com/JukkaL/mypy.git
Playing With Mypy
Once you have Mypy installed, you can just run Mypy on your programs. The following program defines a function that expects a list of strings. It then invokes the function with a list of integers.
```python from typing import List
def case_insensitive_dedupe(data: List[str]): “"”Converts all values to lowercase and removes duplicates””” return list(set(x.lower() for x in data))
print(case_insensitive_dedupe([1, 2])) ```
When running the program, it obviously fails at runtime with the following error:
plain
python3 dedupe.py
Traceback (most recent call last):
File "dedupe.py", line 8, in <module>
print(case_insensitive_dedupe([1, 2, 3]))
File "dedupe.py", line 5, in case_insensitive_dedupe
return list(set(x.lower() for x in data))
File "dedupe.py", line 5, in <genexpr>
return list(set(x.lower() for x in data))
AttributeError: 'int' object has no attribute 'lower'
What’s the problem with that? The problem is that it’s not clear immediately even in this very simple case what the root cause is. Is it an input type problem? Or maybe the code itself is wrong and shouldn’t try to call the lower() method on the ‘int’ object. Another issue is that if you don’t have 100% test coverage (and, let’s be honest, none of us do), then such issues can lurk in some untested, rarely used code path and be detected at the worst time in production.
Static typing, aided by type hints, gives you an extra safety net by making sure you always call your functions (annotated with type hints) with the right types. Here is the output of Mypy:
plain
(N) > mypy dedupe.py
dedupe.py:8: error: List item 0 has incompatible type "int"
dedupe.py:8: error: List item 1 has incompatible type "int"
dedupe.py:8: error: List item 2 has incompatible type "int"
This is straightforward, points directly to the problem, and doesn’t require running a lot of tests. Another benefit of static type checking is that if you commit to it, you can skip dynamic type checking except when parsing external input (reading files, incoming network requests or user input). It also builds a lot of confidence as far as refactoring goes.
Conclusion
Type hints and the typing module are totally optional additions to the expressiveness of Python. While they may not suit everyone’s taste, for large projects and large teams they can be indispensable. The evidence is that large teams already use static type checking. Now that type information is standardized, it will be easier to share code, utilities and tools that use it. IDEs like PyCharm already take advantage of it to provide a better developer experience.
Envato Tuts+ tutorials are translated into other languages by our community members—you can be involved too!Translate this post
