In case someone else stumbles upon this problem, this is the best I could do (suggested by the mypy devs on gitter):
[mypy]
# unrelated but typically required
ignore_missing_imports = True

[mypy-torch.*]
follow_imports = skip
follow_imports_for_stubs = True

Cheers,
Enrico