Issues with mypy when using Transformers
### System Info

```shell
- `transformers` version: 4.19.2
- Platform: Linux-5.18.1-arch1-1-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Since some weeks there seem to be issues when using Transnsformers as a 3rs party lib
and mypy as a type checker.

Example:

```
hpoflow/optuna_transformers.py:83: error: Item "TFPreTrainedModel" of "Union[PreTrainedModel, TFPreTrainedModel]" has no attribute "config"
hpoflow/optuna_transformers.py:85: error: Item "PreTrainedModel" of "Union[PreTrainedModel, TFPreTrainedModel]" has no attribute "config"
hpoflow/optuna_transformers.py:85: error: Item "TFPreTrainedModel" of "Union[PreTrainedModel, TFPreTrainedModel]" has no attribute "config"
tests/test_optuna_transformers.py:135: error: "Trainer" has no attribute "train"
tests/test_optuna_transformers.py:139: error: "Trainer" has no attribute "state"
```

see https://github.com/telekom/HPOflow/runs/6664387914?check_suite_focus=true

It might be because of some lazy loading / import magic.
The Optuna project is doing some extra affords to avoid this. See here:

https://github.com/telekom/lazy-imports#usage--example-for-lazyimporter

```
# Direct imports for type-checking
if TYPE_CHECKING:
    from hpoflow.mlflow import (  # noqa: F401
        check_repo_is_dirty,
        normalize_mlflow_entry_name,
        normalize_mlflow_entry_names_in_dict,
    )
    from hpoflow.optuna import SignificanceRepeatedTrainingPruner  # noqa: F401
    from hpoflow.optuna_mlflow import OptunaMLflow  # noqa: F401
    from hpoflow.optuna_transformers import OptunaMLflowCallback  # noqa: F401
    from hpoflow.utils import func_no_exception_caller  # noqa: F401
else:
    sys.modules[__name__] = LazyImporter(
        __name__,
        globals()["__file__"],
        _import_structure,
        extra_objects={"__version__": __version__},
    )
```

### Expected behavior

```shell
see above
```

This is not a bug in Transformers, so let's remove that label please. MyPy is not part of the Python standard library and we never say anywhere we support it.

If you want to work on a PR that has some fixes that don't make the code unreadable, we'll be happy to have a look, but no one in the team is going to actively work on this.<|||||>Ok. Closing it then.