nn.Transformer
## 🚀 

Use Pytorch's own attention and transformer modules.

## Motivation

Pytorch now offers modules like [nn.MultiheadAttention](https://pytorch.org/docs/stable/nn.html?highlight=attention#torch.nn.MultiheadAttention) and [nn.Transformer](https://pytorch.org/docs/stable/nn.html#transformer-layers). It would be nice to use the official Pytorch implementations in `transformers` now that they are available.

## Additional context

There is an offical Pytorch [tutorial](https://pytorch.org/tutorials/beginner/transformer_tutorial.html) that shows how nn.Transformer can be used and customized.

These modules are only available in Pytorch 1.1 (`nn.MultiHeadAttention`) and 1.2 (`nn.Transformer`). Using them would mean that anyone with Pytorch 1.0 would have to update their own version.

Even though I am in favour of using as many built-ins as possible, I wonder whether it is not too early to do this. You will end up with a lot of pseudo-duplicate code: for those who are on 1.0 (no transformer-like support), 1.1 (only nn.*Attention), and 1.2 (full transformer). I don't know any statistics about people using `transformers` but I can imagine that many are still on PyTorch 1.0. <|||||>We have a small codebase on the side where we use `nn.Transformer` to build both a BERT-style and a GPT2-style model that are compatible with our pretrained weights, but we still think it's a bit too early to refactor/freeze the lib's internals. A lot of research is still going to focus on the models' internals so we don't want to overfit to the current architecture.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@julien-c @BramVanroy just curious, do you guys still think it’s too early to use nn.Transformer?<|||||>A friendly ping to the maintainers to implement the in-built modules.<|||||>That's not something we can do because it will break all existing checkpoints.