BERT with regression head cannot fit one datapoint
Hi, I am trying to use BERT for a token-level regression task (predict a continuous value for each token), and I'm having trouble getting my model to train. As a debugging strategy, I'm trying to get it to overfit one datapoint, which should be easy, but it's failing that also.

Here is a minimal reproducing source code. The model is BERT which feeds into a ``nn.Linear(768, 1)``. To keep things simple, I am feeding it a sequence of length 1, and training it to output 0.5.

```
import torch
import transformers

class RegressionModel(torch.nn.Module):
  def __init__(self):
    super(RegressionModel, self).__init__()

    self.bert = transformers.BertModel.from_pretrained('bert-base-uncased')
    self.linear = torch.nn.Linear(768, 1)

  def forward(self, X_ids):
    return self.linear(self.bert(X_ids).last_hidden_state)


model = RegressionModel().cuda()
model.train()

opt = torch.optim.Adam(model.parameters())
X_ids = torch.LongTensor([[12345]]).cuda()
Y_true = torch.Tensor([[0.5]]).cuda()

steps = 0
while True:
  opt.zero_grad()
  Y_pred = model(X_ids)
  loss = (Y_true - Y_pred)**2
  loss.backward()
  print(steps, Y_pred, float(loss))
  steps += 1
  opt.step()
```

After a few thousand iterations, it predicts around 0.5 but not exactly:

```
2315 tensor([[[0.4669]]], device='cuda:0', grad_fn=<AddBackward0>) 0.0010972624877467752
2316 tensor([[[0.5115]]], device='cuda:0', grad_fn=<AddBackward0>) 0.00013136999041307718
2317 tensor([[[0.4788]]], device='cuda:0', grad_fn=<AddBackward0>) 0.00045129822683520615
2318 tensor([[[0.4658]]], device='cuda:0', grad_fn=<AddBackward0>) 0.0011675604619085789
```

Note that if I set `model.eval()` instead of `model.train()`, then the model is able to fit as expected (predicts 0.5000 after about 200 iterations). The problem exists in the RoBERTa model as well.

## Version information

- `transformers` version: 4.3.2
- Platform: Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!