Wrong results for GLUE task STS-B
## Environment info
- `transformers` version: 4.6.0.dev0
- Platform: Linux-4.13.0-26-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help
 @sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): bert-base-cased

The problem arises when using:
* [ ] the official example scripts:  examples/pytorch/text-classification/run_glue.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: STS-B

## To reproduce

export TASK_NAME=stsb

python run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/

## Expected behavior
According to README, Person/Spearman corr. should be close to 88.64/88.48
However, I get Person/Spearman corr. equal to 28.36/27.70

The following warning occurs:
python3.8/site-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([32])) that is different to the input size (torch.Size([32, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

Are you sure you don't have any bad checkpoints in where your `output_dir` points to? Just executed the command and it works perfectly fine on my end. You can add `--overwrite_output_dir` to your command to make sure to overwrite what is there.<|||||>> Are you sure you don't have any bad checkpoints in where your `output_dir` points to? Just executed the command and it works perfectly fine on my end. You can add `--overwrite_output_dir` to your command to make sure to overwrite what is there.

I'm sure the problem is not `output_dir`, it points to an empty directory in the beginning. I've rerun the code adding `--overwrite_output_dir` just to make sure, still getting the same results. The results for other GLUE tasks are all correct, only problem with STS-B.