MNLI: BERT No Training Progress
## 🐛 Bug

I am using BERT and have successfully set up pipelines for 7/8 GLUE tasks, and I find comparably good accuracy on all of them. However, for MNLI task, training loss does not converge at all. I am correctly using 3 classes with `num_classes`. In fact, I have even tried reduced the scope of MNLI to a 2-class problem (entailment vs neutral/contradiction) for testing purposes to see if this is the issue, and the model fails to converge on this training task as well. However, the exact same code (with only the dataset switched) works for MRPC, for example.

Model I am using (Bert, XLNet....): **BERT**

Language I am using the model on (English, Chinese....): **English**

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: **see co-lab**

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: **MNLI**
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

See linked co-lab for minimum viable example
https://colab.research.google.com/drive/1Thdrpvp0uX2TCaCpqsLoUVn7DuRLlILQ

## Expected behavior

I expect to see good training progress; instead, training loss does not converge.

## Environment

* OS: **Linux**
* Python version: **3.6**
* PyTorch version: **1.3.0**
* PyTorch Transformers version (or branch): **2.1.1**
* Using GPU ? **Yes**
* Distributed of parallel setup ? **No, single GPU**
* Any other relevant information: **None**

I'm assuming you did multiple runs with ≠ seeds?<|||||>Yes -- I have tried with multiple different seeds.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> Yes -- I have tried with multiple different seeds.

@evanweissburg Have you found a solution? I'm also having issues with MNLI data. <|||||>@anthonyfuller7 Nope. I think one resolution was to not use BERT large.<|||||>I too am facing the same issue with BERT and Albert as well. The model does not converge on fine-tuning and the loss does not decrease even over 5 epochs. Did someone manage to solve this issue?