wav2vec2 not converging when finetuning
## Environment info

- `transformers` version: 4.4.0
- Platform: Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.5.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten 

## Information

Model I am using: wav2vec2

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I have a dataset of single English word, 1 second long audio files sampled at 16kHz. I wanted to use wav2vec2 for speech recognition instead of just doing audio classification because I wanted the model to be able to generalize to longer audio samples with more words. I followed [the official wav2vec2 guide](https://huggingface.co/blog/fine-tune-wav2vec2-english) almost exactly (the only difference was the dataset used, but I made sure the dataset format and vocab list format was identical as well) but the model does not seem to be converging. The loss would decrease to approx. 3 and stay around there. Checking the predictions made during evaluation, I realized that the model just kept outputting the padding token regardless of the input.

Other issues with similar behaviour are #10884 and #10983. I have tried suggestions there such as increasing the learning rate with no success. 

## Expected behavior

The model should show signs of convergence, such as slowly starting to output sensible prediction strings.

Any help is greatly appreciated!

Hey @meeps123,

We try to keep the github issues for code related bugs. For such questions, could you please the [forum](https://discuss.huggingface.co/) instead? :-) Feel free to tag me there! 

Also could you attach a google colab so that I can take a look at your training script? It is very difficult to draw any conclusions just from reading the text. 

Cheers,
Patrick<|||||>Hi @patrickvonplaten,

Sure thing! I have opened a topic [here](https://discuss.huggingface.co/t/wav2vec2-not-converging-when-finetuning/6773). The Colab notebook is linked there. Thank you for the assistance! 