Wav2vec2 Training Loss not decreasing
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.4.0
- Platform: Google Colab
- Python version: python 3.6

@patrickvonplaten




Models:

wav2vec2

I am following the recent implementation of wav2vec2 for fine-tuning:
https://huggingface.co/blog/fine-tune-wav2vec2-english

Settings:
Pretrained model: "facebook/wav2vec2-base-960h",
gradient_checkpointing=True,
ctc_loss_reduction="mean",
pad_token_id=processor.tokenizer.pad_token_id
attention_dropout=0.1,
hidden_dropout=0.1,
feat_proj_dropout=0.0,
mask_time_prob=0.05,
layerdrop=0.1,
gradient_checkpointing=True,
ctc_loss_reduction="mean",
group_by_length=True,
per_device_train_batch_size=32,
evaluation_strategy="steps",
num_train_epochs=1500,
fp16=True,
save_steps=400, #this would mean every 400 steps model gets saved which also means Google drive gets full
eval_steps=400,
logging_steps=400,
learning_rate=0.0005,
warmup_steps=500,
save_total_limit=2,

Issue:

Step | Training Loss | Validation Loss | Wer | Runtime | Samples Per Second
-- | -- | -- | -- | -- | --
400 | 5.063200 | 4.566135 | 1.000000 | 0.715900 | 6.984000
800 | 5.115200 | 4.514411 | 1.000000 | 0.732400 | 6.827000
1200 | 5.119200 | 4.485986 | 1.000000 | 0.724300 | 6.903000

The training loss is marginally decreasing and WER is still 1. What can be done to improve and faster training with better accuracy.

I also tried with a higher learning rate but training loss was still very poor, it seems the model is not converging.
It seems that your training epochs is set to 1500. Set it up to 5 for a quick trial<|||||>It is giving training loss of more than a 100 in that case!<|||||>Is your training finished or is still running? If it is the second, just try again with less number of epochs for example<|||||>This is the output for 5 epochs:


TrainOutput(global_step=10, training_loss=93.45169677734376, metrics={'train_runtime': 48.9011, 'train_samples_per_second': 0.204, 'total_flos': 2.6027104384512e+16, 'epoch': 5.0, 'init_mem_cpu_alloc_delta': 348007, 'init_mem_gpu_alloc_delta': 377847808, 'init_mem_cpu_peaked_delta': 18306, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 706705, 'train_mem_gpu_alloc_delta': 1120621568, 'train_mem_cpu_peaked_delta': 161498645, 'train_mem_gpu_peaked_delta': 7221921792})<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.