Wav2Vec2 Speech Pre-Training After a few epochs the contrastive loss was decreased to zero and the model stopped changing
I tried to re-run the demo script with the same parameters on Colab. After a few epochs, the contrastive loss was decreased to zero and the model stopped changing. Original Script can be found here: https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-pretraining

Here is sample code [Colab](https://colab.research.google.com/drive/1IDwie_Te_2GntHay_zZ-oPvriNsacQ8d?usp=sharing):

sample output:

  ```
| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 5.137e-01| ppl: 
  2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 8.068e-19
  
  | loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 4.952e-01| ppl: 
  2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 4.017e-19
  
  
  | loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 4.831e-01| ppl: 
  2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 5.166e-19
```

I used the parameters given in the README file so this behavior is unexpected and may indicate a different problem.

Is this a bug in the official feature or am i doing some mistake if so please help how can I fix this problem?

@patrickvonplaten @anton-l 
Hey @umairahmad-ua,

I was using 8 GPUs - in a simple colab you get just a single GPU, so I think the effective batch_size is very much different which then quickly loads to a collapse of the `contrastive_loss`. TBH, I dont' think you can reproduce the training in a single colab (you would have to let it run for 4 weeks to see significant changes)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>It's interesting, I have the same problem when pre-train wav2vec2 on my own datasets, and the div_loss cannot go down at same value (9.969e-01).
I set batch_size=16, single GPU.