error when running official run_speech_recognition_ctc.py (ValueError: 'a' cannot be empty unless no samples are taken)
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

 `transformers` version: 4.17.0.dev0
- Platform:Ubuntu 20.04.3 LTS
- Python version: 3.8.10
- PyTorch version (GPU?):1.10.2+cu113
- Tensorflow version (GPU?): not installed 
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten @anton-l 

Models:

- Wav2Vec2 

Library:

- Speech

## Information

Model I am using : facebook/wav2vec2-xls-r-300m

The problem arises when using:
* [ ] the official example script: https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py


The tasks I am working on is:
* [ ]  dataset: mozilla-foundation/common_voice_8_0

## To reproduce

Steps to reproduce the behavior:
 I have been running the official script `run_speech_recognition_ctc.py` as provided in
 [https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition](https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition)


1. `python run_speech_recognition_ctc.py \
	--dataset_name="common_voice" \
	--model_name_or_path="facebook/wav2vec2-large-xlsr-53" \
	--dataset_config_name="ta" \
	--output_dir="./wav2vec2-common_voice-tamil" \
	--overwrite_output_dir \
	--num_train_epochs="15" \
	--per_device_train_batch_size="16" \
	--gradient_accumulation_steps="2" \
	--learning_rate="3e-4" \
	--warmup_steps="500" \
	--evaluation_strategy="steps" \
	--text_column_name="sentence" \
	--length_column_name="input_length" \
	--save_steps="400" \
	--eval_steps="100" \
	--layerdrop="0.0" \
	--save_total_limit="3" \
	--freeze_feature_encoder \
	--gradient_checkpointing \
	--chars_to_ignore , ? . ! - \; \: \" “ % ‘ ” � \
	--fp16 \
	--group_by_length \
	--push_to_hub \
	--do_train --do_eval `




## Error Message


`train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/stephen/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1373, in train
    tr_loss_step = self.training_step(model, inputs)
  File "/home/stephen/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1948, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/stephen/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1980, in compute_loss
    outputs = model(**inputs)
  File "/home/stephen/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/stephen/.local/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1742, in forward
    outputs = self.wav2vec2(
  File "/home/stephen/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/stephen/.local/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1342, in forward
    hidden_states = self._mask_hidden_states(
  File "/home/stephen/.local/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1284, in _mask_hidden_states
    mask_time_indices = _compute_mask_indices(
  File "/home/stephen/.local/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 263, in _compute_mask_indices
    spec_aug_mask_idx = np.random.choice(
  File "mtrand.pyx", line 909, in numpy.random.mtrand.RandomState.choice
ValueError: 'a' cannot be empty unless no samples are taken`

## Expected behavior
The same script runs fine without any errors, when the **mask_time_prob** is set to 0 instead of the default value mentioned in the script that 0.05
Hey @StephennFernandes,

Thanks a mille for opening the issue - this PR seems to be related: https://github.com/huggingface/transformers/pull/15423<|||||>Hey @StephennFernandes,

Can you try whether the error goes away with the newest `transformers` version?<|||||>hey @patrickvonplaten i installed tranfromers from source `pip install git+https://github.com/huggingface/transformers` and the issue still persists<|||||>Hmm, that's a bit odd. Rerunning your training command now<|||||>Hey @StephennFernandes, 

I ran the command on master and trained for 3 epochs without error. You can see the run here: https://huggingface.co/patrickvonplaten/wav2vec2-common_voice-tamil<|||||>Could you verify that you are using current master of `transformers`?<|||||>hey patrick i uninstalled and reinstalled transformers and it looks like everything is working fine. 

One strange thing i noticed is that until a couple of epochs the wer went low 0.63 and then the wer came up again and for the rest of the training it was 1.0, the final model was trained for 1.0 wer. 

This isnt the first time i am facing such issues on updated version of transformers. 

the only version of transformers that works fine and converges wers for me was 4.8.0, is there some issue with my cuda drivers perhaps ?? what version of CUDA drivers does huggingface recommend along with pytorch version. 
I have an Nvidia A6000 GPU <|||||>Uff I really don't know sadly :-/ Could you maybe open a new issue since this one does not seem to be related anymore or a forum discussion: https://discuss.huggingface.co/? :-)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.