TypeError (NumPy concatenation) in modeling_wav2vec2 at _sample_negative_indices
### System Info

- `transformers` version: 4.25.1
- Platform: Linux-5.4.0-73-generic-x86_64-with-glibc2.27
- Python version: 3.9.15
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.13.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes - Driver Version: 495.29.05    CUDA Version: 11.5
- Using distributed or parallel set-up in script?: no


### Who can help?

transformers library

@patrickvonplaten 

### Information

- The official example scripts
- [X] My own modified scripts

### Tasks

- An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Error log
```
speechbrain.utils.checkpoints - Would load a checkpoint here, but none found yet.
speechbrain.utils.epoch_loop - Going into epoch 1
speechbrain.core - Exception:
Traceback (most recent call last):
  File "recipes/CommonVoice/self-supervised-learning/wav2vec2/train_hf_wav2vec2.py", line 357, in <module>
    asr_brain.fit(
  File "speechbrain/core.py", line 1207, in fit
    self._fit_train(train_set=train_set, epoch=epoch, enable=enable)
  File "speechbrain/core.py", line 1060, in _fit_train
    loss = self.fit_batch(batch)
  File recipes/CommonVoice/self-supervised-learning/wav2vec2/train_hf_wav2vec2.py", line 111, in fit_batch
    predictions = self.compute_forward(batch, sb.Stage.TRAIN)
  File "recipes/CommonVoice/self-supervised-learning/wav2vec2/train_hf_wav2vec2.py", line 54, in compute_forward
    out, mask = self.modules.wav2vec2(wavs)
  File "torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "speechbrain/lobes/models/huggingface_wav2vec.py", line 434, in forward
    transformers.models.wav2vec2.modeling_wav2vec2._sample_negative_indices(
  File "transformers/models/wav2vec2/modeling_wav2vec2.py", line 285, in _sample_negative_indices
    sampled_negative_indices[batch_idx] += batch_idx * sequence_length
TypeError: Concatenation operation is not implemented for NumPy arrays, use np.concatenate() instead. Please do not rely on this error; it may not be given on all Python implementations.
```

where
```
numpy                    1.23.4
scipy                    1.8.1
```

This issue came up during [reworking testing for SpeechBrain](https://github.com/speechbrain/speechbrain/pull/1600). As part of refactoring & expanding our integration of the HuggingFace transformers library, we ensured to have testing for all SpeechBrain recipes. After lifting an extra_dependency restriction, this error occured.

What was changed?
https://github.com/speechbrain/speechbrain/blob/801b1501b0bde2a940fcb71af44b69b07eafb9f5/recipes/CommonVoice/self-supervised-learning/wav2vec2/extra_requirements.txt#L1
to
https://github.com/anautsch/speechbrain/blob/b7e1b02a8cb3be81640c40c23a99d5af646a24e5/recipes/CommonVoice/self-supervised-learning/wav2vec2/extra_requirements.txt#L1

How to reproduce?
a) either run the [recipe script](https://github.com/speechbrain/speechbrain/tree/develop/recipes/CommonVoice/self-supervised-learning/wav2vec2) from scratch (might be too hardware intense)
b) or use a testing tool, we created that runs the recipe in a very light debug mode

To use the recipe testing, please create an environment using this SpeechBrain version (from our mentioned PR).
```
git clone https://github.com/anautsch/speechbrain.git
cd speechbrain
git checkout refactor-recipe-testing
pip install -r requirements.txt
pip install transformers==4.25.1 huggingface-hub==0.11.1 datasets==2.7.1
pip install --editable .
```

The particular recipe can then be tested using this command:
```
python -c 'from tests.utils.recipe_tests import run_recipe_tests; print("TEST FAILED!") if not(run_recipe_tests(filters_fields=["Hparam_file"], filters=[["recipes/CommonVoice/self-supervised-learning/wav2vec2/hparams/wav2vec2_base.yaml"]], do_checks=False, run_opts="--device=cuda")) else print("TEST PASSED")'
```

This will result in
```
(1/1) Running test for CommonVoice_row_18...
	ERROR: Error in CommonVoice_row_18 (recipes/CommonVoice/self-supervised-learning/wav2vec2/hparams/wav2vec2_base.yaml). Check tests/tmp/CommonVoice_row_18/stderr.txt and tests/tmp/CommonVoice_row_18/stdout.txt for more info.
```
and the above stacktrace is available through either `cat`
```
cat tests/tmp/CommonVoice_row_18/std*
cat tests/tmp/CommonVoice_row_18/log.txt
```

I started an [issue on our end](https://github.com/speechbrain/speechbrain/issues/1787) as part of keeping track of all issues that surfaced during testing all SpeechBrain recipes. There, the suggestion is to reintroduce the dependency restriction `transformers==4.15`.

### Expected behavior

It would be great to lift all extra_dependency restrictions in SpeechBrain recipes and move them to the latest versions, e.g., `transformers>=4.25.2` instead of fixing it to a specific & older version (v4.15 dates back to Dec 22, 2021).
cc @sanchit-gandhi <|||||>Hey @anautsch! Thanks for opening this issue 🤗 Would it be possible to provide a reproducible code snippet that only uses `transformers`? This way we can pinpoint the exact issue in the library.

In an attempt to try and reproduce your issue, I tested the Wav2Vec2 pretraining script from the `transformers` library: 
[run_wav2vec2_pretraining_no_trainer.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py). However, I was not able to reproduce the error you encountered; both the script and model worked fine without the aforementioned numpy concatenation error.

In these tests, I used the ['base' Wav2Vec2 model](https://huggingface.co/patrickvonplaten/wav2vec2-base-v2/blob/main/config.json) and pre-trained on a [dummy dataset](https://huggingface.co/datasets/hf-internal-testing/librispeech_asr_dummy/tree/main) consisting of 73 samples from the LibriSpeech ASR corpus (~9MB of data). 

You can reproduce this dummy run using the following command:
```
accelerate launch run_wav2vec2_pretraining_no_trainer.py \
	--dataset_name="hf-internal-testing/librispeech_asr_dummy" \
	--dataset_config_name="clean" \
	--dataset_split_names validation \
	--model_name_or_path="patrickvonplaten/wav2vec2-base-v2" \
	--output_dir="./wav2vec2-pretrain-issue" \
	--num_train_epoch="1" \
	--max_duration_in_seconds="20.0" \
	--per_device_train_batch_size="8" \
	--per_device_eval_batch_size="8" \
	--validation_split_percentage="10" \
	--gradient_checkpointing
```
**Print Output:**
```
Gradients have overflown - skipping update step... Updating gradient scale to 65536.0...                                
Gradients have overflown - skipping update step... Updating gradient scale to 32768.0...                                
Gradients have overflown - skipping update step... Updating gradient scale to 16384.0...                                
Gradients have overflown - skipping update step... Updating gradient scale to 8192.0...                                 
Gradients have overflown - skipping update step... Updating gradient scale to 4096.0...                                 
Gradients have overflown - skipping update step... Updating gradient scale to 2048.0...                                 
| val_loss: 4.825e+00| val_contrastive_loss: 4.732e+00| val_diversity_loss: 9.208e-01| val_num_losses: 1.000e+00        
100%|█████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:09<00:00,  1.08it/s]
Configuration saved in ./wav2vec2-pretrain-issue/config.json
Model weights saved in ./wav2vec2-pretrain-issue/pytorch_model.bin
100%|█████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:11<00:00,  1.63s/it]
```

As you can see, the script and model worked fine for me here! If you can provide a similar code snippet that demonstrates your issue that would be grand!<|||||>Hi @sanchit-gandhi thank you for providing an alternate example (I couldn't get it running right away). But thanks for nudging me towards a minimal example. After looking at the inputs, it turned out we were giving as input arguments for `(batch_size, sequence_length)` a `(2, tensor(157))` mix. It worked when using `.item()` on a previous variable, so the input argument is `(2, 157)` instead.<|||||>Hey @anautsch! Very glad to hear that - best of luck with your development!