Wav2Vec2 - TypeError: Concatenation operation is not implemented for NumPy arrays
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.16.2
- Platform: Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-44-generic x86_64)
- Python version: 3.8
- PyTorch version (GPU?): 1.8.0+cu111 (Yes)
- Tensorflow version (GPU?): -
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@patrickvonplaten 

## Information

Model I am using (Bert, XLNet ...): Wav2Vec2ForPreTraining

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behaviour:

1. I tried running the [pretraining script](https://github.com/huggingface/transformers/blob/d1fd64e7aa40d6a3c69cb21f7fd411a2a3141e04/examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py)  with the given [command ](https://github.com/huggingface/transformers/tree/d1fd64e7aa40d6a3c69cb21f7fd411a2a3141e04/examples/pytorch/speech-pretraining)(except that I used the dummy dataset to first test it with less data) for the "base-sized" Wav2Vec2 model:

````
accelerate launch run-wav2vec2-pretrain-notrainer.py \
--dataset_name="hf-internal-testing/librispeech_asr_dummy" \
--dataset_config_name="clean" \
--dataset_split_names validation \
--model_name_or_path="patrickvonplaten/wav2vec2-base-v2" \
--output_dir="./wav2vec2-pretrained-demo" \
--max_train_steps="20000" \
--num_warmup_steps="32000" \
--gradient_accumulation_steps="8" \
--learning_rate="0.005" \
--weight_decay="0.01" \
--max_duration_in_seconds="20.0" \
--min_duration_in_seconds="2.0" \
--logging_steps="1" \
--saving_steps="10000" \
--per_device_train_batch_size="8" \
--per_device_eval_batch_size="8" \
--adam_beta1="0.9" \
--adam_beta2="0.98" \
--adam_epsilon="1e-06" \
--gradient_checkpointing \
--validation_split_percentage="10"\
````

## Error
I then get the error:
````
^M  0% 0/20000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/ceph/csedu-scratch/project/agansen/run-wav2vec2-pretrain-notrainer.py", line 724, in <module>
    main()
  File "/ceph/csedu-scratch/project/agansen/run-wav2vec2-pretrain-notrainer.py", line 566, in main
    for step, batch in enumerate(train_dataloader):
  File "/ceph/csedu-scratch/project/agansen/venv2/lib/python3.8/site-packages/accelerate/data_loader.py", line 303, in __iter__
    for batch in super().__iter__():
  File "/ceph/csedu-scratch/project/agansen/venv2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/ceph/csedu-scratch/project/agansen/venv2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/ceph/csedu-scratch/project/agansen/venv2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/ceph/csedu-scratch/project/agansen/run-wav2vec2-pretrain-notrainer.py", line 323, in __call__
    sampled_negative_indices = _sample_negative_indices(
  File "/ceph/csedu-scratch/project/agansen/venv2/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 329, in _sample_negative_indices
    sampled_negative_indices[batch_idx] += batch_idx * sequence_length
TypeError: Concatenation operation is not implemented for NumPy arrays, use np.concatenate() instead. Please do not rely on this error; it may not be given on all Python implementations.
````

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behaviour
I ran similar scripts using '_sample_negative_indices()' before and did not get the error. I am not sure what changed. Any ideas would be very appreciated!!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>cc @ArthurZucker @sanchit-gandhi - this issue popped up again. It seems like it works with Transformers v4.15. In case someone has time to look into it to check if it works on current master that would be great. Otherwise happy to take a look!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hey @annagansen22,

Sorry could you try updating your transformers version and then starting the script again? I think this error should not be present anymore for newer versions<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @patrickvonplaten the issue still persists, see https://github.com/speechbrain/speechbrain/issues/1787

I provide there a short summary only:
```
File "speechbrain/recipes/CommonVoice/self-supervised-learning/wav2vec2/train_hf_wav2vec2.py", line 111, in fit_batch
    predictions = self.compute_forward(batch, sb.Stage.TRAIN)
...
  File "transformers/models/wav2vec2/modeling_wav2vec2.py", line 285, in _sample_negative_indices
    sampled_negative_indices[batch_idx] += batch_idx * sequence_length
TypeError: Concatenation operation is not implemented for NumPy arrays, use np.concatenate() instead. Please do not rely on this error; it may not be given on all Python implementations.
```

Environment:
```
datasets                 2.7.1
huggingface-hub          0.11.1
numpy                    1.23.4
scipy                    1.8.1
torch                    1.12.1
torchaudio               0.12.1
transformers             4.25.1

Python 3.9.13

PyTorch 1.12.1+cu102
```

Titouan restricted the dependency `transformers==4.15` – which I dropped to see if we really need this limitation (or if we can use simply the latest dependencies for all SpeechBrain recipes).

As the error log suggests, there are other Python/PyTorch versions which with that works?<|||||>Hey @anautsch,

Could you maybe open a new issue that includes a reproducible code snippet that only uses `transformers`? 
Thanks! 