unable to pass through create vocabulary_from_data when using a custom dataset loaded from disk which is a result of concatenating multiple datasets
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

 `transformers` version: 4.17.0.dev0
- Platform:Ubuntu 20.04.3 LTS
- Python version: 3.8.10
- PyTorch version (GPU?):1.10.2+cu113
- Tensorflow version (GPU?): not installed 
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten @anton-l 

Models:

- Wav2Vec2 

Library:

- Speech

## Information

Model I am using : facebook/wav2vec2-xls-r-300m

The problem arises when using:
* [ ] the official example script: https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py


The tasks I am working on is:
* [ ]  dataset: using a load_from_disk() method to load a locally preprocessed and saved dataset. i have downloaded and preprocessed marathi dataset from 3 different huggingface datasets locations: (openslr, openslr64) , (mozillafoundation/common_voice_8_0, mr) , (indic_voice, mr) 
 finally the indic_voice dataset was preprocessed into to standard format of common_voice and openslr and the 3 datasets were combined using the datasets.concatenate_datasets() method and finally the dataset was saved to disk, 

the same dataset is then loaded in the official run_speech_recognition_ctc.py file 

## To reproduce

Steps to reproduce the behavior:
download multiple marathi datasets and concatenate and save to disk, load the dataset in the official training script 

 I have been running the official script `run_speech_recognition_ctc.py` as provided in
 [https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition](https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition)


1. `python run_speech_recognition_ctc.py \
	--dataset_name="common_voice" \
	--model_name_or_path="facebook/wav2vec2-large-xlsr-53" \
	--dataset_config_name="mr" \
	--output_dir="./wav2vec2-common_voice-mr" \
	--overwrite_output_dir \
	--num_train_epochs="15" \
	--per_device_train_batch_size="16" \
	--gradient_accumulation_steps="2" \
	--learning_rate="3e-4" \
	--warmup_steps="500" \
	--evaluation_strategy="steps" \
	--text_column_name="sentence" \
	--length_column_name="input_length" \
	--save_steps="400" \
	--eval_steps="100" \
	--layerdrop="0.0" \
	--save_total_limit="3" \
	--freeze_feature_encoder \
	--gradient_checkpointing \
	--chars_to_ignore , ? . ! - \; \: \" “ % ‘ ” � \
	--fp16 \
	--group_by_length \
	--push_to_hub \
	--do_train --do_eval `




## Error Message
"hidden_act": "gelu",
  "hidden_dropout": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-05,
  "layerdrop": 0.1,
  "mask_feature_length": 10,
  "mask_feature_min_masks": 0,
  "mask_feature_prob": 0.0,
  "mask_time_length": 10,
  "mask_time_min_masks": 2,
  "mask_time_prob": 0.075,
  "model_type": "wav2vec2",
  "num_adapter_layers": 3,
  "num_attention_heads": 16,
  "num_codevector_groups": 2,
  "num_codevectors_per_group": 320,
  "num_conv_pos_embedding_groups": 16,
  "num_conv_pos_embeddings": 128,
  "num_feat_extract_layers": 7,
  "num_hidden_layers": 24,
  "num_negatives": 100,
  "output_hidden_size": 1024,
  "pad_token_id": 0,
  "proj_codevector_dim": 768,
  "tdnn_dilation": [
    1,
    2,
    3,
    1,
    1
  ],
  "tdnn_dim": [
    512,
    512,
    512,
    512,
    1500
  ],
  "tdnn_kernel": [
    5,
    3,
    3,
    1,
    1
  ],
  "torch_dtype": "float32",
  "transformers_version": "4.17.0.dev0",
  "use_weighted_layer_sum": false,
  "vocab_size": 32,
  "xvector_output_dim": 512
}

  0%|          | 0/1 [00:00<?, ?ba/s]

stays hung up here and blows up all the RAM (i have 64 GB of ram)  Once the all the RAM in the system is consumed, the program is killed by bash. 
upon the script being killed in the output_dir there is no vocab.json file
[pre-processing and training files.zip](https://github.com/huggingface/transformers/files/8024270/pre-processing.and.training.files.zip)

follow these zip file that contains a jupyter notebook used to concatenate multiple datasets and saving to disk, the .py files contains code on loading the dataset from disk and continuing with the training process

This sounds like it's a problem with the `datasets` library. @StephennFernandes could you try to reproduce the bug just by using the Datasets library? It would be great if you could provide a small code snippet the reproduces the bug easily :-)<|||||>Hey @patrickvonplaten  tried to pass the almost the same datasets through the create_vocabulary_from_data function in a jupyter notebook and it processed it in a second, but in the training script its overloading the ram and crashing. 

Also tried uninstalling ans reinstalling transformers and datasets from source<|||||>Hey @patrickvonplaten  heres something really weird that i've discovered

when i preprocess the 3 datasets slr, commonvoice and indic_voice, i put the datasets through a lot of .map methods 
combined_dataset = combined_dataset.map(...)  and after this when the processed dataset is finally passed through the create_vocabulary_from_data() function it is RAM heavy and slows the entire process down. 

i tried to load the indic_voice dataset, which is almost 90 % of the entire dataset in this specific language. when i tired passing its ['sentence'] column through the create vocabulary function() it works absolutely fine (it processes everything under 3 seconds), 

but the combined dataset blows up the RAM, 

I have a hunch that when the datasets goes back and forth into the .map function it slowing the process down, maybe it has something to do deep internally with how datasets sores and processes data internally. <|||||>Hey @StephennFernandes - this sounds like a question for https://github.com/huggingface/datasets actually. Could you maybe try to ask there? Also cc'ing @lhoestq  and @polinaeterna <|||||>The extraction of all the characters of each text is done completely in memory:

https://github.com/huggingface/transformers/blob/2de99e6c43403d5988ae20cfc0af063797d69c29/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L320

Removing `keep_in_memory=True` should fix this<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.