Can we save tokenized datasets?
It takes a lot of time to tokenize my dataset, is there a way to save it and load it?

Let's say I'm using the IMDB toy dataset, How to save the `inputs` object?
```
from datasets import load_dataset
raw_datasets = load_dataset("imdb")
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
inputs = tokenizer(sentences, padding="max_length", truncation=True)


Sure, that's possible.

You can tokenize your entire dataset using the tokenizer, then save it to disk using  the `save_to_disk` method as explained in the [docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=save_to_disk#datasets.Dataset.save_to_disk).

Small example:

```
from datasets import load_dataset
from transformers import AutoTokenizer

datasets = load_dataset("imdb")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

encoded_datasets = datasets.map(lambda examples: tokenizer(examples['text']), batched=True)
encoded_datasets.save_to_disk('.')
```

You can later load it back in using the `load_from_disk` method as explained [here](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=save_to_disk#datasets.Dataset.load_from_disk).<|||||>> Sure, that's possible.
> 
> You can tokenize your entire dataset using the tokenizer, then save it to disk using the `save_to_disk` method as explained in the [docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=save_to_disk#datasets.Dataset.save_to_disk).
> 
> Small example:
> 
> ```
> from datasets import load_dataset
> from transformers import AutoTokenizer
> 
> datasets = load_dataset("imdb")
> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
> 
> encoded_datasets = datasets.map(lambda examples: tokenizer(examples['text']), batched=True)
> encoded_datasets.save_to_disk('.')
> ```
> 
> You can later load it back in using the `load_from_disk` method as explained [here](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=save_to_disk#datasets.Dataset.load_from_disk).

Thank you Niels! My problem is a bit more complex, I created a custom dataset using method here: https://github.com/huggingface/notebooks/blob/master/transformers_doc/custom_datasets.ipynb, but the imdbDataset object created doesn't has `save_to_disk` method.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.