Embedding layer Pruning implementation
I want to use a multilingual model (in my case MT5) to train it for two languages (English and German).
These models have a very high vocab size because they are multilingual.
This large embedding needs a lot of RAM on GPU.

My idea is: Tokenize a large English and German corpus and find out which tokens are needed.
Then remove all tokens that are not needed. I call this pruning...
This removal needs to be done on the model embedding layer
and on the tokenizer (in my case a sentencepiece tokenizer).

Did anybody ever implement something like that? Is there maybe some example code?
@sgugger @LysandreJik 

If I manage to implement I might provide the functionality as a PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.