Tokenizers as an optional dependency
# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Would it be possible to make `tokenizers` an optional dependency? I see this was already attempted here by @thomwolf: https://github.com/huggingface/transformers/pull/2342.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->
For various reasons we are unable to support Rust in our environment. Given that `tokenizers` is a hard dependency, this means we cannot use `transformers` at all. We would be fine with using the non-fast versions of tokenizers as a workaround.

Sounds reasonable, but – if you're at liberty to share – and out of curiosity, would also like to know why you can't install Rust-built native deps<|||||>Thanks for the quick response. Rust is generally fine – what's causing issues is specifically the `pyo3` crate, which has a somewhat involved build script which doesn't get along with our build system.<|||||>Hi @jeanm, I work on the tokenizers library, can you explain how you are unable to use `tokenizers` ? You should never have to "see" Rust as we ship prebuilt libraries.
Maybe we are missing a platform we should add so that you don't have to build from source and so you don't have an issue with Rust or Pyo3 ?<|||||>Only using the python tokenizers may prevent you from running some example scripts and use some additional functionalities of the library in the future though since we plan to rely more and more on the fast alignements tools provided by the Rust tokenizers to make processing simpler and more accurate.

Do you think you could give us more details on the issue so that we can try to make the tokenizers library compatible with your system?

Happy to talk further by DM/mail if it's easier for you to give some details, you can ping me by email or twitter/linkedin for instance.<|||||>Hi @Narsil @thomwolf, thanks for the responses. As a matter of policy (+ technical reasons I unfortunately cannot get into) we have to build all python wheels from source. If it were possible to make `tokenizers` optional without complicating things on your end, we would be perfectly fine with dealing with reduced functionality, as that's still much better than not being able to run the package at all :)