Not able to load tokenizer
The tokenizer is not getting loaded.
OSError: Can't load tokenizer for 'facebook/xmod-base'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'facebook/xmod-base' is the correct path to a directory containing all relevant files for a XLMRobertaTokenizerFast tokenizer.
@jvamvas , super cool model!
Transformers maintainer here, usually we recommend uploading the tokenizer files as well to the repository so that AutoTokenizer.from_pretrained("facebook/xmod-base") may work correctly.
In this situation, I would recommend uploading this file to your repository: https://huggingface.co/xlm-roberta-base/blob/main/tokenizer.json
as well as a tokenizer_config.json file like the following: https://huggingface.co/facebook/nllb-200-1.3B/blob/main/tokenizer_config.json#L22
that would contain a single key, the following: "tokenizer_class": "XLMRobertaTokenizer", to indicate that it is, indeed, an XLM-R tokenizer definition.
Let me know if something is unclear @jvamvas !
