tokenizer.json + tokenizer_config.json missing?

#25
by am17an - opened

I'm trying to create a GGUF for this model but I don't see these files (tokenizer.json + tokenizer_config.json or related tokenizer files). Will they be added?

Hey @am17an , can you please also open an issue on Transformers?
The Transformers integration was done using mistral-common, but I actually have commits converting the tokenizer and using Transformers code only, so I'm happy to add this codepath if needed! πŸ€—

Sign up or log in to comment