Tokenizer issue
#7
by
MarsupialAI
- opened
Model fails to load with the supplied tokenizer in this repo. I borrowed the tokenizer file from swiss-ai/Apertus-70B-2509 and that version worked.
Thanks for reporting. Can you please share the error and the script that you use to replicate the error?
I don't remember the exact wording, but it was resolved by deleting the tokenizer.json and tokenizer_config.json and replacing them with the versions of those files from the non-instruct 70b repo.
Looking at the repo now, it seems somebody has recently modified the tokenizer file. So I guess it's been fixed.
MarsupialAI
changed discussion status to
closed