Fix tokenizer.json with file from Qwen/Qwen2.5-14B
#3
by
MariusNocturnum
- opened
Tokenizer.json is 11.4 MB and appears to have gotten messed up in the fine tuning. Tokenizer.json should be 7.03 MB as seen on the base model (Qwen/Qwen2.5-14B)
Crystalcareai
changed pull request status to
closed
Thanks! This is a common issue we've encountered with some of our internal merging tools. We'll work on addressing it moving forward—at the very least, by implementing stronger checks. While it doesn't throw errors during evaluation, testing, or quantization, it does cause problems with certain SageMaker endpoints, among other things. I swapped out the tokenizer with the one from Qwen's Instruct model, just to be safe.