model.config.vocab_size has a wrong value compared to the actual vocab size in the tokenizer
#2
by
Owos
- opened
model.config.vocab_size has a higher value that the total number of tokens in the tokenizer
Hi @Owos ,
Welcome to the Google Gemma family of open source models. Thanks for notifying us on the discrepancy in the tokens count and vocab size. I have escalated the above issue to our internal team.
Thanks.