Why don't the models of Yi-*-Chat have the "tokenizer.json" file?

#21

by joeking11829 - opened Apr 2, 2024

Apr 2, 2024

I want to perform TGI benchmark tests on the Yi model's token throughput, but I need the "tokenizer.json" file for the model.

Any Idea ?
Thanks.

YShow

Apr 30, 2024

You can load the tokenizer and save the information to a JSON file on your own

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment