How did you get this to work without Yi's Tokenizer?

#1
by tachyphylaxis - opened

I tried one other model (I forget which one, it's on the tip of my tongue) that was Yi-based that didn't include the tokenizer, and it put commas after every period, heh. Whatever you did to get this to work, I'm interested so I could do the same. Thanks.

Use chargoddard's wonderful Yi llamafied as a base:

https://huggingface.co/chargoddard/Yi-34B-200K-Llama

Sign up or log in to comment