Fix: encoder and decoder in tokenizer

#13

Hi!

When evaluating the RWKV-v5-Eagle-7B-HF model, I found the errors as shown below. This is mostly caused by the tokenizer. In the code, the encoder and decoder are reversed, as discussed here https://huggingface.co/RWKV/v5-Eagle-7B-HF/discussions/9#65d4dc35f9cbfa798c4be4b3

Evaluation error on RWKV-v5-Eagle-7B-HF

This PR is raised to fix this issue.
Thanks,

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment