Update tokenizer_config.json

by weiyezhimeng - opened 7 days ago

←

7 days ago

当add_generation_prompt为False时，增加了end token到后面，适用于SFT训练。不这样的话huggingface的SFTTrainer训练将导致模型无法停止输出。

7 days ago

另外chat_template.jinjia也需要做相应更新

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment