Adjust max length in tokenizer config

by aehrm - opened Jun 13

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

-2

aehrm

Data Science@CAIDAS Uni Würzburg org Jun 13

Update tokenizer configuration to align max length with config.json. This change does not affect the tokenizer's output. It only prevents (incorrect) warnings for sequences exceeding 1024 tokens.

Adjust max length in tokenizer configc396a7b3

JanPf changed pull request status to merged Jun 13

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment