I hear you…
I feel the examples and documentations aren’t as elaborate as we would’ve wished they’d be.
Also I didn’t mention this explicitly, but I’ve set max_length=2000 in this tokenization function:
def tok(example):
  encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=2000)
  return encodings

But you should set it to whatever you think is legit.
I don’t have any new ideas as I’m quite new to this library as well, but update us on the development!
Hopefully you’ll solve it soon