The tokenization seems right and I don’t think it would solve anything but I would give tokenized_dataset = dataset.map(lambda x: flaubert_tokenizer(x['verbatim'], padding="max_length", truncation=True, max_length=512), batched=True) a try. The problem, I think, is different sizes of output tensors after tokenization, which I would verify next as a sanity check. Good luck!