Thank you for this @lhoestq.  Could you please explain what is the benefit of doing this -

#We drop the small remainder, we could add padding if the model supported it instead of this drop,
#you can customize this part to your needs.
total_length = (total_length // max_seq_length) * max_seq_length

Why not just skip that statement ? Thank you.