I’m a NLP researcher from Brazil and our team is training RoBERTa base from scratch with around ~60GB Portuguese dataset. We plan on releasing it on HF model hub. Regarding data-prep, we have two options for documents longer than 512 tokens (maxlen):


Truncate the data point and discard the rest


Break long data points into smaller chunks of 512 tokens (generating new data points)


What are your opinions on these approaches?