Hello Bram,
Thanks for your response, and sure, will mention the line in future.
Regarding your response, Yes I understand that we are doing padding, the problem is this (and correct me if I’m wrong). You are overwriting on the label of the same sample by adding pad tokens which will be ignored, the problem though is this: say the effective length of a sequence is 100 which is the longest in a batch (say  others are around 20), now, when one such sample ends up in a batch, you elongate all other sample to reach 100, now since we are randomly shuffling, after a while majority of samples will have severely prolonged due to outliers. For my use case, this has a huge implications. You may ask why not to truncate those outliers, well, that does not solve the issue of the code. That’s why I asked why you are overwriting. Any thoughts?
best