While I may not have realized that you were talking about CLM, I think there is still relevant information here. The example scripts for both MLM and CLM show tokenizing and then grouping by max_seq_length, which means there will be sequences with BOS/EOS tokens in the middle and other sequences with no BOS/EOS tokens at the beginning/end of the sequence.
I’m wondering if this is problematic, and it seems like you are wondering about the impact of BOS/EOS tokens too