A (possible) bug of sentence permuation in BART pretraining script
### System Info

None

### Who can help?

@sanchit-gandhi 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

As described in the [original BART paper](https://arxiv.org/pdf/1910.13461.pdf), the sentence permutation should be:  A document is divided into sentences based on **full stops**, and these sentences are shuffled in a random order.

However, in the [examples/flax/language-modeling](https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_bart_dlm_flax.py#L318), it uses pad token as the `end_sentence_mask`. 

Hey @StevenTang1998!

Great question! In our Flax pre-training script, use the PAD token as the end of sentence indicator:
https://github.com/huggingface/transformers/blob/76924384af6081e58460231c3c637f9c83cabf97/examples/flax/language-modeling/run_bart_dlm_flax.py#L626-L627

This way, the PAD token is appended after the end of a sentence (full stop, exclamation marks, question marks etc) and used to indicate sentence boundaries. So in splitting a document by the PAD token, we are in effect splitting sentences on end of sentence punctuation (equivalent to the original paper!).

Hope this addresses your concerns! Let me know if you have any questions and I'd be happy to answer 🤗<|||||>Thanks for kind reply! I understand it now!

But I have a concern that the PAD token may affect the results (through position embedding?) although it will not be attend to.<|||||>Hey @StevenTang1998! Glad to hear that clarified things! It should be fine since we build the attention mask based on the position of the PAD token ids :)<|||||>OK, thank you for your reply! Wish you a good day!