Shuffle is not enabled in the default dataloaders in the trainer. If you want to add a PR, you need to add an argument to the TrainingArguments and update the Trainer dataloaders.
See comment by @sgugger below.