If you set shuffle=True, internally the RandomSampler will be used, which just permutes the indices of all samples as seen here 650.
Are you slicing the data before passing it to the DataLoader?
If not, all samples should be used regardless if shuffle is active of not.