I would encourage you to set shuffle=True. I would also not be very troubled if the training loss is a bit worse when you shuffle. How much worse is the loss when you shuffle compared to when you don’t? 10%, 100%, 2000%?