Great example of never tag. 
As @amosella explains much better than I would have, the random init leaves you with different states depending on the number of parameters (and possibly the init method).
So the easiest solution is to re-seed after the weight init and before the dataloader init. If I remember correctly, the CPU random generator is the important one here.
Best regards
Thomas