Thanks for taking a look, this loop happens within a larger loop that instantiates a dataloader for each file containing my data.
The problem was actually that I needed all instances of my network to start with the same weight values for averaging the parameters to work as part of a distributed training setup. So in my case, setting the random seed at instantiation solved this.