Solved by rmorgan in post #3 
              
              
                Thanks for taking a look, this loop happens within a larger loop that instantiates a dataloader for each file containing my data. 
The problem was actually that I needed all instances of my network to start with the same weight values for averaging the parameters to work as part of a distributed tra…