Have another thread now that includes a full code sample.

  
    
    
    Multi GPU Hook not correctly doing filling buffer 
  
  
    Sorry, I kept a bit of extra code since I started with https://github.com/pytorch/examples/blob/master/mnist/main.py. and thought it would be easy to compare to that.  I have further reduced the code below to only have the exact things to reproduce this problem.  My network is 1 layer, the ‘main’ just inititalizes things and runs backprop on one random input.  The only thing that is unique is my saveAverageD function and 2 custom modules.  The custom modules and saveAverageD function were made b…