Solved by ptrblck in post #16 
              
              
                Thank you very much for this great code to reproduce this issue! 
Indeed the memory is growing in each epoch. 
After looking into the code, I think the reason is that you might track the computation graph in self.running_mean and self.running_covar unintentionally. 
This might be the case if you ass…