Solved by ptrblck in post #10 
              
              
                You could try to check all tensors for a valid .grad_fn and detach every tensor, which is not needed anymore. If the backward pass yields the slowdown this might indeed point towards a growing computation graph. 
Do you see an increase in memory usage while you are training the model? 
Just a wild …