Solved by kuonlp in post #3 
              
              
                Edit: I’ve solved the problem, so I re-wrote my initial answer and what I initially thought it was the problem. Then I write the actual problem I had (which was embarrassingly simple). 
I initially though that with torch.nn.Parameter I would be creating new parameters all the time, and because of th…