Solved by tom in post #2 
              
              
                The training step parts of “run backward” and “optimizer step” are independent in PyTorch, so the first computes gradients for everything that has requires_grad=True and is ignorant of whether you use them in the optimizer step. So you would keep allocating gradients for the unused parameters but ne…