Solved by ptrblck in post #2 
              
              
                Optimizers subtract the gradient of all passed parameters using their .grad attribute as seen here. Thus you would minimize the loss using gradient descent.