trengan:

So, we have a loss function with gradients on the variables that decide (along with step size/learning rate) what the next values of variables should be.


The loss function does not know the next values. It just calculates the current gradients for all necessary parameters.
The optimizer uses these gradients of the provided parameters to update the weights.
Some optimizers have a momentum term or other parameters for running averages. Have a look at e.g. Adam 7.
So indeed the optimizer only needs to know which Parameters to update and how to do it. The needed gradient has to be provided by the backward pass of your loss function.



 trengan:

In other words, what if I had another loss function on the same set of parameters (for e.g., loss_2 = nn.NLLLoss(outputs, labels) where outputs and labels are as above)? How would it know which gradient to use to take the next “step”?


The gradients are accumulated by default. So in your case both loss functions would calculate the gradients which are summed for each parameter. That is also the reason you have so zero out the gradient before the next backward pass (optimizer.zero_grad() in your training code).