This is an excerpt from the classic “Training a classifier” tutorial on PyTorch.org 9.
The loss function and optimizer are defined as below:
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
in order to evaluate loss and update gradients to take an optimization step like so:
outputs = net(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
Forgive the possibly stupid question but where is the link between the loss function and optimizer object? All that optimizer needs for initiation seems to be the parameters of net which simply has the definitions of the linear, activation, softmax etc. Where do we tell the optimizer that it is the gradient of the loss function w.r.t these parameters that guides the step? In other words, the parameters are there but not what we are taking the gradient of.