So from my knowledge nesterov momentum should look something like this:
nesterov1113×206 8.21 KB
According to this formula, the loss function should be calculated not according to our model’s parameters, but with a prediction of the future model’s parameters.
And using a torch SGD optimizer with Nesterov should look like the following:
optimizer = torch.optim.SGD(..., nesterov=True)
optimizer.zero_grad()
loss_fn(model(input), target).backward()
optimizer.step()

My question is how can optimizer.step() apply the correct parameter update (in the Nesterov case) if the loss is calculated according to the model’s present parameters and not “future” parameter predictions as in the equation above? I already looked at the source code and I still do not understand how this was done.
Does PyTorch call the forward method of the model and somehow overwrites its parameters in model(input)? If that is the case, how is that done?