optimizer.zero_grad()
loss.backward()
optimizer.step()
This is actually the correct order.
Here is a code snippet given in Pytorch’s docs

optimizer.zero_grad()
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()

DId changing the order really fix it?