Hi
Suppose I have three optimizers and I want to optimize a graph three times within one batch. Which one is the right order to call those functions?





optimizer1.zero_grad()
optimizer2.zero_grad()
optimizer3.zero_grad()

loss1.backward(retain_graph=True)
loss2.backward(retain_graph=True)
loss3.backward()

optimizer1.step()
optimizer2.step()
optimizer3.step()







optimizer1.zero_grad()
loss1.backward(retain_graph=True)
optimizer1.step()

optimizer2.zero_grad()
loss2.backward(retain_graph=True)
optimizer2.step()

optimizer3.zero_grad()
loss3.backward()
optimizer3.step()


Thanks