Procedure_1:
Loss1= crit(pred,target)
Loss1.backward()
Loss2 = crit(pred,target)
Loss2.backward()
Optim.step()
Procedure_2:
Loss1= crit(pred,target)
Loss1.backward()
Optim.step()
Optim.zero_grad()
Loss2 = crit(pred,target)
Loss2.backward()
Optim.step()
Optim.zero_grad()
Are these two procedure same in nature meaning they will yield same amount of gradient update?
I am trying to switch to efficient b5 from b0 for a project but my gpu can only handle batchsize of 3 on b5. If i run with batchsize 3, the loss does not converge as well as before with b0. So i am thinking about gradient accumulation. Is the abovementioned way is the right way to do it?
Also what are the other way to mitigate the batch issue? I am trying to install apex for the last 3 hours but failed! :’(