Hi smth,
So, eventually, there is no necessity to divide loss with iter_size?
I’m still bit confusing since apaszke mentioned about dividing here.

  
    
    
    PyTorch Gradients 
  
  
    Normally when we’re doing backprop we would do the following: 
loss.backward()  # This calculates the gradients
optimizer.step()   # This updates the net

However, what if I wish to accumulate the gradients? Meaning I want to run various loss.backward() multiple times first and accumulate the gradients before applying them in optimizer.step(). 

What do you recommend for storing the gradients? grads = loss.backward()?
How do I feed the accumulated gradients to optimizer.step?
  


Thank you for your help.