A more explicit example in a similar direction as @ruotianluo is the ability to add gradients from several forward passes, for example in GANs:

  
    
    
    How to use the backward functions for multiple losses? 
  
  
    Hello @Nabarun_Goswami, 
to try to clear this up, in the DCGAN example, you have (think about mathematical functions here, I left out everything not relevant). 
loss = criterion(netD(real, params))+criterion(netD(fake, params)) 
Spelling out the chain rule for the gradient of the loss w.r.t. the params: 
∇params loss = ∇params netD(real, params)* ∇netD loss(netD (real,params)) +  ∇params netD(fake, params)* ∇netD loss(netD (fake,params)), 
note how ∇params netD is evaluated at two different poin…
  


If you wanted, you could also achieve minibatches that are larger than fit in your memory by combining several sub-minibatches into one gradient step, but I have not really seen that done.
Best regards
Thomas