I think you want something like discussed in this thread (see also the bug report):

  
    
    
    Efficient computation of per-sample examples autograd
  
  
    Additional discussion of [feature request] Simple and Efficient way to get gradients of each element of a sum 
@tom I hope I interpreted your last post correctly by posting here. 

The code to reproduce the numbers I see is available here. 
The main.py file runs different ways of computing individual gradients and checks/times them. 
Your recommendation seems to be along the line of Goodfellow’s derivation, implemented as goodfellow, timing output under goodf. 
I don’t think it is a bug per-se, …
  


It’s not something that you would change at the “loss end” of the computational graph, but at the parameter end.
Best regards
Thomas