@Giuseppe_Castiglione: could you please elaborate your answer more?
How did you use torch.autograd.grad() to get the gradient of each individual input vector from a batch?
Some code snippet will be appreciated.
One relevant discussion is

  
    
    
    Efficient computation of per-sample examples autograd
  
  
    Additional discussion of [feature request] Simple and Efficient way to get gradients of each element of a sum 
@tom I hope I interpreted your last post correctly by posting here. 

The code to reproduce the numbers I see is available here. 
The main.py file runs different ways of computing individual gradients and checks/times them. 
Your recommendation seems to be along the line of Goodfellow’s derivation, implemented as goodfellow, timing output under goodf. 
I don’t think it is a bug per-se, …