Hi @albanD,
Based on Tony’s code, suppose we have a batch of data x, and we need to calculate the grad one by one like:
output[i] = fn(x[i])
grad = torch.autograd.grad(output[i], x[i])

Considering the explanations you provided, either we can do
grad_x, = torch.autograd.grad(output[i], x)
grad_x_i = grad_x[i]

or we can do
u = x[i]
output = fn(u)
grad_x_i = torch.autograd.grad(output, u)
grad_x[i] = grad_x_i 

Will the latter way be faster than the former one? Since now it seems we need to calculate the input one by one, can the calculation be accelerated if in GPU? Thank you for your time in advance!