I have a problem when computing batch Jacobian. I am not sure if it is bug or I am using autograd engine incorrectly.
I used the following snippet to compute Jacobian of the output. I compared Jacobian computed by pytorch against Theano/Lasagne network initialized with the identical parameters. For the starting output 0 the results are identical. However, for the subsequent backward calls i > 0, the results differ by some constant factor (in some cases 2 or 3 but not always deterministic).
What are the reasons that the gradient accumulated multiple times in the leafs even after the input gradient was reset?
import torch
from torch.autograd.gradcheck import zero_gradients

def compute_jacobian(inputs, output):
    assert inputs.requires_grad
    num_classes = output.size()[1]

    jacobian = torch.zeros(num_classes, *inputs.size())
    grad_output = torch.zeros(*output.size())
    if inputs.is_cuda:
        grad_output = grad_output.cuda()
        jacobian = jacobian.cuda()

    for i in range(num_classes):
        zero_gradients(inputs)
        grad_output.zero_()
        grad_output[:, i] = 1
        output.backward(grad_output, retain_variables=True)
        jacobian[i] = inputs.grad.data

    return jacobian