FYI I think there’s something a little strange going on with gradcheck. For example, check out what happens with this simple function:
correlation_product = lambda f, d: torch.mul(d, torch.dot(f, d).expand_as(d))
print(gradcheck(correlation_product, (f, d), eps=1e-6, atol=1e-3))
print(gradcheck(correlation_product, (f, d), eps=1e-6, atol=1e-2))
print(gradcheck(correlation_product, (f, d), eps=1e-6, atol=1e-1))

prints False, False, True (f and d are declared as V(torch.FloatTensor(np.random.randn(5)), requires_grad=True)). My understanding is that this is using PyTorch’s built-in autograd, so it seems strange to have precision issues of this magnitude. This is on pytorch 0.1.11+8aa1cef.