I have also encountered a similar issue just now with gradcheck on custom cuda extension. Moreover, by only switching an activation function from tanh to relu the error is suddenly gone … Seems to work without gradcheck, even with double Tensors.