As far as I understand you are not updating any weights, just computing the gradients w.r.t. the input?
Are you using BatchNorm layers? They might be updated in the first few iterations and then converge to the input mean and std.