Yeah, you are right.
A non-contiguous input tensor would also create non-contiguous gradients or is this not always true?
I think _gather_flat_grad might get an additional check or a better error message? What do you think?