ptrblck:

A non-contiguous input tensor would also create non-contiguous gradients or is this not always true?


We try to follow in the input layout in most cases but this is not guaranteed indeed.

I think _gather_flat_grad might get an additional check or a better error message? What do you think?

We definitely can improve that error message to at least recommend to the user to make his weights contiguous as a first step (but we do not want to enforce it I think as it will break valid user code when the view is valid on a non-contiguous Tensor).