nn.Linear accepts a variable number of dimensions as [batch_size, *, in_features], so you might use the temporal dimension as dim1.
The view operations in your code should work, since you are not permuting the dimensions (if I understand your code correctly).