@ptrblck Hello. I am also very interested in this question. Setting the sequence dimension to the first dimension would make the matrix multiply extremely inefficient due to memory access issues.
Is this really because of compatibility with RNNs?
If completely designed from scratch, the most reasonable order would appear to be (N, E, S) instead of the current (S, N, E) order. This would allow the sequence length axes, which are the ones being multiplied in the (self-attention matrix @ value matrix) operation, to be contiguous. As this is the most computationally intensive, this operation should be the most heavily optimized.
cuDNN and cuBLAS optimizations mean that (N, S, E) order, which is called batch_first order in torchtext, will probably have similar performance, though this should be tested.