No transposes would help, the sizes of unf1 are (10,5,5) and the strides are (9,1,1), and for lda/ldb blas standard specifies (I’m quoting for lda, but ldb is similar modulo permutation of m,n,k and transpose value)
 LDA    - INTEGER.
             On entry, LDA specifies the first dimension of A as
             declared in the calling (sub) program. When  TRANSA =
             'N' or 'n' then LDA must be at least  max( 1, m ),
             otherwise  LDA must be at least  max( 1, k ).
             Unchanged on exit.

so, at least one of the last strides should be 5 for pytorch to be able to correct situation with setting appropriate transposes.
Using self-overlapping tensors (like this one) on cuda is not the greatest idea anyway. Pytorch tries to detect it in some places and throw errors or call .contiguous(), but I don’t think it was ever made robust. Perhaps bmm should be changed to detect this situation and call .contiguous() on the tensor?
@albanD do you think it makes sense to open a github issue to discuss this?