When I use mm or bmm on a cuda tensor with custom strides (either obtained through unfold or as_strided), cublas runtime error occurs. The operations run correctly when device = cpu.
Reproducing examples:
bmm example:
import torch

N = 10
x = torch.arange(N * (N - 1), device='cuda:5').float().reshape(N, N - 1) # shape (10, 9)

unf1 = x.unfold(1, N // 2, 1) # shape (10, 5, 5)
out1 = unf1.bmm(unf1) # error below

unf2 = x.as_strided((10, 5, 5), (9, 1, 1)) # shape (10, 5, 5)
out2 = unf2.bmm(unf2) # error below

In both cases I get
On entry to SGEMM  parameter number 8 had an illegal value
RuntimeError: cublas runtime error : an invalid numeric value was used as an argument at /pytorch/aten/src/THC/THCBlas.cu:441
mm example:
import torch

N = 10
x = torch.arange(N - 1, device='cuda:5').float() # shape (9,)
unf1 = x.unfold(0, N // 2, 1) # shape (5, 5)
out1 = unf1.mm(unf1) # error below

unf2 = x.as_strided((5, 5), (1, 1)) # shape (5, 5)
out2 = unf2.mm(unf2) # error below

In both cases I get
On entry to SGEMM  parameter number 8 had an illegal value
RuntimeError: cublas runtime error : an invalid numeric value was used as an argument at /pytorch/aten/src/THC/THCBlas.cu:258
Version:
pytorch 1.0.0
cuda 10.1
gpu Titan Xp or GTX 1080