x=torch.ones(2,3,4).permute(0,2,1)
(x * 2).stride()
Out[54]: (12, 1, 4)
(x.log()).stride()
Out[55]: (12, 1, 4)

So, elementwise ops don’t create contiguous tensors, but I believe they have the same speed on non-contiguous tensors (at least unary and scalar operand ops)
(x.matmul(torch.ones(3,5))).stride()
Out[56]: (20, 5, 1)
F.conv1d(x, torch.ones(5,4,1)).stride()
Out[58]: (15, 3, 1)

Contiguous tensors are created by usual “layer” ops. They can work slower on non-contiguous tensors, yes.
However:
nhwc = torch.ones(2,3,4,5,device="cuda")
nchw = nhwc.permute(0,3,1,2)
(nchw.stride(), nchw.is_contiguous(), nchw.is_contiguous(memory_format=torch.channels_last))
Out[69]: ((60, 1, 20, 5), False, True)
F.conv2d(nchw, torch.ones(6,5,1,1,device="cuda")).stride()
Out[72]: (72, 1, 24, 6)

here torch.channels_last format is auto-detected for a cuda 4d tensor, and is preserved