@richard,
I also thought that if the input tensor is a cuda array then it should use the GPU but running
T = torch.randn(400,400).cuda()
 for i in range(100):
    somevar = torch.qr(T)

causes only a marginal increase in volatile GPU utilization (8-10%)
but blows up all CPU cores.