Hi,
Here is the code that does proper timings:
import pdb
import time
import torch

x = torch.rand([30,30])
x = x.cuda()
y = torch.rand([30,1])
y = y.cuda()

torch.cuda.synchronize()
start = time.time()
w = torch.gesv(y,x)
torch.cuda.synchronize()
end = time.time()

print "GPU before warmup"
print (end-start)

x = torch.rand([30,30])
x = x.cuda()
y = torch.rand([30,1])
y = y.cuda()

torch.cuda.synchronize()
start = time.time()
w = torch.gesv(y,x)
torch.cuda.synchronize()
end = time.time()

print "GPU after warmup"
print (end-start)

x = torch.rand([30,30])
x = x.cuda()
y = torch.rand([30,1])
y = y.cuda()

torch.cuda.synchronize()
start = time.time()
w = torch.gesv(y,x)
# MISSING synch here, so we only measure the time to launch the kernel
# not the time to execute it
# torch.cuda.synchronize()
end = time.time()

print "GPU just launch time"
print (end-start)

x = torch.rand([30,30])
y = torch.rand([30,1])

w = torch.gesv(y,x)

torch.cuda.synchronize()
start = time.time()
w = torch.gesv(y,x)
torch.cuda.synchronize()
end = time.time()

print "CPU"
print (end-start)

From the results you can see that the time is mostly spent launching the job on the gpu, no actually doing any computation.
For very small workload, it is expected that running on gpu is going to be slower as the cost to launch the job is fixed and will become dominant for very small workloads.