Hi @ptrblck!
It looks like this is a batch-size issue.
I ran some timings with dummy data.  Note, this is just a loop
over the forward pass – no loss, backward, or optimization.
For large batch sizes (>~ 1000) the gpu runs much faster than
the cpu; for the batch size of 25 I’ve been using with my toy
model, the gpu runs rather more slowly than the cpu.
I’m running a toy model, and using a batch size of 25 for no
particular reason, so I don’t really care.  But I would be
curious whether this behavior persists in newer versions of
pytorch and on other cpus, and if there is a way to run “small”
batch sizes efficiently on the gpu.
Here is the timing script:
import time

import torch
print  (torch.__version__)

batchSize = 25
nEpoch = 100  # notioanal epoch of 5000
nBatch = (nEpoch * 5000) // batchSize

data = torch.autograd.Variable (torch.randn (9298, 256))
batchInd = torch.LongTensor (range (batchSize))
model = torch.nn.Sequential(
    torch.nn.Linear (256, 512),
    torch.nn.Tanh(),
    torch.nn.Linear (512, 10),
)

print ('run cpu ...')
cpuTime = time.time()
for  i in range (nBatch):
    preds = model (data[batchInd])
cpuTime = time.time() - cpuTime

print ('torch.cuda.get_device_name (0) =', torch.cuda.get_device_name (0))
data = data.cuda()
batchInd = batchInd.cuda()
model.cuda()

print ('run gpu ...')
gpuTime = time.time()
for  i in range (nBatch):
    preds = model (data[batchInd])
gpuTime = time.time() - gpuTime

print  ('batchSize =', batchSize, ', nBatch =', nBatch, ', cpuTime =', cpuTime, ', gpuTime =', gpuTime)

Here is the complete output for batchSize = 25:
0.3.0b0+591e73e
run cpu ...
torch.cuda.get_device_name (0) = Quadro K1100M
run gpu ...
batchSize = 25 , nBatch = 20000 , cpuTime = 5.642305850982666 , gpuTime = 7.473216533660889

And here are timings for a series of batch sizes:
batchSize = 5 , nBatch = 100000 , cpuTime = 11.414417505264282 , gpuTime = 28.121765613555908
batchSize = 10 , nBatch = 50000 , cpuTime = 7.832420349121094 , gpuTime = 14.703708171844482
batchSize = 25 , nBatch = 20000 , cpuTime = 5.642305850982666 , gpuTime = 7.473216533660889
batchSize = 50 , nBatch = 10000 , cpuTime = 4.890258312225342 , gpuTime = 3.968475341796875
batchSize = 100 , nBatch = 5000 , cpuTime = 4.532914876937866 , gpuTime = 2.2654550075531006
batchSize = 250 , nBatch = 2000 , cpuTime = 4.889425277709961 , gpuTime = 1.4718339443206787
batchSize = 500 , nBatch = 1000 , cpuTime = 5.050147533416748 , gpuTime = 1.1878798007965088
batchSize = 1000 , nBatch = 500 , cpuTime = 5.072073698043823 , gpuTime = 0.7794077396392822
batchSize = 2500 , nBatch = 200 , cpuTime = 4.958431720733643 , gpuTime = 0.1718151569366455
batchSize = 5000 , nBatch = 100 , cpuTime = 4.986212730407715 , gpuTime = 0.1562213897705078

(The cpu timings of <~5 are all “about the same”, with differences
within the run-to-run noise.)
Thanks.
K. Frank
Edit:  I also tried taking the indexing out, i.e.,
data = torch.autograd.Variable (torch.randn (batchSize, 256))
...
for  i in range (nBatch):
    preds = model (data)

The timings became modestly smaller, but they showed the same
pattern with respect to the batch size and gpu vs. cpu.