Hello @ptrblck!



 ptrblck:

Could you add the synchronization and profile the code again?


Here I add the synchronization:
torch.cuda.synchronize()
gpuTime = time.time()
for  i in range (nBatch):
    preds = model (data[batchInd])
torch.cuda.synchronize()
gpuTime = time.time() - gpuTime

And here are the synchronized timings for the batch sizes:
batchSize = 5 , nBatch = 100000 , cpuTime = 11.282513856887817 , gpuTime = 27.322113752365112
batchSize = 10 , nBatch = 50000 , cpuTime = 7.730835914611816 , gpuTime = 14.358086585998535
batchSize = 25 , nBatch = 20000 , cpuTime = 5.687099933624268 , gpuTime = 7.724848031997681
batchSize = 50 , nBatch = 10000 , cpuTime = 4.814324617385864 , gpuTime = 3.956281900405884
batchSize = 100 , nBatch = 5000 , cpuTime = 4.489711046218872 , gpuTime = 2.366971969604492
batchSize = 250 , nBatch = 2000 , cpuTime = 4.938727378845215 , gpuTime = 1.6444478034973145
batchSize = 500 , nBatch = 1000 , cpuTime = 5.077881097793579 , gpuTime = 1.5218729972839355
batchSize = 1000 , nBatch = 500 , cpuTime = 4.937913656234741 , gpuTime = 1.419776201248169
batchSize = 2500 , nBatch = 200 , cpuTime = 5.141258001327515 , gpuTime = 1.3623504638671875
batchSize = 5000 , nBatch = 100 , cpuTime = 4.970717668533325 , gpuTime = 1.3436284065246582

Thanks.
K. Frank