Hi whoab!



 whoab:

Is it true that you can increase your batch size up till your ~maximum GPU memory before loss.step() slows down? I thought a GPU would do computation for all samples in the batch in parallel, but it seems like Pytorch GPU-accelerated backprop takes much longer for bigger batches. It could be swapping to CPU, but I look at nvidia-smi Volatile GPU Memory usage and it is under 70%.


The short answer to your question is no.
In general, a GPU can only do calculations in parallel up to the
number of pipelines it has.  A GPU might have, say, 12 pipelines.
So putting bigger batches (“input” tensors with more “rows”) into
your GPU won’t give you any more speedup after your GPUs are
saturated, even if they fit in GPU memory.  Bigger batches may
(or may not) have other advantages, though.
As an aside, you probably didn’t mean to say loss.step().
Pytorch loss functions don’t have step methods.  (Optimizers
do.)  The part of the typical training iteration that processes a
batch (potentially partially in parallel) is when you call something
like prediction = model (input).
Also it’s not clear to me which part of the calculation you mean
when you say “backprop”.  If you mean updating your model
weights, this occurs when you call optim.step(), and this
piece is independent of the size of the batches.  (However, the
gradients used by optim.step() are being accumulated when
you call model (input).)
Best regards.
K. Frank