Hello whoab!



 whoab:

@K_Frank Awesome, great to know.
I actually meant .backwards(). Typically, the three steps for backprop:
opt.zero()
loss.backwards().
opt.step()
I believe backwards() usually takes the longest by far, way more than the inference and opt step etc.
So how do we determine the maximum batch size for 100% parallel computation? Is there a code / diagnostic tool we can use to determine when doing multi GPU backprop will increase speed?


Let me correct something I said above that wasn’t entirely right.
model (input) does (typically) work on an entire batch, but
that is not the only part of training that does.  As both you and
@jmaronas correctly point out, loss.backwards() does as
well.  Derivatives can involve more calculation, so, as you noted,
loss.backwards() is likely to be your most time-consuming step.
But my main points still stand:


A GPU typically only has a handful of pipelines, so you
don’t need much of a batch to saturate them.  (And there
is plenty of opportunity for vectorization / parallelism even
with a batch size of one.)


The primary purpose of using batches is to make the
training algorithm work better, not to make the algorithm
use GPU pipelines more efficiently.  (People use batches
on single-core CPUs.)


So increasing your batch size likely won’t make things
run faster.  (More precisely, it won’t generally let you run
through an epoch faster.  It might make your training
converge more quickly or more slowly.)


I don’t have any advice on how to make your training run
faster.  There might be some tuning you can do, but my
working assumption is that, under the hood, pytorch does
a pretty good job of using the GPUs efficiently.  It may be
that you need faster hardware – a faster GPU with more
pipelines, or a machine with multiple GPUs.  (Or maybe
more time and patience …)
Best.
K. Frank