@K_Frank Awesome, great to know.
I actually meant .backwards(). Typically, the three steps for backprop:
opt.zero()
loss.backwards().
opt.step()
I believe backwards() usually takes the longest by far, way more than the inference and opt step etc.
So how do we determine the maximum batch size for 100% parallel computation? Is there a code / diagnostic tool we can use to determine when doing multi GPU backprop will increase speed?