But the fact that you do as many backwards as there are samples in the batch makes them slower when the batch size increases.
In that case, I can buy that there is a sweet spot for a small number of samples.
Also note that each call to backward uses the FULL graph. So the more sample, the more expensive the backward for each individual sample.