Hi pytorch friends!
I want to optimize per-example gradient computation, and found an interesting behavior when increasing the batch size.   My findings can be reproduced in this notebook 3.  Essentially I compute per-example losses then to call backward on each one:
t0 = time()
losses = loss_fn(model(samples), labels)
model.zero_grad()
for i, l in enumerate(losses):
    l.backward(retain_graph=True)
    # do something with the individual gradients
sampling_rate = batch_size / (time() - t0)
print(f"{sampling_rate:.0f} Hz")

I find that the computed sampling rate has a maximum for 20 to 60 samples dependent on the used GPU.
How can you (a) explain this behavior, and (b) possibly fix it so that I can get the most of my computations?
Thanks for considering my problem, please AMA