Solved by albanD in post #15 
              
              
                But the fact that you do as many backwards as there are samples in the batch makes them slower when the batch size increases. 
In that case, I can buy that there is a sweet spot for a small number of samples. 
Also note that each call to backward uses the FULL graph. So the more sample, the more exp…