Thanks for your reply. Now the time required for the calculation of the loss is constant.
But of course, synchronization is now consuming a lot of time.
Why is it the case, that the time required for synchronization is increasing until it ‘saturates’ and is dropping again after num_worker batches have been processed?
If I set non_blocking=False during cuda calls, the required time for synchronization is constant, but then it takes more time to move each batch to GPU.  Is there any way to speed up the overall processing time or is synchronization a bottleneck I can not avoid?