Even with 16 workers there might be a large imbalance between data loading time and GPU time. Do you have some statistics on the proportion of time spent on data in each batch (e.g., like in the classic ImageNet example here 5)?