@Andreas_Lu, @nabsabs
I assume you are both using the DGX-1 with V100 GPUs?
Have you checked, if the data loading/processing might be the bottleneck in your setup?
If so, I would suggest to have a look at DALI 75, which should speedup the data processing pipeline.
PS: It should be already preinstalled in our containers.