Can you see how much your GPUs are utilized using nvidia-smi. If they are not utilized fully it means the pre-processing (could be reading time from hard-drive) is taking much more time to utilize the GPU properly.