What do you see in nvidia-smi -l 1 regarding GPU utilization? If it’s significantly less than 100%, then you have a bottleneck before the GPUs, which is indeed probable.
How many workers do you use in the DataLoader? The number may vary depending on your configuration and preprocessing, but I’ve used 4x the number of GPUs with optimal performance consistently, this might be a starting point…