In case of multi gpu, can we still do this? I have two gpus, each has enough memory to load the data into the gpu before training. If I load the data and train it with single gpu, the gpu utilization is 25% higher than loading from cpu at each batch. However, if I load to gpu and train it with two gpus the performance is worse than loading from cpu. I guess this happens because it’s moving data from gpu0 to gpu1 which becomes a bottleneck.