Solved by 111189 in post #8 
              
              
                I solved it. The reason GPUs are not fully utilized is that I have not used enough CPU. 
The CPU has to pass the dataset to the GPU quickly, but I limited the number of CPUs in the DataLoader (num_workers=4), so the dataset was not fed to the GPU enough. 
In fact, by increasing the num_workers(=16) …