Solved by eqy in post #9 
              
              
                You can see the short note in the docs here for why Distributed Data Parallel can be faster for multi-GPU training. 
Yes, switching HDD to SSD can make a large difference, especially for “random” file I/O as loading many ImageNet images can look like random reads to storage. If you have sufficient m…