Using the process of elimination, from the simplest fix to trickier modifications:


I moved the dataset to SSD, it turns out that the overall training time improved insignificantly, while still showing that saw-like utilization graph. Thus, hardware bottleneck may have little contribution to this phenomenon. The main bottleneck probably lies in code, as you suggested.


The code I attached in the original post has a ‘try… except…’ wrapper to handle an ‘OSError: image file is truncated’ exception. This exception seems to have something to do with PIL according to here and this issue. I wonder if this try-catch behavior had led to the bottleneck, so I preprocessed the images by reducing their sizes with the following code:


from PIL import Image

def shrink_large_img(img_path):
    try:
        img = Image.open(img_path)
        if min(img.size) > 512:
            print(f'File {img_path} shrinked!')
            img.thumbnail((275, 275))
            img.save(img_path, 'JPEG')
    except OSError as e:
        print(f'Bad image: {img_path}')

(credit to this post 1
After that, when training the model again, the utilization graph changed a little bit, as follows:

2.png1255×1287 36.7 KB

You can notice that the space between the peaks disappeared. Furthermore, now there’s no  ‘OSError: image file is truncated’  anymore. So it’s natural to try out the code with ‘try-catch’ behavior removed:


After removing the try block, the utilization graph stays the same. So the try-catch block played no role in bottlenecking the training.


I measured average loading and training time of an epoch using suggested code, the result is as follows:


Training started at Sat Jan  5 14:24:50 2019
Let's use 2 GPUs!
C:\ProgramData\Anaconda3\envs\dl\lib\site-packages\torch\cuda\nccl.py:24: UserWarning: PyTorch is not compiled with NCCL support
  warnings.warn('PyTorch is not compiled with NCCL support')
Epoch: 1 	Training Loss: 0.686348 	Validation Loss: 0.697969
Validation loss decreased (inf --> 0.697969).  Saving model ...
Time 1.014 (1.926) Data 0.936 (1.677)
Epoch: 2 	Training Loss: 0.627555 	Validation Loss: 0.590918
Validation loss decreased (0.697969 --> 0.590918).  Saving model ...
Time 0.891 (1.880) Data 0.813 (1.689)
Epoch: 3 	Training Loss: 0.402697 	Validation Loss: 0.264288
Validation loss decreased (0.590918 --> 0.264288).  Saving model ...
Time 0.979 (1.858) Data 0.900 (1.688)
Epoch: 4 	Training Loss: 0.160563 	Validation Loss: 0.104242
Validation loss decreased (0.264288 --> 0.104242).  Saving model ...
Time 1.005 (1.849) Data 0.925 (1.688)
Epoch: 5 	Training Loss: 0.086767 	Validation Loss: 0.062527
Validation loss decreased (0.104242 --> 0.062527).  Saving model ...
Time 0.950 (1.842) Data 0.872 (1.686)
Epoch: 6 	Training Loss: 0.061195 	Validation Loss: 0.048539
Validation loss decreased (0.062527 --> 0.048539).  Saving model ...
Time 0.912 (1.838) Data 0.834 (1.686)
Epoch: 7 	Training Loss: 0.047683 	Validation Loss: 0.043027
Validation loss decreased (0.048539 --> 0.043027).  Saving model ...
Time 0.875 (1.833) Data 0.788 (1.685)
Epoch: 8 	Training Loss: 0.037087 	Validation Loss: 0.034808
Validation loss decreased (0.043027 --> 0.034808).  Saving model ...
Time 0.921 (1.828) Data 0.832 (1.682)
Epoch: 9 	Training Loss: 0.033605 	Validation Loss: 0.031286
Validation loss decreased (0.034808 --> 0.031286).  Saving model ...
Time 0.908 (1.828) Data 0.838 (1.683)
Epoch: 10 	Training Loss: 0.029369 	Validation Loss: 0.029808
Validation loss decreased (0.031286 --> 0.029808).  Saving model ...
Time 1.071 (1.833) Data 0.987 (1.690)
Epoch: 11 	Training Loss: 0.026300 	Validation Loss: 0.028633
Validation loss decreased (0.029808 --> 0.028633).  Saving model ...
Time 0.913 (1.837) Data 0.840 (1.694)
Epoch: 12 	Training Loss: 0.023830 	Validation Loss: 0.025342
Validation loss decreased (0.028633 --> 0.025342).  Saving model ...
Time 1.014 (1.841) Data 0.944 (1.699)
Epoch: 13 	Training Loss: 0.021417 	Validation Loss: 0.022512
Validation loss decreased (0.025342 --> 0.022512).  Saving model ...
Time 1.014 (1.844) Data 0.945 (1.703)
Epoch: 14 	Training Loss: 0.019662 	Validation Loss: 0.023515
Time 0.949 (1.848) Data 0.871 (1.708)
Epoch: 15 	Training Loss: 0.020129 	Validation Loss: 0.022286
Validation loss decreased (0.022512 --> 0.022286).  Saving model ...
Time 0.889 (1.853) Data 0.809 (1.713)
Epoch: 16 	Training Loss: 0.016985 	Validation Loss: 0.018854
Validation loss decreased (0.022286 --> 0.018854).  Saving model ...
Time 0.959 (1.857) Data 0.879 (1.718)
Epoch: 17 	Training Loss: 0.017549 	Validation Loss: 0.020241
Time 0.993 (1.854) Data 0.909 (1.715)
Epoch: 18 	Training Loss: 0.015921 	Validation Loss: 0.016541
Validation loss decreased (0.018854 --> 0.016541).  Saving model ...
Time 0.966 (1.855) Data 0.895 (1.717)
Epoch: 19 	Training Loss: 0.015160 	Validation Loss: 0.020073
Time 0.910 (1.853) Data 0.837 (1.715)
Epoch: 20 	Training Loss: 0.015058 	Validation Loss: 0.015649
Validation loss decreased (0.016541 --> 0.015649).  Saving model ...
Time 1.010 (1.851) Data 0.941 (1.713)
Training ended at Sat Jan  5 14:50:24 2019, total training time is 0hours 25minutes 34 seconds.



I suspect whether the model-saving operation or the train-eval switch might be the source of the phenomenon. So,  I tried a clean training without all these stuff. Sadly, still the same.


I tried to double the batch_size (from 512 to 1024), the peaks are taller but still, in a zig-zag pattern:

3.png1251×1111 38.4 KB



Finally, I set pin_memory=True. The utilization graph has some changes (shared memory), but still, the GPU is not fully utilized.

4.PNG1251×1269 41.9 KB



I wonder whether this has something to do with DataParallel, will update after I run an experiment.