Ardeal:

the training speed of num_workers=2 is quite faster than that of num_workers=1, however, there is small difference for num_workers=2 and 4.


It is enough to have 2 workers to prepare the data so to fully utilize GPU.



 Ardeal:

when num_workers=4, there are much CPU ram and GPU ram left, num_workers=8 doesn’t work. it is very weird.


It is not clear from your data. Actually, the amount of used memory (both CPU and GPU) increases then you increase number of workers. Also, first 2 columns have CPU RAM 16 and second 2 - 48. Are those two different machines?



 Ardeal:

num_workers assigns a few threads to load image to RAM(CPU). If batch size keeps 32, the RAM needed for CPU should not change. however, more RAM is used for num_workers = 4.


Not exactly true. Workers prepare data in advance to push it into GPU as fast as possible so to utilize GPU the most. All those workers require memory to put processed data from disk to memory, so increasing number of workers will increase memory footprint.