@Alexey_Demyanchuk ，
Thank you!
I did another interesting experiment on my another computer:
CPU i9-9900K, ram 32G, GPU 2080Ti(11G),
I trained the same yolov3 code, but with coco datasets.
the batch size is 8, and there are totally 122218 images that are separated to 15278 batches.  num_workers =8, it could be successfully trained. Furthermore, the training time for each batch is around 0.28s.

this experiment confirm my doubts:  

the training speed on my 3090 could be faster. there might be some hardware or software bottleneck.
my 3090 computer has much more hardware resource than my 2080Ti(11G) computer. however, on 3090 computer, num_workers could be set to 4 but not 8. there must be something that I should explore.