@Alexey_Demyanchuk ,
Thank you!
I am training yolov3 with input image size 640*640 to the network.
I agree with you.



 Alexey_Demyanchuk:

Also, It seems like 2 workers are actually enough for the task, as you can see that overall training time doesn’t improve much with further increasing number of workers.


with 2 workers, the CPU and GPU might be fully used. however, I would like to know more details about  how GPU and CPU are used. I would also like to know where the bottleneck in hardware is.
for my case,
during inference, if I run a video for inference(forward), the time needed is around 20ms.
during training, the time needed for one batch is around 0.73s=730ms which includes the time for forward and backward. If the time for forward is 20ms, the time for backward is around 710ms. 20ms VS 710ms, this seems to be unreasonable for powerful 3090 GPU. This is my major concern about the data in my table.
I would like to know your data. I think the ratio of the time needed for forward and back on your machine should be similar as mine. Furthermore, the time for forward and backward should be comparable for yours and mine.