UPDATE
It turns out my server was full-occupied when I was running the PyTorch version of my network. It now runs ~100 examples/sec with ~80% GPU utilization. The speed may improve further when the server has more free computation power.
Original answer
The same problem appeared in my case too. But instead of reading from imdb, I am reading the original JPEG file. GPU utilization is constantly lower than 50%. I have implemented my network based on TensorFlow too. In TensorFlow, it can run ~120 examples/sec, while in PyTorch it can only run ~60 examples/sec.
My Dataset implementation is similar to ImageFolder. Multiple workers didn’t help. In fact, the more workers I use, the slower the loading speed. This happens in my TensorFlow implementation too, so only 1 worker is used in my TensorFlow implementation. However, In PyTorch, 0 worker is optimal for me. 1 worker will dramatically decrease the speed to ~30 examples/sec.