Well, not that simple ;). What I found out: My model consists of a about 200 layers, when running on GPU, the GPU execution is extremely fast for individual layers, so that the CPU overhead of the “forward” calls (all the Sequentials in my model) becomes prominent and finally increases the CPU usage! So when running the forward() method of my module in a loop to test maximum throughput, I cannot even fully utilize the GPU for this reason - GPU load goes up to 60-70 percent, but the loop consumes an entire CPU core so that it cannot run the model faster…
Anyone experienced this when running model in eval() on GPU?
Thanks.
A.