Hi,
I am not 100% sure but I would say this is expected for the main thread. Indeed the autograd engine works by using worker threads for different GPUs during backward. So the main thread will mostly wait for them to finish.
You can check the autograd profiler here 255 to see which part of your net is taking the most time.