To get a clear answer you could profile the code e.g. via Nsight Systems (or the PyTorch profiler) and check the timeline to see which operations are executed.
Based on your description, I would guess that the CPU might be busy with some (single-core) work.