If the CPU computation uses the model, you would have to wait for the synchronizing to('cpu') operation since you would have to push the model to the CPU and back to the GPU again.
On the other hand, GPU operations are already executed asynchronously, so in case the CPU operation has no dependency on the model etc. it would already be executed while the model is busy with the training on the GPU.