Not really. I checked the GPU usage with
watch -n 0.1 nvidia-smi

It seems that they actually runs in sequential. The GPU usage will be high to low one after the other.
I can better confirm that if there is any profiling tool for pytorch GPU.