Hi,
Nothing within pytorch will move the data between CPU/GPU automatically. So the computations are definitely not done on the CPU.
I am not sure though that the batched version of svd exists. It may be doing a for-loop over the batch under the hood and call svd on each submatrix. If your NxM matrices are small, then you would see a very high cpu usage due to the launching of the kernels for each matrix while the GPU will stay almost idle as it has almost nothing to do.
Unfortunately, this is expected behavior for the GPU: unless you have large parallelizable operations, it will be less efficient than the CPU