We are actively working on adding cuSolver methods to the torch.linalg namespace and the PR for svd is here 4.
Currently MAGMA would be used, which can fallback to the CPU for smaller sizes of the inputs.