There notes seem relevant for you

  
      
      GitHub
  
  
    

gchanan/pytorch 116

Tensors and Dynamic neural networks in Python  with strong GPU acceleration - gchanan/pytorch


  
  
    
    
  
  


But there is not yet support for batched svd in pytorch. It seems that numpy just do a for loop over the batches 51, which would be easy to add on the CPU side, but on the GPU side it would be trickier.