Sorry, I think you might have misunderstood the first line of my reply. What I meant was that as I understood this (http://icl.cs.utk.edu/magma/software/index.html 8) is the magma original library. And this (https://github.com/maxhutch/magma/blob/master/src/sgesdd.cpp 7) is a fork of magma to include more BLAS. The second is what PyTorch uses. Is that correct?
And yes from what I see in the file you pointed out this is not a pure GPU implementation, there are several calls to LAPACK. Thanks very much for the clarification.