Hi. I have a weight matrix (call it Q). I’d like to factorize it as product of low-rank matrices to reduce model parameters. So i’d like to have q(left)q(right) = Q, where q(left) has a shape of (mxk) and q(right) has a shape of (kxn), where k <min(m,n). Is there any function in Pytorch to do this, or can I implement it like this:
m = 512 
k = 85  # if set to 85: (512 * 85) + (85 * 512) << (512 * 512)
n = 512  

q_left = nn.Parameter(torch.FloatTensor(m, k).uniform_(-math.sqrt(1/n)/2, math.sqrt(1/n)/2))
q_right = nn.Parameter(torch.FloatTensor(k, n).uniform_(-math.sqrt(1/n)/2, math.sqrt(1/n)/2))
q = torch.matmul(q_left, q_right)