Hi Raphikowski!



 Raphikowski:

Given I have some column vector of shape (5 x 1) I want to calculate the inverse of (q @ q.t()). This actually shouldn’t be such a difficult task, however, it seems that the inverse of this expression, torch.inverse(q @ q.t()) is way off:
…
So is there something special by matrix multiplying a column vector with its transpose to obtain a matrix?
So maybe what I am missing is that q @ q.t() is often singular or close to singular? And hence the inverse is actually not defined?


Yes, your suspicion is correct.
By construction, your matrix is mathematically non-invertible.
(Numerically, it is likely technically invertible, but singular within
round-off error, so you get nonsense results when you try to invert
it numerically.)
Your matrix – by construction – is a projection matrix; that is, by
matrix-vector multiplication it projects any vector onto the vector q.
As such, it has only one non-zero eigenvalue, with corresponding
eigenvector (proportional to) q.
This is illustrated by the following (version 0.3.0) script:
import torch
torch.__version__

torch.manual_seed (2020)
q = torch.randn(5,1)
q   # your random vector
qq = q @ q.t()
qq   # your square "projection" matrix

vals, vecs = torch.eig (qq, True)

vals   # all but one zero eigenvalue

max, ind = vals[:,0].max (0)   # get index of non-zero eigenvalue
vec = vecs[:,ind]   # get non-zero eigenvector

q.div (vec)   #  q is (proportional to) the non-zero eigenvector

And here is the output:
>>> import torch
>>> torch.__version__
'0.3.0b0+591e73e'
>>>
>>> torch.manual_seed (2020)
<torch._C.Generator object at 0x000001F5A11D6630>
>>> q = torch.randn(5,1)
>>> q   # your random vector

 1.2372
-0.9604
 1.5415
-0.4079
 0.8806
[torch.FloatTensor of size 5x1]

>>> qq = q @ q.t()
>>> qq   # your square "projection" matrix

 1.5307 -1.1882  1.9072 -0.5046  1.0895
-1.1882  0.9224 -1.4805  0.3917 -0.8457
 1.9072 -1.4805  2.3763 -0.6287  1.3574
-0.5046  0.3917 -0.6287  0.1664 -0.3592
 1.0895 -0.8457  1.3574 -0.3592  0.7754
[torch.FloatTensor of size 5x5]

>>>
>>> vals, vecs = torch.eig (qq, True)
>>>
>>> vals   # all but one zero eigenvalue

 0.0000  0.0000
 5.7712  0.0000
 0.0000  0.0000
-0.0000  0.0000
-0.0000  0.0000
[torch.FloatTensor of size 5x2]

>>>
>>> max, ind = vals[:,0].max (0)   # get index of non-zero eigenvalue
>>> vec = vecs[:,ind]   # get non-zero eigenvector
>>>
>>> q.div (vec)   #  q is (proportional to) the non-zero eigenvector

 2.4023
 2.4023
 2.4023
 2.4023
 2.4023
[torch.FloatTensor of size 5x1]

>>>

Best.
K. Frank