Hello Raphikowski!



 Raphikowski:

If I have a multivariate linear regression kind of problem of the form:
X @ b = y
with X being a (N x M) matrix, b being a (M x 1) column vector and y being a (N x 1) column vector. And I want to solve this now for X


To be clear:  Given vectors b and y, you wish to find a matrix X
such that X @ b = y.
The first thing to note is that there will be many such X.  If Q is a
matrix that has y as an eigenvector (with eigenvalue one) and R
is a matrix that has b as an eigenvector (with eigenvalue one),
then Q @ X and X @ R will both also solve your equation, that is,
both (Q @ X) @ b = y and (X @ R) @ b = y.
One solution to your equation is given by
X = (y @ b.t()) / b.norm()**2

as illustrated by the following script:
import torch
torch.__version__

torch.manual_seed (2020)

# let M = 5 and N = 7, and generate random values for b and y
b = torch.randn(5,1)
y = torch.randn(7,1)
b
y

X = (y @ b.t()) / b.norm()**2
X

X @ b
# check result two ways
X @ b - y
X @ b / y

# try another random vector
c = torch.randn(5,1)
# multiplication by this X always gives a vector proportional to y
X @ c / y

And here is its output:
>>> import torch
>>> torch.__version__
'0.3.0b0+591e73e'
>>>
>>> torch.manual_seed (2020)
<torch._C.Generator object at 0x0000022EF2646630>
>>>
>>> # let M = 5 and N = 7, and generate random values for b and y
... b = torch.randn(5,1)
>>> y = torch.randn(7,1)
>>> b

 1.2372
-0.9604
 1.5415
-0.4079
 0.8806
[torch.FloatTensor of size 5x1]

>>> y

 0.0529
 0.0751
 0.4777
-0.6759
-2.1489
-1.1463
-0.2720
[torch.FloatTensor of size 7x1]

>>>
>>> X = (y @ b.t()) / b.norm()**2
>>> X

 0.0113 -0.0088  0.0141 -0.0037  0.0081
 0.0161 -0.0125  0.0201 -0.0053  0.0115
 0.1024 -0.0795  0.1276 -0.0338  0.0729
-0.1449  0.1125 -0.1805  0.0478 -0.1031
-0.4607  0.3576 -0.5740  0.1519 -0.3279
-0.2457  0.1908 -0.3062  0.0810 -0.1749
-0.0583  0.0453 -0.0727  0.0192 -0.0415
[torch.FloatTensor of size 7x5]

>>>
>>> X @ b

 0.0529
 0.0751
 0.4777
-0.6759
-2.1489
-1.1463
-0.2720
[torch.FloatTensor of size 7x1]

>>> # check result two ways
... X @ b - y

1.00000e-07 *
  0.0000
  0.0000
  0.2980
  0.0000
 -2.3842
  0.0000
  0.0000
[torch.FloatTensor of size 7x1]

>>> X @ b / y

 1.0000
 1.0000
 1.0000
 1.0000
 1.0000
 1.0000
 1.0000
[torch.FloatTensor of size 7x1]

>>>
>>> # try another random vector
... c = torch.randn(5,1)
>>> # multiplication by this X always gives a vector proportional to y
... X @ c / y

-0.2834
-0.2834
-0.2834
-0.2834
-0.2834
-0.2834
-0.2834
[torch.FloatTensor of size 7x1]

I will say, however, that I don’t really understand what you are trying
to do.  There’s a lot of freedom in the matrix X – in fact, probably too
much freedom – so I suspect that what you are trying to here isn’t
really what you want to be doing.
Good luck!
K. Frank