I have two matrix X and Y. Both are of dimension [bs, hidden_size].
I want to take inverse of matrix obtained by torch.cat([X, Y], dim=1).
Pseudo inverse is not stable (docs say the same). I obtain inverse to deal with this. So I used a Linear layer of dimension (hidden_size, batch_size//2) to convert X and Y of dimensions [bs, bs//2]. After concat, I get a matrix of form [bs, bs]. The downside of this approach is that it requires me to use a batch size = hidden_size/2 to perform well, otherwise the network doesn’t learn anything well.
Is there a mathematical elegant way you can suggest as a workaround ?