Thanks for taking interest. The matrix inversion is part of a formula that is supposed to project features orthogonally.

$F_{G}$ (comes from concatenation of two vectors) is not a square matrix, that’s why I make this modification.
The issue is, that my network performs much better if I reduce second dimensions of X and Y to half of what it was (768->384), in all other cases, it doesn’t learn anything. There might be better ways of projection, which I’ve not explored. (For instance, if bs=512, 768->512, then I don’t notice any improvement.)
This is an unstable method. The main issue is to perform inversion. That implies I need square matrices. That’s why I’m asking if there’s a better mathematical way to perform this.
Yes, I realize that my implementation is not batch size agnostic once model is trained. As a result, I am unable to train bigger models (because batch_size = hidden_size/2 won’t fit in memory).
So in general sense, what is a better way to perform inversion when the matrix is not square (pinverse isn’t stable) ?