Solved by KFrank in post #2 
              
              
                Hi Zeyuyun! 

While this “orthogonality penalty” is zero if, and only, if X is orthogonal, 
and is positive otherwise, it doesn’t work well for orthogonalizing X 
with gradient descent, because its structure doesn’t fit* well with the 
natural geometry of the set of orthogonal matrices. 
Replacing t…