Subspace method fails at identifying parameters in a state space system I am trying to infer the parameters of a linear multivariate time-invariant state space system using a subspace method. However, the inferred parameters do not match the ground-truth parameters used to generate the observations.
My system is defined as follows:
$$
x_{t+1} = Ax_t
$$
$$
y_t = Bx_t
$$
where $x_t$ is a vector of latent state of dimension $n_x$, $y_t$ is a vector of observations of dimension $n_y$, $A$ is the $n_x \times n_x$ transition matrix and $B$ is the $n_y \times n_x$ observation matrix. $A$ is supposed to have all its eigenvalues in the unit circle. The ground truth parameters are generated as follows:
A = Diagonal(rand(Uniform(-1,1),nx)) (i.e. $A$ is simply a diagonal matrix which eigenvalues are in the unit circle)
B = rand(Normal(),ny,nx) (i.e. coefficients in $B$ are randomly drawn from a normal distribution)
x[1,:] = rand(Normal(),nx) (i.e. the initial state is randomly drawn from a normal distribution).
$N$ observations and hidden states are then generated using the system defined above. I then stack these observations to obtain the Hankel matrix of order $1$:
$$
M =  \begin{bmatrix}
   y_1 & y_2 & \cdots & y_{N-1} \\
   y_2 & y_3 & \cdots & y_N  
 \end{bmatrix}
$$
using the following code:
i = 1
M = zeros((i+1)*ny,N-i)
for row in 1:(i+1)
    for col in 1:(N-i)
        M[((row-1)*ny+1):row*ny,col] = y[col+row-1,:]
    end
end

$M$ can be expressed as
$$
M = \begin{bmatrix}
   Bx_1 & Bx_2 & \cdots & Bx_{N-1} \\
   BAx_1 & BAx_2 & \cdots & BAx_{N-1} 
 \end{bmatrix}
= \begin{bmatrix}
   B \\
   BA 
 \end{bmatrix}
\begin{bmatrix}
   x_1 & x_2 & \cdots & x_{N-1}  
 \end{bmatrix}
$$
From my simulation, I can indeed verify that the upper block of
$$
M[x_1 \  x_2   \cdots \  x_{N-1}]^{\dagger}
$$
is equal to $B$. I then compute the SVD of $M$:
$$
M = USV^t = \Gamma \Omega
$$
where $\Gamma = US^{1/2}$ and $\Omega = S^{1/2}V^t$. Classical literature on subspace identification (e.g. this paper) then indicates that $B$ can be directly read from the first block row of $\Gamma$. On the other hand, the BRML textbook (Chapter 24.5.3) states that $B$ is contained in the first block of $U$ while the vector of hidden states (from which $A$ can be estimated) can be read from the first block of $SV^t$. But none of these methods seem to work: I am using the following code to obtain my estimates $\hat{A}$ and $\hat{B}$:
B_hat = U[1:ny,1:nx]
X_hat = (Diagonal(S)*V')[1:nx,1:(N-i)]
A_hat = X_hat[:,2:end]*pinv(X_hat[:,1:end-1])

My estimates $\hat{A}$ and $\hat{B}$ greatly differ from the ground-truth values $A$ and $B$. Moreover, using $\hat{A}$ and $\hat{B}$ to generate observations instead of $A$ and $B$ (using the same original hidden state $x_1$) leads to different observations:

How comes subspace identification fails at estimating $A$ and $B$? Can it be due to the dynamics of $x_t$ induced by $A$ (since all eigenvalues are within the unit circle, $x$ converges rapidly to 0)? The same problem arises for different values for $A$. More generally, I am only aware of 2 methods for identifying the parameters of a state space system from its observations: the subspace method described above, or the EM algorithm. Are there other solutions? Thanks!

It turns out that, even though the estimated matrices $\hat{A}$ and $\hat{B}$ differ from the ground-truth values $A$ and $B$, we still have:
$$
\hat{B} \hat{x} = \hat{y} \approx y
$$
(i.e. observations can be correctly estimated from the inferred states) and that $\hat{A}$ and $A$ have the same eigenvalues.
