How do Sparsity Priors help for Identifiability? Let's say we have a Factor Analysis model with a latent variable $\mathbf{z}_t \in \mathbb{R}^k$:
$$x_t = A z_t + \epsilon_t, \qquad \epsilon_t \sim \mathcal{N}(0, \Sigma)$$
Let $A \in \mathbb{R}^{g\times k}$, and $\Sigma \in \mathbb{R}^{g\times g}$ and $\Sigma$ is diagonal.
Many sources say that the model is not identifiable. If I understand correctly, identifiability is studied by showing the uniqueness of the covariance matrix $\Lambda = A A^T + 
\Sigma$.
One issue is rotation: given an orthogonal rotation matrix $Q$, it holds that: $\Lambda = (A Q^T) (Q A^T) + \Sigma = A A^T + \Sigma$.
This problem can be tackled by e.g. assuming that $A$ is a lower-triangular (LT) matrix.
Instead of using LT matrices, some people use a sparsity-inducing spike-and-slab or horseshoe prior on all elements of $A$. It's clear to me that I have to impose a constraint on $A$ to make the solution unique, but I don't understand why placing sparsity inducing priors is one solution, because it is just a prior which is not enforced in the posterior solution. In Baysian inference, I place a normal prior on $z$ and a horseshoe prior on $A$:
$$z_t \sim \mathcal{N}(0, 1)$$
and
$$A_{gk} \sim \mathcal{N}(0, \tau^2 \cdot \lambda_{gk}^2)$$
How does this render the problem identifiable?

They don't --- adding a prior does not solve the identifiability problem
In this model you have an identifiable parameter $\Lambda = AA^\text{T} + \Sigma$ but the parameter $A$ is not identifiable.  The notion you are putting forward (which is unfortunately a popular belief) is that if you use a Bayesian model with a prior for $A$ then this will make up for the fact that $A$ is not identifiable.  The idea that this works comes from the fact that if you impose a prior then you will get a valid posterior for the unidentifiable parameter, which some analysts take to mean that identifiability is no longer a problem.  Unfortunately, it turns out that this posterior is not consistent, and it is almost entirely determined by the specified relationship between $\Lambda$ and $A$ in the prior.  (For a closely related discussion, see this related question.)
To see what happens when you impose a prior in this type of model, suppose you observe $n$ data values giving the sample vector $\mathbf{x}_n$.  The likelihood function here is fully determined by $\Lambda$, so the posterior for both parameters can be decomposed as:
$$\begin{align}
\pi(\Lambda,A|\mathbf{x}_n) 
&= p(A|\Lambda,\mathbf{x}_n) \ \pi(\Lambda|\mathbf{x}_n) \\[6pt]
&= p(A|\Lambda) \ \pi(\Lambda|\mathbf{x}_n). \\[6pt]
\end{align}$$
The second term in this product is the posterior for the identifiable parameter and the first is the conditional distribution of the unidentifiable parameter given the identifiable parameter.  The latter is fully determined by the prior distribution on the parameters; it is not influenced by the data.  So, you can see that the posterior you get for your non-identifiable parameter is determine in large part by the stipulated relationship between the parameters in the prior.
It gets worse.  Under appropriate conditions, as $n \rightarrow \infty$ the posterior for the identifiable parameter will converge towards a point-mass distribution on its true value $\Lambda_*$ (which we call posterior consistency).  We would hope that we would also have posterior consistency for the non-identifiable parameter of interest, but instead we get:
$$\begin{align}
\pi(\Lambda,A|\mathbf{x}_n) 
&\rightarrow p(A|\Lambda) \ \pi(\Lambda|\mathbf{x}_\infty) \\[6pt]
&\propto p(A|\Lambda = \Lambda_*). \\[6pt]
\end{align}$$
This shows us that the asymptotic posterior for the non-identifiable parameter is affected by the data only to the extent that it effectively allows us to condition on the true value of the identifiable parameter.  The uncertainty in the resulting asymptotic prior is fully determined by the specified relationship between $\Lambda$ and $A$ in the prior.  As you can see, even though we get a valid posterior from this type of analysis, we do not have posterior consistency in our inference and the resulting inference is determined primarily by our prior, rather than by the data.  This means that you can do a Bayesian analysis with a non-identifiable parameter, but it does not solve the identifiabilty problem and it does not let you get a sensible inference for non-identifiable parameters.
