How do Sparsity Priors help for Identifiability? Let's say we have a Factor Analysis model with a latent variable $\mathbf{z}_t \in \mathbb{R}^k$:
$$x_t = A z_t + \epsilon_t, \qquad \epsilon_t \sim \mathcal{N}(0, \Sigma)$$
Let $A \in \mathbb{R}^{g\times k}$, and $\Sigma \in \mathbb{R}^{g\times g}$ and $\Sigma$ is diagonal.
Many sources say that the model is not identifiable. If I understand correctly, identifiability is studied by showing the uniqueness of the covariance matrix $\Lambda = A A^T + 
\Sigma$.
One issue is rotation: given an orthogonal rotation matrix $Q$, it holds that: $\Lambda = (A Q^T) (Q A^T) + \Sigma = A A^T + \Sigma$.
This problem can be tackled by e.g. assuming that $A$ is a lower-triangular (LT) matrix.
Instead of using LT matrices, some people use a sparsity-inducing spike-and-slab or horseshoe prior on all elements of $A$. It's clear to me that I have to impose a constraint on $A$ to make the solution unique, but I don't understand why placing sparsity inducing priors is one solution, because it is just a prior which is not enforced in the posterior solution. In Baysian inference, I place a normal prior on $z$ and a horseshoe prior on $A$:
$$z_t \sim \mathcal{N}(0, 1)$$
and
$$A_{gk} \sim \mathcal{N}(0, \tau^2 \cdot \lambda_{gk}^2)$$
How does this render the problem identifiable?

This answer is not a proof that those priors will render the problem identifiable but an intuitive explanation that it should work.
As you have pointed out, the problem with ordinary factor analysis (FA) is that there are infinitely many different linear maps $A: z\to x$ which lead to the same likelihood of the data in this FA model. Thus, the idea is to weigh all those $A$ differently by giving them different prior probabilities.
The original FA could be thought of as all $A$ having the same weight (an improper prior), and by giving them different priors (weights) the above tie is broken and, hopefully, there is only one $A$ left that gives the largest likelihood to the data $\{x_t\}$. I am not sure what you refer to by "it is just a prior which is not enforced in the posterior solution", but the prior does change the posterior.
You have shown that, if $A$ is a solution to the original FA, i.e. an MLE solution (frequentist) or a MAP for a constant improper prior (Bayesian), then all $A^\prime = AQ$, with $Q$ any orthonormal map, are also MLEs (MAPs). I.e., they all have the same a posteriori density. Thus, intuitively, the plan is to choose a prior that is not constant on this set of matrices $A^\prime$, which also makes the posterior there nonconstant.
And while we are at it, we should also design the prior such that certain nice features of $A$, e.g. sparsity, are favored. In this case, it makes sense to use e.g. horseshoe priors.
In other words, the solution with horseshoe priors will be the sparsest matrix that is also "near" the set $\{A^\prime=AQ| Q\;\mbox{orthonormal}\}$.
Finally, note that restricting to LT matrices can be interpreted as a similar approach, where priors are just very strongly penalizing all matrices with nonzero values in the upper triangle.

Edit: Why did I write that "the solution with horseshoe priors will be the sparsest matrix that is also "near" the set $\{A^\prime=AQ| Q\;\mbox{orthonormal}\}$.", why only "near"?
Let's give this set a name, $S:=\{AQ|Q\in O(k)\}$, where $O(k)$ are the orthonormal matrices. For the improper prior $p^i$, the posterior $f^i$ has infinitely many MAPs, given by the set $S$. Note, that this doesn't mean that all the other matrices $A$ are impossible. They are very well possible, they are just not as likely. In fact, when computing the Bayesian predictive distribution, all $A$ are integrated over.
Next, we create a new posterior $f^n$ by multiplying the old posterior $f^i$ with a nonconstant prior $p^n$ (and normalizing), $f^n\sim f^i p^n$. Now it is not clear that the maximum of $f^n$ should also be a maximum of $f^i$, ie. be in $S$. In general, $f^i$ is preferring matrices in $S$, while $p^n$ is preferring sparse matrices, and thus the product $f^n$ will be a compromise which in general is not completely sparse (read: not a maximum of the horseshoe prior $p^n$) and not completely in $S$.
