  The posterior collapse phenomenon in variational autoencoders (VAEs), where
the variational posterior distribution closely matches the prior distribution,
can hinder the quality of the learned latent variables. As a consequence of
posterior collapse, the latent variables extracted by the encoder in VAEs
preserve less information from the input data and thus fail to produce
meaningful representations as input to the reconstruction process in the
decoder. While this phenomenon has been an actively addressed topic related to
VAEs performance, the theory for posterior collapse remains underdeveloped,
especially beyond the standard VAEs. In this work, we advance the theoretical
understanding of posterior collapse to two important and prevalent yet less
studied classes of VAEs: conditional VAEs and hierarchical VAEs. Specifically,
via a non-trivial theoretical analysis of linear conditional VAEs and
hierarchical VAEs with two levels of latent, we prove that the cause of
posterior collapses in these models includes the correlation between the input
and output of the conditional VAEs and the effect of learnable encoder variance
in the hierarchical VAEs. We empirically validate our theoretical findings for
linear conditional and hierarchical VAEs and demonstrate that these results are
also predictive for non-linear cases.

Machine Learning;Machine Learning