Understanding disentangling in \beta$ VAE
We present new intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate distortion theory perspective, we show the circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising the modified ELBO bound in \beta$ VAE, as training progresses. From these insights, we propose a modification to the training regime of \beta$ VAE, that progressively increases the information capacity of the latent code during training. This modification facilitates the robust learning of disentangled representations in \beta$ VAE, without the previous trade off in reconstruction accuracy.