I don’t understand why the second method is wrong. I am actually trying to maximize ELBO which can be written as E[log p(x|z) + log p(z) - log q(z|x)] = E[log p(x|z)] - KL(q||p). Now this KL divergence can be written as the difference of the cross-entropy H(q,p) and the entropy H(q). So we can write H(q) as -postlogprob * samples and H(q,p) as -priorlogprob * samples where samples are sampled from the q distribution.
What’s wrong with this formulation?
I know we can compute KL analytically using a closed form formula for Univariate Gaussians.
I am actually confused as to how the KL divergence computed in this link is correct:
https://github.com/ericjang/gumbel-softmax/blob/master/Categorical%20VAE.ipynb 13 (Cell 5)