deltaskelta:

KL doesn’t replace CE, its just an added regularization term on whatever distribution is being reparameterized to keep it from drifting too far from the prior.


Thanks, I agree that KL doesn’t replace CE, I’m just questioning whether “empirically” is really contributing to the final model? E.g. have you seen any results/differences between a vae with and without a KL regularisation term? Is the final model with kl regularisation orders of magnitude better than the one without or is it just negligible difference?