IDK what I was really asking in the original question. KL is just a autograd enabled version of the KL divergence for the given distributions. the kl_divergence function does nothing for reparameterization. Reparameterization happens by manually computing it or using the rsample() function which is present on many of the distributions in Pytorch.
Also, KL doesn’t replace CE, its just an added regularization term on whatever distribution is being reparameterized to keep it from drifting too far from the prior.