Alex,
Good to hear it fixed it.
One thing that helps is to plot the evolution of Reconstruction Loss and the KL Loss to have a better assessment of whats going on.
I have also been through some issues in training VAE, as mentioned, and one tricky problem is the issue called " Posterior Collapse".  Basically, your decoder ignores latent codes from the encoder and tries to model the data on it´s own (“auto decoding”).
This issue is well documented here: Fixing the Broken ELBO 1. In addition, I also like these two papers to better understand the whole idea: Deep Variational Information Bottleneck and Learning Sparse Latent Representations with the Deep Copula Information Bottleneck. Also googling “VAE Posterior Collapse” may also point out to other interesting things.
I would suggest the following:

Tweak hyperparameters (learning rate, network architecture) and see how the losses evolve. Maybe the architecture is not power enough and may not be the “Posterior Collapse issue”. However, if so move to point 2 below.
After understanding the “Posterior Collapse” issue, then tweak the losses as pointed out in the papers. For example, one such thing could be add a penalization term beta to KL part.
If somehow you tried all these things, and reconstruction loss may still be weird and there is still the suspicion of posterior collapse you may just change your loss to the Mean Maximum Discrepancy. It has been shown to reduce (or eliminate?) this issue. A proper reference is found here InfoVAE: Balancing Learning and Inference in Variational Autoencoders 2. And there is a tutorial implementing it here: A Tutorial on Information Maximizing Variational Autoencoders (InfoVAE) 3.

By following the steps above I guess it may help.
Regards,
G.