I think you should look into how to generate variable speech data look into Tacotron and other idea’s there is VAE-Tail 2  which gives an idea on how to do it with just VAE.

Also, why is it so hard to reconstruct raw audio data?

Audio has a lot of information in other frequency domains. These features are not captured directly with the audio waveform, you need better feature representations for it.