It’s been a while, but I think I was able to get this work by mimicking the architecture in the paper, which is roughly linear > tanh > linear > tanh. See the definition here: https://github.com/mattjj/svae/blob/master/experiments/gmm_svae_synth.py 20.