Actually, I was trying to replicate this research. https://arxiv.org/pdf/1805.07848.pdf 8
In this paper, the authors trained multiple decoders that share an encoder using ADAM.
I don’t know how they avoided this problem.