Solved by Hari_Krishnan in post #3 
              
              
                My bad, Solved it by upscaling the latent dimension to embedding size before passing it to the TransformerDecoderLayer.