I am wondering if you use a TransformerEncoder 11 followed by a TransformerDecoder 2, will it help your case?
The output of the TransformerEncoder could be the latent representation in this case. This paper 14 maybe useful.