Thanks all for the discussion! Has anyone had problems in inference despite training using the teacher-forcing with decoder input shifted left and loss computed on the target shifted right? I am beginning to suspect that too much teacher forcing during training is disrupting the ability of the network to independently estimate the sequence without using ground-truth priors. The inference on training data itself fails badly in my case despite a successful training. I am training the transformer model to generate spectrogram like features.