The Conformer Encoder May Reverse the Time Dimension
<|reference_start|>The Conformer Encoder May Reverse the Time Dimension: We sometimes observe monotonically decreasing cross-attention weights in our Conformer-based global attention-based encoder-decoder (AED) models. Further investigation shows that the Conformer encoder internally reverses the sequence in the time dimension. We analyze the initial behavior of the decoder cross-attention mechanism and find that it encourages the Conformer encoder self-attention to build a connection between the initial frames and all other informative frames. Furthermore, we show that, at some point in training, the self-attention module of the Conformer starts dominating the output over the preceding feed-forward module, which then only allows the reversed information to pass through. We propose several methods and ideas of how this flipping can be avoided. Additionally, we investigate a novel method to obtain label-frame-position alignments by using the gradients of the label log probabilities w.r.t. the encoder input frames.<|reference_end|>
@article{schmitt2024the,
    title={The Conformer Encoder May Reverse the Time Dimension},
    author={Robin Schmitt, Albert Zeyer, Mohammad Zeineldeen, Ralf Schl"uter,
  Hermann Ney},
    journal={arXiv preprint arXiv:2410.00680},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00680},
    primaryClass={eess.AS cs.SD stat.ML}
}