DropAttention: A Regularization Method for Fully Connected Self Attention Networks
Variants dropout methods have been designed for the fully connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully connected self attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.
ArXiv