title: GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
abstract: Attention based language models have become a critical component in state-of-the-art natural language processing systems. However, these models have significant computational requirements, due to long training times, dense operations and large parameter count. In this work we demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. First, we rely on grouped transformations to reduce the computational cost of dense feed-forward layers, while preserving the expressivity of the model . Secondly, we add a grouped convolution module to complement the self-attention module, decoupling the learning of local and global interactions. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales. We further highlight its improved efficiency, both in terms of floating-point operations (FLOPs) and time-to-train.
authors: Ivan Chelombiev Daniel Justus Douglas Orr Anastasia S. D. Dietrich Frithjof Gressmann Alexandros Koliousis Carlo Luschi
hIndex: Alexandros Koliousis 10