Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks
<|reference_start|>Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks: Recent studies have shown that, relative position encoding performs well in selective state space model scanning algorithms, and the architecture that balances SSM and Attention enhances the efficiency and effectiveness of the algorithm, while the sparse activation of the mixture of experts reduces the training cost. We studied the effectiveness of using different position encodings in structured state space dual algorithms, and the more effective SSD-Attn internal and external function mixing method, and designed a more efficient cross domain mixture of experts. We found that the same matrix is very wonderful in different algorithms, which allows us to establish a new hybrid sparse architecture: Cheems. Compared with other hybrid architectures, it is more efficient and more effective in language modeling tasks.<|reference_end|>
@article{shi2024wonderful,
    title={Wonderful Matrices: More Efficient and Effective Architecture for
  Language Modeling Tasks},
    author={Jingze Shi and Bingheng Wu and Lu He and Luchang Jiang},
    journal={arXiv preprint arXiv:2407.16958},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.16958},
    primaryClass={cs.LG cs.AI}
}