Transformer tricks: Removing weights for skipless transformers
<|reference_start|>Transformer tricks: Removing weights for skipless transformers: He and Hofmann (arXiv:2311.01906) detailed a skipless transformer without the V and P (post-attention projection) linear layers, which reduces the total number of weights. However, this scheme is only applicable to MHA (multi-head attention), but not for MQA (multi-query attention) and GQA (grouped-query attention). The latter schemes are used by many popular LLMs such as Llama 2, Mistral, Mixtral, PaLM, and Gemma. Therefore, this micro-paper proposes mathematically equivalent versions that are suitable for MQA and GQA. For example, removing Q and P from a skipless version of Mistral-7B would remove 15% of its weights (and thus reduce its compute and memory complexity). See arXiv:2402.13388 and https://github.com/OpenMachine-ai/transformer-tricks for code and more transformer tricks.<|reference_end|>
@article{graef2024transformer,
    title={Transformer tricks: Removing weights for skipless transformers},
    author={Nils Graef},
    journal={arXiv preprint arXiv:2404.12362},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.12362},
    primaryClass={cs.LG}
}