Sine Activated Low-Rank Matrices for Parameter Efficient Learning
<|reference_start|>Sine Activated Low-Rank Matrices for Parameter Efficient Learning: Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates a sinusoidal function within the low-rank decomposition process. This approach not only preserves the benefits of the parameter efficiency characteristic of low-rank methods but also increases the decomposition's rank, thereby enhancing model accuracy. Our method proves to be an adaptable enhancement for existing low-rank models, as evidenced by its successful application in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF), and 3D shape modeling. This demonstrates the wide-ranging potential and efficiency of our proposed technique.<|reference_end|>
@article{ji2024sine,
    title={Sine Activated Low-Rank Matrices for Parameter Efficient Learning},
    author={Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, Simon
  Lucey},
    journal={arXiv preprint arXiv:2403.19243},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.19243},
    primaryClass={cs.LG cs.CV cs.NE}
}